Much like shared github access I need to be able to specify which ssh key a particular project’s connections to heroku should use. Since I want to use client side checkouts I can’t specify a private key within TeamCity but I do know that all my compatible build agents are local so I can add a new entry to `~/.ssh/config` and define a host which will allow me to connect to Heroku with a particular key.

# Heroku for #{new_project}
Host heroku-#{new_project}
HostName heroku.com
User git
ForwardAgent yes
IdentityFile /home/teamcity/.ssh/teamcity_#{new_project}_rsa


Add a ssh key to the Heroku account
Now I need to add this new key to the Heroku account for this project by installing the heroku gem in my project’s gemset and using the `heroku` command line to add the key. Heroku stores the account credentials used in `~/.heroku` and the user account running my build agents might need to connect to many different heroku accounts. I could switch my heroku credentials, upload a key, and have different projects deploy using different accounts. However since I find it useful to be able to run the heroku command line (see below) I found it preferable to have a single `teamcity` heroku account added to every project as a collaborator.
If you find it necessary to switch between Heroku accounts frequently this tool from Aeonscope might be useful: managing multiple heroku accounts

Now I can add a key to the current project’s Heroku account.
heroku keys:add ~/.ssh/teamcity_#{new_project}_rsa

Deploying to Heroku
Now I can add a simple TeamCity build configuration which pushes code to Heroku once a particular revision passes the tests.

A project's deployment build configuration

I found that I need to add a delay between turning on maintenance mode for the app and pushing to heroku. Attempting to push immediately fails with the error `Heroku push rejected, your slug is currently being compiled. Please try again later.` Even with a significant delay I still see this error occasionally and have to rerun the deploy build but at this point I don’t know of a way to make sure heroku can handle a new revision before attempting to push changes.

Now I can use TeamCity to automatically deploy changes to Heroku as soon as my project’s tests pass.

1. Create a key for github access
> ssh-keygen -t rsa -C "#{project_name}"
Upload the public key to github as a deploy key for the project so that the build agent can pull from the github repository.

2. Setup rvm and a gemset for the project
> rvm install ruby-#{version}-r#{patch level}
> rvm use ruby-#{version}-r#{patch level}
> rvm gemset create #{project name}
> rvm gemset use #{project name}
> gem install bundler

This could probably all be automated as part of the project’s .rvmrc file.

3. Setup a ssh config for the project
The keys used by many of the services we interact with are unique and tied to an account so we need to have TeamCity use different ssh keys when connecting to projects under different accounts. Add a new entry to ~/.ssh/config to alias a host to use a specific ssh key

# GitHub for #{new_project}
Host github-#{new_project}
HostName github.com
User git
ForwardAgent yes
IdentityFile /home/teamcity/.ssh/teamcity_#{new_project}_rsa


Connections to `github-#{new_project}` will now default to use `teamcity_#{new_project}_rsa` rather than the default `id_rsa`. By specifying the key in the ssh config file we’re able to use client side checkouts which will be useful when deploying using git. For projects able to use server side checkouts we can specify a private key to use in TeamCity’s VCS settings.

4. Create a new TeamCity project
On projects with multiple build steps using the same build runner we could create a template to avoid repeating these settings in each build phase. Unfortunately TeamCity 5.1 templates do not support running multiple types of build runners so projects with both command line (`bundle install`) and rake runners (`rake spec`) require some repetition.

Client checkouts have a full git repository while a server side checkout performs a git pull and then copies the resulting files but not the full git work-tree to the build agents leaving the agents unable to easily run git commands. A client side checkout is therefore preferable when using `git push` to deploy a project.

If using a template set the `rake.tasks`, `rake.opts`, and any other configuration parameters the template uses. Each build configuration based on the template can specify its own configuration parameters but cannot override the rake task and command line parameters directly.

To use the project’s RVM ruby and gemset add the following environment variables with appropriate paths:

Now I can finally build a set of build configurations for a project which will all use the project’s RVM settings to isolate it from other projects on the TeamCity server.

JetBrains appears committed to adding RVM support to TeamCity and added the `RVM Gemset` property to rake tasks in TeamCity 5.1.3 so hopefully we’ll see additional RVM support in future releases.

It’s not in every agile software team’s bag of tricks. Asking around our fairy savvy office, I discovered it’s far from universal. In the “State of Agile” report from Version One, which includes survey results from 2500 software developers, it wasn’t mentioned. Since I haven’t seen it show up that much in other agile writings, I thought I’d share my experiences here.

What is “5 Whys”? I picked up “5 Whys” from the lean software movement, which sprang from Toyota manufacturing. You can read about its history on wikipedia, but it’s pretty simple: at the end of the assembly line, when a widget comes out with a problem, you stop the line and ask “Why?” Whatever the reason, you ask the “Why?” again. Repeat at least 5 times. The goal is to discover the “root cause” of the defect, and fix the root cause, not just some symptom. Wikipedia has a good example around car repair. Here’s a software example:

1. Why did Sheryl say the sign-up flow was broken?
A: Because she was trying to sign up a second time. Duh.
2. Yeah, but why did the flow break if the user is already signed up?
A: It’s a bug. We didn’t have a test case for that.
3. Why didn’t we have a test case for it?
A: I just didn’t think of it.
4. Why didn’t you think about it?
A: The story seemed easy– I guess I didn’t think it through.
5. Why do you think you didn’t think it through?”
A: I guess I was working alone and was in a hurry.
…

Obviously you don’t have ask “Why?” exactly five times, but that does seem to be a pretty good number. The point is you start articulating the behaviors and procedures that cause the problems.

Taking It for a Spin. I’ve come up with exercise for agile software teams. Like I mentioned, when quality drops below your comfort level, give it a try:

1. (2 minutes) Make a list of the most recent bugs you’ve uncovered. I’ve found that the last 10 is usually enough– if not too much. You can put these on a white board, index cards or a wiki page– all will work.
2. (2 minutes) Prioritize them based on the ones you want to talk about. Which caused the most embarrassment or cost? You usually only have energy to talk about 5 or 6 of them.
3. (15 minutes) For each bug, ask “Why did this happen?” Then, probe deeper until you can’t get much further. Write the answers down where everyone can see.
4. (5 minutes) When you’re done with all the bugs (or out of time), circle common root causes.
5. (10 minutes) Brainstorm ways to mitigate or eliminate the root causes. Come of with one or two SMART goals for the team.

Some of the root causes we’ve gotten to recently:

• we didn’t pair on hard features
• we didn’t ask for clarifications on the requirements
• one engineer didn’t understand the intent behind the code
• nobody looked at the app on Internet Explorer 6

Just like in agile reflections, the changes made coming out of these meetings stick with the team. During one session, the developers decided that there were just too many mistakes that someone else would have easily caught– so they always needed two sets of eyes on every checkin. I’d advocated this before, but it was never applied consistently. To my surprise though, after the reflection, everyone was  disciplined about this policy– and our quality was much better.

app/
  views/
    home/
      index.html.rb
      index.css
      index.js

The convention is clear: to add page-specific CSS code, just create a new file with the same name as the view, in the same folder. Easy to add, edit and remove. The alternative (using the public folder), usually leads to a parallel hierarchy and the inconvenience of that.

I also thought, that this being Rails and all, the files should be included automatically. It turned out to be pretty easy.

The Implementation

Serving the Assets

To serve this file, we need a route and a controller.

First the route:

map.connect '/asset/:path',
    :controller => 'asset', :action => 'serve_asset', :path => /.*.(js|css)/

We need to be careful here, since we’re going to be serving files directly out of our view hierarchy– make sure the route only picks up the javascript and CSS files.

The corresponding controller:

class AssetController < ApplicationController
  def serve_asset
    path = params[:path]
    format = path.sub(/.*.(w+$)/, '1')
    respond_to do |format|
      format.js { render :file=>"#{RAILS_ROOT}/app/views/#{path}"}
      format.css { render(:file=>"#{RAILS_ROOT}/app/views/#{path}")}
    end
  end

Now the assets mentioned above are served as /assets/index.css and /assets/index.js.

Including the Assets

With this convention, we added code to our layout that includes these files automatically, but only if they exist. First, the additional controller code:

class AssetController < ApplicationController
  ...
  def self.include_asset(rel_path)
    return '' unless AssetController.asset_exists?(rel_path)
    case rel_path
      when /.js$/
        "<!--mce:0-->"
      when /.css$/
        "
"
    end
  end

  def self.asset_exists?(rel_path)
    FileTest.exists?(RAILS_ROOT + "/app/views/#{rel_path}")
  end

And the code needed in the (Erector) layout:

head do
  c = self.class
  rawtext AssetController.include_asset("#{c.name.underscore}.js")
  rawtext AssetController.include_asset("#{c.name.underscore}.css")
end

Actually, since it’s Erector, the views are Ruby classes, so there is a hierarchy. These assets may be specific to any one of these:

c = self.class
while c != Object
  rawtext AssetController.include_asset("#{c.name.underscore}.js")
  rawtext AssetController.include_asset("#{c.name.underscore}.css")
  c = c.superclass
end

Discussion

Although some people will bristle at the proliferation of files, I have used this technique repeatedly over the last five years (in Java and PHP projects). I find it leads to sound, modular development. With Ruby, it’ll be pretty straightforward to concat the assets together later on, to reduce requests, or even embed the code directly in the served HTML files, if it’s small enough. It’s also convenient to add page caching in the controller, when the time comes. Adding

  caches_page(:serve_asset)

copies all these files into public/assets (as they are accessed), which conveniently takes your mongrel (or whatever) out of the picture.

You might imagine this keeps you from organizing and structuring your CSS and Javascript; it doesn’t do that. It lowers the barier to using a “local” scope for the CSS and Javascript. It allows you to write page-specific CSS and Javascript, which if you’re developing code quickly, is a lifesaver. We have repeatedly opened a view-specific Javascript file, seen that it no longer applies to the code, and removed it– safely. It’s easy to edit scripts when you know they only effect one page. Without this, I tend to lump things together that don’t really belong, and it becomes hard to tease them apart later.

Thoughts?