There’s a (Vim Plugin) for That! Part I: Fuzzy Finder

One of the most popular features of TextMate and RubyMine is Command-T, the ability to jump to a file with just a couple of keystrokes. Vim lacks this feature out of the box, leaving Vim newcomers less efficient.

For the casual Vim users out there, when in doubt, remember this: there is a plugin for that!

Continue reading

Posted in Process | Tagged , | 7 Comments

Vim Text Objects: The Definitive Guide

To edit efficiently in Vim, you have to edit beyond individual characters. Instead, edit by word, sentence, and paragraph. In Vim, these higher-level contexts are called text objects.

Vim provides text objects for both plaintext and common programming language constructs. You can also define new text objects using Vim script.

Learning these text objects can take your Vim editing to a whole new level of precision and speed.

Posted in Process | Tagged | 33 Comments

A Modern Guide to Threads

I spoke recently at Rubyconf 2011 on some advanced topics in threading. What surprised me was how little experience people had with threads so I decided to write this post to give people a little more background on threads. Matz actually recommends not using threads (see below for why) and I think this is a big reason why Rubyists tend not to understand threading.
Continue reading

Posted in Web | 11 Comments

Crafting Commits in Git

A commit, like a well-designed function or class, should represent a single concept. A distinct, cohesive commit is easy to understand, review, and, if necessary, revert.

Git’s powerful staging area allows you to finely craft your commits. You decide which files and even which changes in a file, down to the individual line level, to commit.

Let’s take a look at how to use Git’s staging area to divide a large set of un-related changes into several smaller, well-defined changes.

Continue reading

Posted in Process | Tagged | 8 Comments

Explorations in Go: A dupe checker in Go and Ruby

I've recently started exploring the new(ish) programming language "Go". Go is intended to be a systems programming language and offers speed and low-level API along with some sweet features and the beginnings of a great standard library.

At Carbon Five we do most of our work in Ruby, JavaScript, Objective-C, and lately, node.js. I've really been enjoying Objective-C but realistically half the value is in the standard library, which is not public, which precludes its use in any kind of server environment to which we're likely to deploy (i.e. non-Mac). I also spent some time this year contributing C code to an open source project and remembering why I don't program in C, given the choice.

I'm exploring Go as a possible solution for times when I want to get close-to-the-metal and really control what's going on, but without having to reinvent the wheel. Example: having to write your own collection frameworks. Blehft.

I wrote a dupe checker in Ruby for a project recently and thought I'd write a Go port as an experiment. (In case you're curious about the motivation: tons of files get moved from a tree to a flat namespace, leading to collisions -- and confusion.)

I've written walkthroughs of the Ruby and Go versions below - please check it out.

Continue reading

Posted in Everything Else | Tagged , | 23 Comments

Lightning talks at Carbon Five, 9/28/2011

Carbon Five had a set of lightning talks over lunch recently. Giving our designers and developers a chance to share recent experiences and side projects.

A debate over the relative performance of different languages at our last hack night called for a measurable comparison. Mike demonstrated the resulting head to head to head speed test serving “Hello World” from Ruby, Node.js, and Go.
https://github.com/carbonfive/hellod

Dave gave us a summary of Agile UX presentations from the Balanced Team Conference

Jonah has been working on a tool to parse and manipulate Xcode project files. Using a parsing expression grammar defined in treetop to parse the file and generate a tree of ruby objects which could be manipulated to resolve merges, organize resources, or manage dependencies.
https://github.com/jonah-carbonfive/Project_err

Our resident node.js expert Ben explained how he has been using jasmine and zombie to practice BDD while working with node.
http://node-bdd.heroku.com

Rudy has been working on iOS apps and gave us a demonstration of how to use objective-c metaprogramming to monkey-patch new behaviors and state onto existing framework classes.
http://rudyjahchan.github.com/Monkey-Patching-Objective-C

We will be playing with these and other ideas at our next Hack Night on Wednesday October 5th if you would like to see more.

Posted in Everything Else | 1 Comment

Destructuring Assignment in CoffeeScript

CoffeeScript supports a subset of pattern matching called destructuring assignment. Destructuring assignment uses patterns to extract out the parts of an object during assignment. It’s a simple and elegant technique but when overused can have unintended consequences on a codebase.

CoffeeScript’s two data structures, Arrays and Objects, can both be destructured. Let’s first take a look at destructuring Arrays.

Continue reading

Posted in Web | 13 Comments

Automating Node.js deployment to EC2 with Chef

This post is a follow up to my colleague Ben’s excellent post from three weeks ago describing how he got node.js up and running on Amazon EC2. He asked me if I could automate his deploy in order to help him provision his production environment. I decided it was a good opportunity to learn Chef.

Continue reading

Posted in Ops, Web | Tagged , , | 10 Comments

Testing Doesn’t Scale

The Ruby community’s obsession with testing is unrivaled. Over the years, Rubyists have gone from old school TDD using test/unit, to modern BDD with RSpec and finally to comprehensive integration testing, including JavaScript support, via Cucumber. The goal was tests at all layers and to get as close as possible to simulating a real browser.

There’s no question that extensive test suites allow us to rapidly develop and deploy apps with confidence. Unfortunately as apps grow so do their test suites. Eventually a once agile test suite will become massive enough to slow development. Then it’s hard to find anyone willing to work on the app.

I’ve seen this happen time and time again on apps and I know I’m not alone. An app’s testing strategy has now become a major factor in its long-term success. How can we manage these large test suites? Let’s take a look at some possible solutions that might be able to help us out.

Continue reading

Posted in Process | 33 Comments

Improving Resque’s memory efficiency

Resque is a very popular message queueing system for Rails applications.  Here’s how I recently improved the memory efficiency of a Carbon Five customer’s resque processing farm by 68x!

The Problem

This customer has an existing investment in Resque and is a heavy user of a third-party Java API so they need to run their resque workers using JRuby.  Unfortunately this gets them the worst of all worlds:

  • JRuby does not fork so they don’t get the benefit of memory isolation by working in a child process
  • The JVM is relatively demanding in terms of memory
  • Resque is single threaded

To scale to the levels they wanted to get to, they projected that they would need to run hundreds of Resque processes, each consuming 512MB.  Insanity! If your problem requires a lot of concurrency to solve, using lots of large processes is a terrible idea.

The Solution


Figure 1 – The improvement is obvious. I’m still trying to figure out how to provision 2.25 machines though.

To fix this I spent a few days modifying Resque to use multiple threads when on JRuby. The changes were relatively straightforward; Resque was already thread safe so no brain surgery was required. There were three major changes:

  1. Modify the main processing method so it spawns N threads, each of which run the work loop
  2. Modify the redis connection to use a connection pool so the connection does not become a point of contention with lots of threads
  3. Modify the signal handling so Resque can shutdown gracefully (applications cannot use SIGQUIT in JRuby)

Before my changes they were running 9 machines with a total of 135 processes with 512MB of RAM each for each test run. Subsequent testing has shown that a single JRuby process with 135 threads and 1GB of RAM performs just as quickly so they’ve gone from 68GB to 1GB and needing several machines to just one. Now instead of a large processing farm, they just need a small garden. :-)

You can find my modified Resque project on github.

Posted in Web | 8 Comments