Resque is a very popular message queueing system for Rails applications. Here’s how I recently improved the memory efficiency of a Carbon Five customer’s resque processing farm by 68x!
The Problem
This customer has an existing investment in Resque and is a heavy user of a third-party Java API so they need to run their resque workers using JRuby. Unfortunately this gets them the worst of all worlds:
To scale to the levels they wanted to get to, they projected that they would need to run hundreds of Resque processes, each consuming 512MB. Insanity! If your problem requires a lot of concurrency to solve, using lots of large processes is a terrible idea.
The Solution
Figure 1 – The improvement is obvious. I’m still trying to figure out how to provision 2.25 machines though.
To fix this I spent a few days modifying Resque to use multiple threads when on JRuby. The changes were relatively straightforward; Resque was already thread safe so no brain surgery was required. There were three major changes:
Before my changes they were running 9 machines with a total of 135 processes with 512MB of RAM each for each test run. Subsequent testing has shown that a single JRuby process with 135 threads and 1GB of RAM performs just as quickly so they’ve gone from 68GB to 1GB and needing several machines to just one. Now instead of a large processing farm, they just need a small garden. :-)
You can find my modified Resque project on github.