Resque is a very popular message queueing system for Rails applications. Here’s how I recently improved the memory efficiency of a Carbon Five customer’s resque processing farm by 68x!
This customer has an existing investment in Resque and is a heavy user of a third-party Java API so they need to run their resque workers using JRuby. Unfortunately this gets them the worst of all worlds:
- JRuby does not fork so they don’t get the benefit of memory isolation by working in a child process
- The JVM is relatively demanding in terms of memory
- Resque is single threaded
To scale to the levels they wanted to get to, they projected that they would need to run hundreds of Resque processes, each consuming 512MB. Insanity! If your problem requires a lot of concurrency to solve, using lots of large processes is a terrible idea.
To fix this I spent a few days modifying Resque to use multiple threads when on JRuby. The changes were relatively straightforward; Resque was already thread safe so no brain surgery was required. There were three major changes:
- Modify the main processing method so it spawns N threads, each of which run the work loop
- Modify the redis connection to use a connection pool so the connection does not become a point of contention with lots of threads
- Modify the signal handling so Resque can shutdown gracefully (applications cannot use SIGQUIT in JRuby)
Before my changes they were running 9 machines with a total of 135 processes with 512MB of RAM each for each test run. Subsequent testing has shown that a single JRuby process with 135 threads and 1GB of RAM performs just as quickly so they’ve gone from 68GB to 1GB and needing several machines to just one. Now instead of a large processing farm, they just need a small garden. 🙂
You can find my modified Resque project on github.