Improving Resque’s memory efficiency

Posted on by in Web

Resque is a very popular message queueing system for Rails applications.  Here’s how I recently improved the memory efficiency of a Carbon Five customer’s resque processing farm by 68x!

The Problem

This customer has an existing investment in Resque and is a heavy user of a third-party Java API so they need to run their resque workers using JRuby.  Unfortunately this gets them the worst of all worlds:

  • JRuby does not fork so they don’t get the benefit of memory isolation by working in a child process
  • The JVM is relatively demanding in terms of memory
  • Resque is single threaded

To scale to the levels they wanted to get to, they projected that they would need to run hundreds of Resque processes, each consuming 512MB.  Insanity! If your problem requires a lot of concurrency to solve, using lots of large processes is a terrible idea.

The Solution


Figure 1 – The improvement is obvious. I’m still trying to figure out how to provision 2.25 machines though.

To fix this I spent a few days modifying Resque to use multiple threads when on JRuby. The changes were relatively straightforward; Resque was already thread safe so no brain surgery was required. There were three major changes:

  1. Modify the main processing method so it spawns N threads, each of which run the work loop
  2. Modify the redis connection to use a connection pool so the connection does not become a point of contention with lots of threads
  3. Modify the signal handling so Resque can shutdown gracefully (applications cannot use SIGQUIT in JRuby)

Before my changes they were running 9 machines with a total of 135 processes with 512MB of RAM each for each test run. Subsequent testing has shown that a single JRuby process with 135 threads and 1GB of RAM performs just as quickly so they’ve gone from 68GB to 1GB and needing several machines to just one. Now instead of a large processing farm, they just need a small garden. :-)

You can find my modified Resque project on github.


Feedback

  Comments: 13


  1. Nice!

    Since you are already using JRuby, have you considered using Torquebox (which uses JBoss HornetMQ for messaging)? I’ve been using Torquebox in production since they first released a stable version.


  2. I wrote an implementation of Resque in Java some months ago. Might be worth going pure Java if they’re that tied to that 3rd party library.

    https://github.com/gresrun/jesque


  3. Nice work. Another implementation I planned to do: Resque on EventMachine, using a pool of fibers, instead of threads. The majority of my workers do remote calls, and I see a lot of time lost there. I still have to ensure that it would not be a nightmare to understand/manage in resque-web.


  4. Threads worry me. For a greenfield project, I’d consider breaking the app into smaller subcomponents so you could have different workers each using only the memory they need to run the parts of the app actually being used for that job, instead of the entire Rails environment


  5. Very nice! Have you thought doing a pull request to get these changes incorporated upstream? I assume they’d have to isolated to JRuby, but surely that’s possible.


  6. Man I just migrated a Resque project from JRuby to MRI since Resque is works so poorly on JRuby by default. Wish I had seen this sooner.

Your feedback