Improving Resque’s memory efficiency

Resque is a very popular message queueing system for Rails applications.  Here’s how I recently improved the memory efficiency of a Carbon Five customer’s resque processing farm by 68x!

The Problem

This customer has an existing investment in Resque and is a heavy user of a third-party Java API so they need to run their resque workers using JRuby.  Unfortunately this gets them the worst of all worlds:

  • JRuby does not fork so they don’t get the benefit of memory isolation by working in a child process
  • The JVM is relatively demanding in terms of memory
  • Resque is single threaded

To scale to the levels they wanted to get to, they projected that they would need to run hundreds of Resque processes, each consuming 512MB.  Insanity! If your problem requires a lot of concurrency to solve, using lots of large processes is a terrible idea.

The Solution


Figure 1 – The improvement is obvious. I’m still trying to figure out how to provision 2.25 machines though.

To fix this I spent a few days modifying Resque to use multiple threads when on JRuby. The changes were relatively straightforward; Resque was already thread safe so no brain surgery was required. There were three major changes:

  1. Modify the main processing method so it spawns N threads, each of which run the work loop
  2. Modify the redis connection to use a connection pool so the connection does not become a point of contention with lots of threads
  3. Modify the signal handling so Resque can shutdown gracefully (applications cannot use SIGQUIT in JRuby)

Before my changes they were running 9 machines with a total of 135 processes with 512MB of RAM each for each test run. Subsequent testing has shown that a single JRuby process with 135 threads and 1GB of RAM performs just as quickly so they’ve gone from 68GB to 1GB and needing several machines to just one. Now instead of a large processing farm, they just need a small garden. :-)

You can find my modified Resque project on github.

This entry was posted in Web. Bookmark the permalink.

13 Responses to Improving Resque’s memory efficiency

  1. Rubem Azenha says:

    Nice!

    Since you are already using JRuby, have you considered using Torquebox (which uses JBoss HornetMQ for messaging)? I’ve been using Torquebox in production since they first released a stable version.

  2. Greg Haines says:

    I wrote an implementation of Resque in Java some months ago. Might be worth going pure Java if they’re that tied to that 3rd party library.

    https://github.com/gresrun/jesque

  3. Nice work. Another implementation I planned to do: Resque on EventMachine, using a pool of fibers, instead of threads. The majority of my workers do remote calls, and I see a lot of time lost there. I still have to ensure that it would not be a nightmare to understand/manage in resque-web.

  4. themgt says:

    Threads worry me. For a greenfield project, I’d consider breaking the app into smaller subcomponents so you could have different workers each using only the memory they need to run the parts of the app actually being used for that job, instead of the entire Rails environment

  5. Pingback: Ruby一日一技 #8 | Ruby迷

  6. Wes Morgan says:

    Very nice! Have you thought doing a pull request to get these changes incorporated upstream? I assume they’d have to isolated to JRuby, but surely that’s possible.

  7. Pingback: Kick Resque with Sidekiq (Lightweight Queue System for Rails) | Active GIT

  8. martijnstorck says:

    Man I just migrated a Resque project from JRuby to MRI since Resque is works so poorly on JRuby by default. Wish I had seen this sooner.

  9. Pingback: Use sidekiq to make resque clusters run happier | Raw thoughts from Alex Dong

  10. Robert says:

    Nice job!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>