Asynchronous Processing with girl_friday

Posted on by in Development

I want to introduce you to my new gem, girl_friday. The problem: current asynchronous processing tools with Ruby are too inefficient and too complex.

Efficiency

It’s sad to admit but commonly with Ruby if you want to process 5 messages at the same time you have to spin up 5 processes, each of which boots your application and loads your code into memory; if each process is 100MB, that’s 500MB, most of which is redundant code.

Threads are the best answer, long term. Threads are hard to get right if you are managing them yourself but they are simpler to use than Fibers and give us real parallelism in JRuby and the upcoming Rubinius 2.0 release. Ruby 1.9’s threading isn’t quite as good but it is still useful for typical IO-heavy, server-side systems. girl_friday uses Actors for safe and simple concurrency on top of Ruby threads. With actors, we get the benefits of threads with fewer drawbacks! Since we are using multiple threads in a single process, the memory overhead is far less than booting another process.

Complexity

With girl_friday, your queue processing happens in-process with the rest of your application. You don’t need a separate project, deployment, process monitoring and alerts, etc. If your application is running, so is girl_friday.

Usage

You define your queues and how to process incoming messages when your application starts:

    # config/initializers/girl_friday.rb
    EMAIL_QUEUE = GirlFriday::WorkQueue.new(:user_email) do |msg|
      UserMailer.registration_email(msg).deliver
    end

then just push a message hash with your data onto the queue to be processed:

    EMAIL_QUEUE << { :name => @user.name, :email => @user.email }

Dead simple by design.

Design

Each queue in girl_friday is composed of a supervisor actor and a set of worker actors. The supervisor actor is the only one that manages the internal state of a queue. It receives work to perform and hands work to workers as they become available to process more work. If you are interested in the nitty-gritty detail, here’s the WorkQueue class to peruse.

Advanced Options

girl_friday has a number of nice options built-in already:

  • Send worker errors to Hoptoad Notifier or a custom error processor
  • Persist jobs to Redis so a restart does not lose queued jobs
  • Asynchronous callbacks (call a block with the result when the message is processed)
  • Runtime metrics for monitoring
  • Clean shutdown (stop processing new jobs)

See the girl_friday wiki for more specifics about each of these options.

Caveats and the Future

girl_friday supports Ruby 1.9.2, Rubinius 1.2.3 and JRuby 1.6.0 and above. Moving forward, I’d like to see a web UI added for girl_friday, like Resque’s web UI. If you want to help out, please fork the Github project and send pull requests!