I want to introduce you to my new gem, girl_friday. The problem: current asynchronous processing tools with Ruby are too inefficient and too complex.
It’s sad to admit but commonly with Ruby if you want to process 5 messages at the same time you have to spin up 5 processes, each of which boots your application and loads your code into memory; if each process is 100MB, that’s 500MB, most of which is redundant code.
Threads are the best answer, long term. Threads are hard to get right if you are managing them yourself but they are simpler to use than Fibers and give us real parallelism in JRuby and the upcoming Rubinius 2.0 release. Ruby 1.9’s threading isn’t quite as good but it is still useful for typical IO-heavy, server-side systems. girl_friday uses Actors for safe and simple concurrency on top of Ruby threads. With actors, we get the benefits of threads with fewer drawbacks! Since we are using multiple threads in a single process, the memory overhead is far less than booting another process.
With girl_friday, your queue processing happens in-process with the rest of your application. You don’t need a separate project, deployment, process monitoring and alerts, etc. If your application is running, so is girl_friday.
You define your queues and how to process incoming messages when your application starts:
# config/initializers/girl_friday.rb EMAIL_QUEUE = GirlFriday::WorkQueue.new(:user_email) do |msg| UserMailer.registration_email(msg).deliver end
then just push a message hash with your data onto the queue to be processed:
EMAIL_QUEUE << { :name => @user.name, :email => @user.email }
Dead simple by design.
Each queue in girl_friday is composed of a supervisor actor and a set of worker actors. The supervisor actor is the only one that manages the internal state of a queue. It receives work to perform and hands work to workers as they become available to process more work. If you are interested in the nitty-gritty detail, here’s the WorkQueue class to peruse.
girl_friday has a number of nice options built-in already:
See the girl_friday wiki for more specifics about each of these options.
girl_friday supports Ruby 1.9.2, Rubinius 1.2.3 and JRuby 1.6.0 and above. Moving forward, I’d like to see a web UI added for girl_friday, like Resque’s web UI. If you want to help out, please fork the Github project and send pull requests!