Micromessaging: Connecting Heroku Microservices w/Redis and RabbitMQ

Posted on by in Everything Else

While attempting to deploy a system of interconnected microservices to Heroku recently, I discovered that processes running in dynos in the same application cannot talk to each other via HTTP. I had originally planned on each microservice implementing a “REST” API – but this wasn’t going to be an option if I wanted to stick with Heroku. Much head-scratching ensued.

The solution, it turns out, is to communicate between microservices through a centralized message broker – in my case, a Redis database (but I’ll show you how do it with RabbitMQ as well, free of charge). The design of each microservice API has been decoupled from HTTP entirely: Client/server communication is achieved by enqueueing JSON-RPC 2.0-encoded messages in a list, with BRPOP and return-queues used to emulate HTTP request/response semantics. The Redis database serves as a load balancer of sorts, enabling easy horizontal scaling of each microservice (in Heroku dyno) on an as-needed basis. Redis will ensure that a message is dequeued by only a single consumer, so you can spin up a lot of dynos without worrying that they’ll clobber each other’s work. It’s pretty sa-weet.

So how’d I do it, you ask? Read on!

The Microservice Core

To keep things simple, the server side of our application will consist of a service that calculates the sum or difference between two numbers:

Pretty straightforward. Nothing about transport here at all – just domain logic: adding, subtracting.

JSON-RPC 2.0

Communication between client and server is achieved by sending and receiving JSON-RPC 2.0-encoded messages. These messages include the information required to – you guessed it – affect a method call in a remote system. To give you an idea of what this looks like in the context of our calculator:

JSON-RPC 2.0 Request

JSON-RPC 2.0 Response

Our request-messages have four properties: “jsonrpc”, which is always “2.0” as per the spec; “id”, which is a unique identifier created by the client; “method”, which is the name of the method we intend to call on the server; and “params,” which include values that we intend to pass to the method call. The response-message includes “jsonrpc” and “id” properties in addition to “result” (if the method call was successful) or “error” (if it was not).

Note that the message is transport-agnostic. Seeing where I’m going here? Using JSON-RPC allows us to communicate between components in our system – even when HTTP isn’t an option.

Redis as a Message Broker

As you’ve seen, nothing about our JSON-RPC messages say anything about how they’re transported between client and server; we can send our message however we want (HTTP, SMTP, ABC, BBD, TLC, OPP, etc.). In this case, we’ll implement “sending” a message to the server as an LPUSH on to a Redis list. Let’s bang out a quick client and I’ll explain the interesting parts:

First, we instantiate a Redis client. We use the Redis client to LPUSH our messages to a list that both the API client and server know about (“calc“). After enqueueing a request-message we can use BRPOP to block on receiving a response-message that will be enqueued into a separate return list. This return list’s name will be equal to the id on the JSON-RPC request-message. Once we get a result (encoded as JSON), we simply parse it and write to the console.

Next, let’s build out the server:

The server implementation reduces to a loop in which we block on the receival of an inbound JSON-RPC message in our calc list. When a message is received, we parse it and pass its arguments to the appropriate method on our calculator instance. Using the id on the request, we enqueue our response-message in a return list and resume polling.

The cool thing about this approach is that we can spin up as many Heroku dynos as we want and Redis will do the load balancing for us. Web scale!

Connecting Through RabbitMQ

In case Redis is not for you, we can achieve the same goals by using RabbitMQ as a message broker. The implementation is straightforward and has a similar feel to what we’ve done with Redis:

The only major difference here is that instead of a never-ending while loop, the call to subscribe is passed a block: true. This will cause the calling thread to block and will prevent the program from exiting until we interrupt it.

Now, for our client:

If You Come Crawling Back to HTTP…

If you decide to migrate to a new platform that allows you to use HTTP, you can do so with low impact to your codebase. We’ll use Sinatra to handle HTTP requests, parsing the request body and marshalling the necessary bits to our calculator:

Clients can now communicate with our calculator by issuing HTTP POST requests to our “/calc” endpoint:

Cleaning Up the Cruft

Our client code is easy to understand, but a bit verbose. Let’s reduce some of the boilerplate by introducing a new class, using method_missing to remote calls to your API.

Now we have a reusable Redis-enabled API client whose interface hides the details of serializing hashes to JSON and other boring stuff. Something similar could be done on the server side, deserializing JSON to a method name and args to pass to the calculator instance.

In Summary

Occasionally, platforms prevent us from using transport technologies that we’re familiar with – HTTP, in this case – and we’re stuck investigating new ways of linking things together. In this tutorial I’ve shown you a few ways to connect the pieces of your system through a centralized message broker. By decoupling our API design from any one particular transport, we’ve achieved a flexibility unattainable by traditional “REST” APIs, unlocking the ability to horizontally scale our microservices across Heroku dynos with ease.

Notes

  1. I’m referring to a “Level Two” implementation as per the Richardson Maturity Model.

Feedback

  Comments: 25


  1. Nice stuff. I wonder if you looked into speed/load testing this stuff? I recall, from back in my Java days, building an xml-rpc service to query a in-house search engine and we found that we were able to serve far more requests than we could using the engine’s built in REST interface.

    • Erin Swenson-Healey


      Hey Mr Rogers,

      I did a quick comparison of HTTP vs. Redis vs. RabbitMQ using Benchmark and the results were pretty interesting:

      Going through HTTP (including Sinatra), I was able to complete 1000 RPC request/response cycles in ~2.43 seconds. With Redis, I was able to complete 1000 RPC request/response cycles in ~0.42 seconds (a 16x speed improvement). RabbitMQ came in at 1000 RPC cycles in 1.60 seconds. Link to Benchmark output here:

      https://gist.github.com/laser/11377229

      …and code run for comparison here:

      https://gist.github.com/11377327.git

      • Carl Hörberg


        The reason for the lower RabbitMQ result is that you’re canceling the consumer and then opens a new subscribe again. In a normal use case this is probably neglectable time, but in a tight benchmark loop it looks bad.

        • Erin Swenson-Healey


          Hey Carl,

          Thanks for the tip. What would be a better way to write the RabbitMQ portion of the benchmarking code such that I was doing an apples-to-apples comparison with Redis?

          E

  2. Brendon Murphy


    Dig this article, shared it with some co-workers.

    A nice robustness enhancement to the redis implementation could be the brpoplpush command (http://redis.io/commands/brpoplpush ). There’s a simple queue implementation Ost (https://github.com/soveran/ost/blob/master/lib/ost.rb#L31) that uses it in a clever fashion for failures.

    Basically you put requests on a temp list per worker. If disaster strikes between the time the worker pops the request and has a chance to reply, you still have the request hanging around in the ‘backup’ list for that worker for inspection, recovery, etc.

    • Erin Swenson-Healey


      Hey Brendon Murphy,

      Thanks for the tip. These implementations are not bombproof; rather, I’m trying to communicate (in as little code as possible) the flexibility that a developer affords herself when building APIs whose design has been decoupled from any single transport mechanism. HTTP, SMTP, Redis, IronMQ – whatever. They’re just implementation details.

  3. stephennguyen


    Have you had a chance to benchmark IronMQ as apart of this test of microservice messaging? The backend servers were built for speed, efficiency, reliability, and has some great features using golang as our foundation.

    • Erin Swenson-Healey


      Hey stephennguyen,

      I have not tried this with IronMQ. Conceptually, I presume that an implementation using IronMQ would look similar to the Redis and RabbitMQ examples. If you were to provide an example, I’d include it in the article.


  4. Check out: https://github.com/rack-amqp/jackalope
    I saw it presented at RailsConf, and basically does what you want, and more.

    • Erin Swenson-Healey


      Hey findchris,

      I’m having a difficult time understanding how jackelope relates to the article. Perhaps you could elaborate?


      • It seems like a kindred project.

        – “The solution, it turns out, is to communicate between microservices through a centralized message” – check
        – “emulate HTTP request/response semantics” – check
        – “The design of each microservice API has been decoupled from HTTP entirely” – check

        I feel like your motivations align with that of the rack-amqp/jackalope projects.

        • Erin Swenson-Healey


          Gotcha, gotcha. I did some digging into the jackelope source; makes sense to me now what they’re trying to do. Excellent slides (that you linked to), btw.


    • @foundchris:disqus it looks like this code basically allows you to swap amqp in as a replacement for http. Just curious, have you integrated this into a system? I’m curious why you would want to implement it

  5. Christopher Dell


    Great article, and very timely for me. Much <3<3<3<3.

    I've wrapped the client/server implementation for RabbitMQ into a small gem call `burrow` that you can have a look at here https://github.com/tigrish/burrow. This means not having to repeat connection info and so on for every app.

    Again, thanks for the article!

    • Erin Swenson-Healey


      Hey Christopher Dell,

      No problemo. I’m glad you enjoyed the post. Thanks for the link to your burrow lib.

      Erin


  6. I’m a bit confused by the statement that you can’t communicate between dynos. Certainly you can’t communicate between distinct dyno processes, but I’m not sure why you’d want to do that. Wouldn’t you put each microservice application behind its own domain name and communicate via http/rest on those end points?

    • Erin Swenson-Healey


      Hey Tammer Saleh,

      There are a few motivations for this approach:

      1. Since I won’t be exposing those services to the Internet (they’re running as instances of non-‘web’ dynos), I won’t have to think much about authentication. If I ever did want to expose them to the outside world, I could centralize all authentication in an HTTP gateway bound to $PORT, which could then bridge incoming HTTP requests to the Redis/RabbitMQ/whatever back end.

      2. If possible, I’d like to avoid a world in which I’d have to incur the mental overhead of deploying the services as separate Heroku applications. If each API were to be available via HTTP, each service would need to be running as an instance of distinct Heroku application’s ‘web’ dyno. I’d prefer to deploy the entire application in a single shot.

      3. Even if HTTP were an option for sending RPC messages between the UI portion of my application (bound to $PORT) and each service – I’d still go with a message queue as an intermediary; the speed difference is huge.

      • Tammer Saleh


        I get #1, and it’d be great if Heroku offered an “internal-only” app (or the equivalent of AWS security groups.

        I personally disagree on #2. One of the benefits of a microservice architecture is that you can decouple the development and deployment of each bit of the overall application. It also facilitates scaling each microservice independently, where in this approach, you must run each microservice process inside all of your dynos, using up resources.

        #3 is a great point, but I still feel that this architecture is overall pretty confusing when compared to just http communication.

        But it’s a great article – thanks for posting it!

        • Erin Swenson-Healey


          Hey Tammer Saleh,

          Regarding scaling: What I’ve been doing is running each service as, essentially, an instance of a worker dyno. That is to say, if I have a UI (the ‘web’ dyno) backed by three services: contact service, user service, reporting service – then I’d have a Procfile that’d look something like:

          # Procfile
          web: bundle exec rails s -p $PORT
          user_svc: sh -c ‘cd services/user; ./service_runner.rb’
          contact_svc: sh -c ‘cd services/contact; ./service_runner.rb’
          reporting_svc: sh -c ‘cd services/reporting; ./service_runner.rb’

          This allows me to horizontally scale each service independently; it’s just a simple matter of adding more dynos. Beefing up my persistence layer can be handled independently from the services themselves.

Your feedback