I’m a big fan of ZeroMQ, the peer-to-peer message queuing system. If you’ve never heard of it before, go check out the project’s website. You may need to read the guide and then wait a day or two before it hits you, but it’s the kind of simple, brilliant technology that will eventually make you wonder, “golly, why isn’t everything built this way?” What really sold me on the concept is the Mongrel2 web server. Real-time HTTP becomes a lot easier when you can exchange requests and responses asynchronously over a message queue, and the fact that it’s peer to peer makes the approach practical and easy to scale. Fanout Cloud uses ZeroMQ for the majority of communication between server nodes and components. While Fanout Cloud does support XMPP features, it notably does not use XMPP for internal communication. More on that decision in a future article.

As Fanout needs to handle both inbound HTTP and outbound HTTP, I thought it would be neat to try and write essentially the inverse of Mongrel2. The result is Zurl, an event-driven server that makes HTTP requests. The software has been released as open source, so it is free to download and use on your own machines.

How it works

You use a PUSH socket to submit HTTP requests and a SUB socket to retrieve HTTP responses. Like Mongrel2, the use of pub/sub as the response mechanism means that multiple workers could conceivably handle a single HTTP response. To quote the Mongrel2 manual, “not exactly sure what you could write with that, but it’s probably something really damn cool.” It additionally supports a REQ interface, so that for simple cases you can just use a single socket.

Zurl has a number of features that make it stand out. Most importantly, it is event-driven, which means it can handle a thousand simultaneous outbound HTTP requests without needing a thousand threads. A special guest appearance is made by the JDNS library, originally developed for Psi, as it handles pure event-driven DNS resolution. Zurl supports HTTPS, streaming uploads/downloads, and fire-and-forget usage. The message protocol uses TNetStrings, which is also used by Mongrel2. The format is useful as a simple alternative to JSON without having to resort to binary. Zurl does one thing and does it well, and it should be easy to integrate it into existing projects.

What can I do with it?

Whatever you dream up. One thing Fanout Cloud uses it for is calling webhooks. When you publish data to a channel of URL subscribers using the Fanout Cloud API, the data is delegated to a cluster of Zurl servers that fire out the requests quickly and efficiently. For example, if today you use a blocking HTTP method to call a webhook, you can easily replace this with a few lines of code that sends a message to Zurl instead. This gives you the power of an async request without necessarily having to write your application in an async way. Fanout also uses Zurl for proxying incoming HTTP requests to remote servers. The bidirectional streaming support allows large request and response payloads to pass through the system without needing to buffer them entirely in memory before relaying.

This is Fanout’s first release of open source software. We hope you enjoy it. If you are interested in non-open-source licensing of Zurl, please contact us.