Django is an awesome framework for building web services using the Python language. However, it is not well-suited for handling long-lived connections, which are needed for realtime data push. In this article we’ll explain how to pair Django with Fanout to reach realtime nirvana.
Do you have a microservice contingency plan?
How many APIs does your company consume? How many microservices do you depend on? This brave new world of public APIs and microservices is great, but what happens when it’s not?
What happens when a service raises their prices, changes their metering, dies (bankruptcy, acquisition, sunset), pivots, experiences a major outage, or any number of problems facing modern technology companies? These are cross-team problems affecting ops, engineering, and business teams. How will each handle the problem?
Think about how your company consumes these APIs and plan for the worst.
There is no one right answer, but here are some things that should be explored when considering a microservice.
- Do they offer an open-source version that you could have ready to go if necessary?
- Will they place code in escrow in the event that they go under? You have a better chance at making this work if you’re a big player, but it’s worth bringing up.
- How interoperable are they with the competition? Think Amazon vs Rackspace. Switching from one to another isn’t exactly fun, but it is possible to migrate without breaking much.
- Have plans in place to “roll your own”. Writing code for this might be taking it too far (that’s the reason you’re consuming someone else’s API, right?), but at least have an idea of how you could accomplish the same features with your in-house team and a rough timeline.
So do you have a contingency plan? We’d love to hear your stories (good, bad, or otherwise) on the topic!
Mongrel2 is a fast and simple HTTP & WebSocket server that communicates to backend workers via ZeroMQ. It does one thing and does it very well, making it an ideal part of a componentized architecture. The code is event-driven, allowing it to support thousands of concurrent connections and also asynchronous behaviors. These properties are especially important to realtime applications.
Fanout has been one of the most active contributors to the Mongrel2 project over the past year, adding features such as TLS SNI and improved streaming capability. We’ve also been working on making the server easier for people to get started with. And with that, we are proud to announce official packages for Debian and Ubuntu!
Before I begin, I want to say that WebSockets are great. I’ve even implemented RFC 6455 myself in Zurl and Pushpin, which are used by the Fanout.io service. Fanout.io also supports WebSockets via its XMPP-FTW interface using Primus.
However, after spending quite some time working on large distributed applications and gaining a greater appreciation of REST and messaging patterns, I feel that much of what typical web applications want to accomplish with WebSockets (or with socket-like abstractions) is perhaps better solved by other means.
Zurl is a gateway that converts between ZeroMQ messages and outbound HTTP requests or WebSocket connections. It gives you powerful access to these protocols from within a message-oriented architecture. The following diagram shows how Zurl may fit in among the entities involved:
Any number of workers can contact any number of HTTP or WebSocket servers, and Zurl will perform conversions to/from ZeroMQ messages as necessary. It uses libcurl under the hood.
The format of the messages exchanged between workers and Zurl is described by the ZHTTP protocol. ZHTTP is an abstraction of HTTP using JSON-formatted or TNetString-formatted messages. The protocol makes it easy to work with HTTP at a high level, without needing to worry about details such as persistent connections or chunking. Zurl takes care of those things for you when gatewaying to the servers.
Earlier this week, a bookmarklet was released that enlivens Hacker News comment threads. This thing makes me feel like I have magic powers. You have no idea how often I would refresh pages on HN before I started using this, especially on posts that are my own or that I had commented on. Now I can simply leave pages open in tabs and go about my day. The page titles update whenever there are new comments.
HTTP callbacks (aka Webhooks) are great for sending notifications to remote servers in realtime. In most setups, the URLs to contact are provided by foreign entities. All your application needs to do is allow such URLs to be registered, and then hit them whenever interesting things happen. Easy enough, right?
Not so fast. What if someone provides a URL such as “http://localhost:10000/destructive-command/” and you’ve got an internal web service running on that port? Under normal circumstances, you might not expect this service to be accessible from the outside. Perhaps you have a firewall, or perhaps the internal service binds explicitly to the localhost interface. Either way, the HTTP callback pattern provides attackers an avenue to access this service from within your internal network, bypassing these kinds of expected security measures.
Earlier this year we announced the open source Pushpin project, a server component that makes it easy to scale out realtime HTTP APIs. Just what kind of scale are we talking about though? To demonstrate, we put together some code that pushes a truckload of data through a cluster of Pushpin instances. Here’s the output of its dashboard after a successful run:
Before getting into the details of how we did this, let’s first establish some goals:
- We want to scale an arbitrary realtime API. This API, from the perspective of a connecting client, shouldn’t need to be in any way specific to the components we are using to scale it.
- Ideally, we want to scale out the number of delivery servers but not the number of application servers. That is, we should be able to massively amplify the output of a modest realtime source.
- We want to push to all recipients simultaneously and we want the deliveries to complete in about 1 second. We’ll shoot for 100,000 recipients.
To be clear, sending data to 100K clients in the same instant is a huge level of traffic. Disqus recently posted that they serve 45K requests per second. If, using some very rough math, we say that a realtime push is about as heavy as half of a request, then our demonstration requires the same bandwidth as the entire Disqus network, if only for one second. This is in contrast to benchmarks that measure “connected” clients, such as the Tigase XMPP server’s 500K single-machine benchmark, where the clients participate conservatively over an extended period of time. Benchmarks like these are impressive in their own right, just be aware that they are a different kind of demonstration.
At Fanout, we’ve developed a powerful publish-subscribe system for fronting custom APIs. Sometimes, though, API design is too much to think about and all you want to do is push some JSON.
To address this need, we initially created our own proprietary JSON-publishing protocol (a.k.a. “FPP”) and corresponding client library. This made it possible for developers to implement realtime updates in just a few lines of code, transported by magic. While this system worked well enough, it wasn’t terribly satisfying to have invented yet-another-protocol in order to accomplish this. Sure, every other pubsub cloud service has done the same thing, but the status quo makes for a bunch of redundant efforts that all achieve more-or-less the same result, and it hampers interoperability. So, lately, we’ve been looking into how we could adapt an existing standard for pushing JSON.