fanout blog

Pushpin makes it easy to create HTTP long-polling and streaming services using any web stack as the backend. It’s compatible with any framework, whether Django, Rails, ASP, or even PHP. Pushpin works as a reverse proxy, sitting in front of your server application and managing all of the open client connections.

Communication between Pushpin and the backend server is done using conventional short-lived HTTP requests and responses. There is also a ZeroMQ interface for advanced users.

The approach is powerful for several reasons:

• The application logic can be written in the most natural way, using existing web frameworks.
• Scaling is easy and also natural. If your bottleneck is the number of recipients you can push realtime updates to, then add more Pushpin instances.
• It’s highly versatile. You define the HTTP exchanges between the client and server. This makes it ideal for building APIs.

(more…)

Pollymer is a general-purpose AJAX library designed for long-polling any protocol. Before you set off to write manual long-polling code again, consider using Pollymer for the task. Features include:

• Automatic retrying of failed requests, with an exponentially increasing delay between retries.
• Automatic re-polling on success, separated by a small randomized delay. Pass a function for the URL to be able to dynamically adjust the polling target.
• Workarounds for browser “busy” indications.
• JSON-P fallback for cross-domain requests on older browsers (requires server support).

(more…)

With all the interest in Websockets lately, it would be easy to write off HTTP long-polling as a less capable, legacy mechanism. The truth is that long-polling is completely sufficient for today’s modern web applications, and additionally it enjoys many benefits not found in Websockets or other mechanisms. Here’s why you might want to implement HTTP long-polling on purpose in the present day:

(more…)

Fanout.io‘s ability to power any realtime HTTP API is based on what I call the proxy-and-hold technique. With this technique, an edge server handling HTTP connections for realtime purposes need not be concerned with the inner workings of the web application that it fronts. This keeps edge nodes dumb and generalized, allowing for straightforward scaling and also the possibility of sharing the nodes across multiple applications and services. To help explain how the technique works and why I believe it is an (the?) ideal way to develop any realtime HTTP application, I’ll walk us through my original thought process:

(more…)

As you may know, the Fanout.io cloud supports delivering data to both HTTP and XMPP clients/endpoints. What you may not know is that Fanout’s own API is accessible via both protocols, too. This means that not only can you publish messages or make configuration changes via REST, you can perform these same tasks by sending stanzas. Why would you want to do this? Well, in certain cases, XMPP may be a more convenient or more optimal way of accessing Fanout’s API. Additionally, the fact that you can use the API via one protocol but publish data to an endpoint of the opposite protocol allows for crossover possibilities. For example, you can use the REST API to fire off XMPP stanzas, or use XMPP ad-hoc commands to stream data to HTTP clients.

(more…)

Realtime Conference 2012 was held in Portland last week. Formerly known as Keeping it Realtime in 2011, this was the conference’s second iteration of what is poised to become a regular annual event. Over the course of two days, speakers from around the world came to talk about web development, browser hacks, scaling, optimizations, standards, and anything else of interest to those furthering (or merely observing) the evolution of the realtime Internet. While last year’s talks were focused primarily on realtime web tech and had two tracks, this year was a single track and boasted a broadened scope. Sure, there were still plenty of talks about Node, but we also got a taste of computer history, power grid greenery, wearable devices, and deejay hackery.

(more…)

Hey people! I’ll be at the Realtime Conference and XMPP Summit in Portland this week. If you want to talk about push, open protocols, federation, etc, just find the guy in the Fanout shirt.

(more…)

As I’ve been a long-time contributor to XMPP, people who know me are generally surprised to discover that Fanout.io is not in some way centrally powered by an XMPP server. XMPP is often touted as a solution for realtime communication among servers in a cluster. While this suggestion is not necessarily unsubstantiated, and many services operate this way today, the reality is that many applications have greater needs when it comes to messaging than your typical XMPP software stack provides. I’m talking about real queuing patterns, such as worker balancing, reliability, flow control, “job” queues, etc.

(more…)

I’m a big fan of ZeroMQ, the peer-to-peer message queuing system. If you’ve never heard of it before, go check out the project’s website. You may need to read the guide and then wait a day or two before it hits you, but it’s the kind of simple, brilliant technology that will eventually make you wonder, “golly, why isn’t everything built this way?” What really sold me on the concept is the Mongrel2 web server. Real-time HTTP becomes a lot easier when you can exchange requests and responses asynchronously over a message queue, and the fact that it’s peer to peer makes the approach practical and easy to scale. Fanout.io uses ZeroMQ for the majority of communication between server nodes and components. While Fanout does support XMPP features, it notably does not use XMPP for internal communication. More on that decision in a future article.

As Fanout needs to handle both inbound HTTP and outbound HTTP, I thought it would be neat to try and write essentially the inverse of Mongrel2. The result is Zurl, an event-driven server that makes HTTP requests. The software has been released as open source, so it is free to download and use on your own machines.

(more…)

When I tell people about Fanout.io, their reaction usually takes one of two forms:

• Sign me up!
• Why would I use an external service for this? Push is easy. I can do it in 20 minutes with <insert modern web framework>.

It is true that there are a bunch of software solutions that make pushing data in realtime easier, and if you’re enthusiastic about maintaining your own servers then a cloud service may not be that interesting. It’s important to recognize, though, that Fanout is about more than just making push easy. It’s about making it scalable.

(more…)