subscribe via RSS
A note from our founder, Justin.
Holy cow, it’s been five years. Five years ago today, Fanout was registered as a company. Back then, all we had were some scribbles in a text file. These days we have a production-ready product and many paying customers. I want to give a big thanks to our investors, advisors, mentors, early adopters, friends, and everyone else who helped get us here.
I also want to take this moment to share some of our experiences over the years.
Are you all about realtime apps? Well, now you can be famous and win $500 to teach the Internet. Fanout is excited to team up with the hack.guides() 2016 Tutorial Contest. Any realtime app submitted to the contest that uses Pushpin or Fanout Cloud will qualify for a chance to win a $500 prize.
See the contest here: http://tutorials.pluralsight.com/contest
The blog content is all the same, and Livefyre commenting is still here as well as RSS/Atom feeds and the SubToMe buttons. The reason for the migration (and thus the differences) are mostly internal:
RethinkDB is a modern NoSQL database that makes it easy to build realtime web services. One of its standout features is called Changefeeds. Applications can query tables for ongoing changes, and RethinkDB will push any changes to applications as they happen. The Changefeeds feature is interesting for many reasons:
- You don’t need a separate message queue to wake up workers that operate on new data.
- Database writes made from anywhere will propagate out as changes. Use the RethinkDB dashboard to muck with data? Run a migration script? Listeners will hear about it.
- Filtering/squashing of change events within RethinkDB. In many cases it may be easier to filter events using ReQL than using a message queue and filtering workers.
This makes RethinkDB a compelling part of a realtime web service stack. In this article, we’ll describe how to use RethinkDB to implement a leaderboard API with realtime updates. Emphasis on API. Unlike other leaderboard examples you may have seen elsewhere, the focus here will be to create a clean API definition and use RethinkDB as part of the implementation. If you’re not sure what it means for an API to have realtime capabilities, check out this guide.
We’ll use the following components to build the leaderboard API:
- Database: RethinkDB, hosted on a Rackspace server.
- Web service: Django, hosted by Heroku.
- Realtime push to clients: Pushpin, hosted by Fanout Cloud.
Since the server app targets Heroku, we’ll be using environment variables for configuration and foreman for local testing.
Read on to see how it’s done. You can also look at the source.
Bayeux is one of the few standard protocols for publish-subscribe messaging on the web. Today we announce Bayeux compatibility in the Fanout Cloud realtime push service. When you publish JSON objects through our service, they can be received by any Bayeux-compatible client library, such as Faye.
Do you have a microservice contingency plan?
How many APIs does your company consume? How many microservices do you depend on? This brave new world of public APIs and microservices is great, but what happens when it’s not?
Mongrel2 is a fast and simple HTTP & WebSocket server that communicates to backend workers via ZeroMQ. It does one thing and does it very well, making it an ideal part of a componentized architecture. The code is event-driven, allowing it to support thousands of concurrent connections and also asynchronous behaviors. These properties are especially important to realtime applications.
Fanout has been one of the most active contributors to the Mongrel2 project over the past year, adding features such as TLS SNI and improved streaming capability. We’ve also been working on making the server easier for people to get started with. And with that, we are proud to announce official packages for Debian and Ubuntu!
Earlier this week, a bookmarklet was released that enlivens Hacker News comment threads. This thing makes me feel like I have magic powers. You have no idea how often I would refresh pages on HN before I started using this, especially on posts that are my own or that I had commented on. Now I can simply leave pages open in tabs and go about my day. The page titles update whenever there are new comments.
At Fanout, we’ve developed a powerful publish-subscribe system for fronting custom APIs. Sometimes, though, API design is too much to think about and all you want to do is push some JSON.
To address this need, we initially created our own proprietary JSON-publishing protocol (a.k.a. “FPP”) and corresponding client library. This made it possible for developers to implement realtime updates in just a few lines of code, transported by magic. While this system worked well enough, it wasn’t terribly satisfying to have invented yet-another-protocol in order to accomplish this. Sure, every other pubsub cloud service has done the same thing, but the status quo makes for a bunch of redundant efforts that all achieve more-or-less the same result, and it hampers interoperability. So, lately, we’ve been looking into how we could adapt an existing standard for pushing JSON.
You know how it’s hard to notice your own typos? That’s how it felt to finally implement the latest round of usability fixes for Fanout Cloud. The service has always meant to be easy to use, and for the longest time I didn’t think we needed to change anything. Despite the fact that we’d sometimes receive feedback to the contrary, or that we’d sometimes observe confusion (judging by the number of incomplete accounts), I insisted that the service was as easy to use as it possibly could be. If there was any difficulty, it was of the necessary kind.
UPDATE: The official GRIP specification now lives here and is more up to date.
Fanout’s ability to power any realtime HTTP API is based on what I call the proxy-and-hold technique. With this technique, an edge server handling HTTP or WebSocket connections for realtime purposes need not be concerned with the inner workings of the web application that it fronts. This keeps edge nodes dumb and generalized, allowing for straightforward scaling and also the possibility of sharing the nodes across multiple applications and services. To help explain how the technique works and why I believe it is an ideal way to develop any realtime HTTP application, I’ll walk us through my original thought process:
As you may know, Fanout Cloud supports delivering data to both HTTP and XMPP clients/endpoints. What you may not know is that the Fanout Cloud API itself is accessible via both protocols, too. This means that not only can you publish messages or make configuration changes via REST, you can perform these same tasks by sending XMPP stanzas. Why would you want to do this? Well, in certain cases, XMPP may be a more convenient or more optimal way of accessing the Fanout Cloud API. Additionally, the fact that you can use the API via one protocol but publish data to an endpoint of the opposite protocol allows for crossover possibilities. For example, you can use the REST API to fire off XMPP stanzas, or use XMPP ad-hoc commands to stream data to HTTP clients.
When I tell developers about Fanout Cloud, their reaction usually takes one of two forms:
- Sign me up!
- Why would I use an external service for this? Push is easy. I can do it in 20 minutes with <insert modern web framework>.
It is true that there are a bunch of software solutions that make pushing data in realtime easier, and if you’re enthusiastic about maintaining your own servers then a cloud service may not be that interesting. It’s important to recognize, though, that Fanout Cloud is about more than just making push easy. It’s about making it scalable.
Push is hard. Often, implementing realtime push in a website or web service means having to use unfamiliar software or frameworks. If it isn’t an XMPP server with BOSH, then it’s an esoteric HTTP server framework like Node.js or Tornado. Further, while the knowledge and tools necessary for load balancing a traditional web service are commonly found these days, scaling out and operating a push architecture is less straightforward. The interactions between server nodes are different. The traffic patterns are different. Even if you’ve got a handle on whatever particular approach for push you’re using today, do you know how you’ll scale it tomorrow?
Here we’ll discuss Fanout’s various products (such as Fanout Cloud), as well as topics such as realtime, push, distributed systems, synchronization, federation, and open protocols.