fanout blog

Wafting bits in your general direction

Pushpin and RethinkDB meetup video

Posted by justin on October 21st, 2015 — Filed under API, Pushpin, Python

A few months ago I presented at the RethinkDB meetup on the topic of building realtime APIs. Fortunately the talk was recorded and the video is below. Enjoy!

Runscope JWT authentication

Posted by justin on July 29th, 2015 — Filed under API, Security

Monday evening we had a particularly nasty outage: JWT authentication was broken, preventing anyone from using our HTTP API to publish data. The reason we didn’t catch this early on is because our manual test scripts turned out to be broken (reporting auth success when auth had failed.. yeesh!), and there was no authentication coverage in our external monitoring to fall back on.

In a perfect world, our external monitoring would test authentication. I’m happy to report that we are now doing this with Runscope! Getting this to work right was a little tricky since we use JWT, but it was made possible thanks to Runscope’s scripting feature.


Realtime API management with Pushpin and Kong

Posted by justin on July 14th, 2015 — Filed under API, GRIP, Pushpin, WebSockets

Pushpin is the open source reverse proxy for the realtime web. One of the benefits of Pushpin functioning as a proxy is that it can be combined with an API management system, such as Mashape’s Kong. Kong is the open source management layer for APIs. To use Kong with Pushpin, simply chain the two together on the same network path.


Why would you want to use an API management system with Pushpin? Realtime web services have many of the same concerns as request/response web services, and it can be helpful to centrally manage those aspects.


Building a realtime API with RethinkDB

Posted by justin on May 20th, 2015 — Filed under API, Fanout, Pushpin, Python

RethinkDB is a modern NoSQL database that makes it easy to build realtime web services. One of its standout features is called Changefeeds. Applications can query tables for ongoing changes, and RethinkDB will push any changes to applications as they happen. The Changefeeds feature is interesting for many reasons:

  • You don’t need a separate message queue to wake up workers that operate on new data.
  • Database writes made from anywhere will propagate out as changes. Use the RethinkDB dashboard to muck with data? Run a migration script? Listeners will hear about it.
  • Filtering/squashing of change events within RethinkDB. In many cases it may be easier to filter events using ReQL than using a message queue and filtering workers.

This makes RethinkDB a compelling part of a realtime web service stack. In this article, we’ll describe how to use RethinkDB to implement a leaderboard API with realtime updates. Emphasis on API. Unlike other leaderboard examples you may have seen elsewhere, the focus here will be to create a clean API definition and use RethinkDB as part of the implementation. If you’re not sure what it means for an API to have realtime capabilities, check out this guide.

We’ll use the following components to build the leaderboard API:

Since the server app targets Heroku, we’ll be using environment variables for configuration and foreman for local testing.

Read on to see how it’s done. You can also look at the source.


Realtime API design guide

Posted by justin on April 2nd, 2015 — Filed under API, API Design, Webhooks, WebSockets

New to the subject of realtime APIs? This article is the place to start! We’ll discuss the most common design approaches and their pros/cons, as well as link to the documentation of 16 public realtime APIs that you can use for inspiration.


Stateless WebSockets with Express and Pushpin

Posted by justin on March 9th, 2015 — Filed under GRIP, Pushpin, WebSockets

One of the most interesting features of the Pushpin proxy is its ability to gateway between WebSocket clients and plain HTTP backend servers. In this article, we’ll demonstrate how to build a WebSocket service using Express as the HTTP backend behind Pushpin.


Bayeux / Faye compatibility

Posted by justin on January 21st, 2015 — Filed under Fanout, Protocols

Bayeux is one of the few standard protocols for publish-subscribe messaging on the web. Today we announce Bayeux compatibility in the realtime push service. When you publish JSON objects through our service, they can be received by any Bayeux-compatible client library, such as Faye.


Building a realtime API in Django

Posted by justin on November 6th, 2014 — Filed under Fanout, GRIP, Python, WebSockets

Django is an awesome framework for building web services using the Python language. However, it is not well-suited for handling long-lived connections, which are needed for realtime data push. In this article we’ll explain how to pair Django with Fanout to reach realtime nirvana.


Do you have a contingency plan?

Posted by Chris Stewart on October 24th, 2014 — Filed under Fanout

Do you have a microservice contingency plan?


How many APIs does your company consume? How many microservices do you depend on? This brave new world of public APIs and microservices is great, but what happens when it’s not?

What happens when a service raises their prices, changes their metering, dies (bankruptcy, acquisition, sunset), pivots, experiences a major outage, or any number of problems facing modern technology companies? These are cross-team problems affecting ops, engineering, and business teams. How will each handle the problem?

Think about how your company consumes these APIs and plan for the worst.

There is no one right answer, but here are some things that should be explored when considering a microservice.

  • Do they offer an open-source version that you could have ready to go if necessary?
  • Will they place code in escrow in the event that they go under? You have a better chance at making this work if you’re a big player, but it’s worth bringing up.
  • How interoperable are they with the competition? Think Amazon vs Rackspace. Switching from one to another isn’t exactly fun, but it is possible to migrate without breaking much.
  • Have plans in place to “roll your own”. Writing code for this might be taking it too far (that’s the reason you’re consuming someone else’s API, right?), but at least have an idea of how you could accomplish the same features with your in-house team and a rough timeline.

So do you have a contingency plan? We’d love to hear your stories (good, bad, or otherwise) on the topic!

Calling Webhooks asynchronously with Zurl

Posted by justin on October 3rd, 2014 — Filed under Webhooks, ZeroMQ, Zurl

Zurl is an HTTP client daemon based on libcurl that makes outbound HTTP requests asynchronously. It’s super useful for invoking Webhooks. Zurl supports fire-and-forget invocation, error monitoring, and protection from evil callback URLs. Sounds pretty great, right? Let’s see how it’s done.