Turn Any Data Source into a Firehose API

Information and data transfer today is becoming fundamentally realtime. We have so much data at our finger tips, but managing such a high volume of data, then transferring that data can be challenging. As a result, we’ve seen the rise of the data firehose API. In this tutorial, we’ll walk you through one of our demonstration data streams to show just how easy it is to take any data and stream it, turning it into a custom data firehose API.

Looking Back on Data Streaming

Although it’s difficult to imagine an internet without streaming media, HTTP was originally designed to transfer only text. The 1990s saw the rise of the consumer internet and with it the demand for more substantial data transfer capabilities. In 1992, the multipart content-type was introduced, allowing us to transfer audio, images, and video. Live streaming was missing from the picture however, and work by RealNetworks, Netscape, and Columbia University culminated in Realtime Streaming Protocol being submitted to the Internet Engineering Taskforce in 1996. The rest, quite literally, is history.

Firehose API Data Stream Demo – State Capital Weather

Let’s take a look at the State Capital Weather Stream we have available on our demo data stream page. This particular stream is written in Python; we’ve got SDKs for over 70 languages and frameworks so you’re free to use whatever you’d like.

First, a connection to the PubNub network is made. You’ll first need to sign up for a PubNub account. Once you sign up, you can get your unique PubNub keys in the PubNub Developer Portal. Once you have, clone the GitHub repository, and enter your unique PubNub keys on the PubNub initialization, like so:

Then, weather data is periodically retrieved, in this case from the Weather Underground API:

Finally, it’s published over the PubNub Data Stream Network (the message payload can be anything up to 32k):

To run this example you’ll need an API key from Weather Underground, which you can find here

You’ll also need a pair of application keys for sending (publish) and receiving (subscribe), which you can get by signing up for a free account. Your keys will then be available in the developer’s portal.

In the coming months we’ll be rolling out more Data Streams; if you’re interested in having yours listed on our site or would like to give us ideas for what types of live data you’d like to be made publicly available, please drop us a line.