The Era of the Realtime Data Stream Network Defined

Subscribe to Our Newsletter

Stay updated with the latest on web, mobile, and IoT, delivered weekly.

Thanks for subscribing!

Get ready for some great content.

Our customers are pushing the envelope in how they create and utilize realtime data streams. Each application developer comes to PubNub with a set of demanding expectations for a data stream network that are driven by a satisfying user experience for their customers. At PubNub, we deliver millions of realtime data streams to an enormous global audience in a quarter second or less, all day, every day.

The concept of a realtime data stream has changed through the course of history. Today, not everyone is familiar with exactly what a real time data stream is but we definitely know realtime when we see it. Few of us realize how often realtime data streams are used to satisfy our never ending thirst for knowing what is happening “right now” and even fewer are familiar with what is required to deliver realtime data streams across the Internet at scale.

Near-Time vs. Realtime Data

Not all applications require realtime data. They really only require that the data delivered be relevant to the intended task. The challenge in today’s world is that rate at which data becomes irrelevant has accelerated beyond what traditional network delivery solutions can handle.

Hourly or Daily Data Stream

It used to be that information was fairly static and needed to be updated infrequently. Once a day, or less for example. While this is occasionally still true of applications today, the value of such information appears to be in decline.

Take the daily outcome of the stock-market results as an example. This data is either static data (yesterday’s result) or dynamic (today’s result) and it only changes every 24 hours or so. If we only needed to know a summary of the day’s events, but didn’t need to know what happened in any one instant in the day, then our application could provide a summary of the day’s stock market performance and our users could check it out at their leisure.

Before this, as far back as 1797, we just read the newspaper or waited for a Reuter’s carrier-pigeon to bring the latest financial news of the day to us. This was outstanding up-to-date information for its time. Today, you are at a frightening disadvantage as a market trader if you are using this type of data stream network.

Minutes or Seconds Data Stream

Today’s dynamic web applications provide updated information every so often. This type of data is dynamic with an update frequency of a few minutes or seconds.

Sticking with the financial example, for those looking to track the performance of the stock market throughout the day, using an update that occurs every few seconds might suffice. This provides all the information needed at a rate that lets us feel reasonable sure we understand what is happening during the course of the day. Updates that happen every few seconds might be considered near time, but certainly are not real time.

Data stream network subscribers in 1869.

Back in 1870, it was possible get the same frequency update by watching a ticker-tape print out 20 minute delayed results at a rate of one character per second. This was a cutting edge data stream network and was probably the first time ever that you could subscribe to a networked data stream. Today you can go to the financial section of your online news source to get these type of dynamic updates.

Delivering this type of data over the internet in the 21st century is still fairly challenging. Building an application that keeps information up to date and reliably available with reasonable level of performance for a global audience requires an infrastructure that scales to demands and delivers a quality of service expected by consumers. This is what a Content Distribution Network is all about and for the last decade CDNs have been a requirement for doing business at scale on the internet.

Continual & Instantaneous Data Stream with Millisecond Timescale (or less)

Today, applications are publishing data about events at the moment of their creation. Data is published to subscribers and processed extremely fast making it possible to react to changing conditions in real time. This type of data is required for:

It’s simply not possible to realize these types of use cases with a traditional “Near-Time” architecture. These use cases are about as realtime as realtime can get; they enable swift reactions to a near instantaneous stream of fast data before an opportunity is lost.

Next generation taxi companies such as Uber, Lyft and Get-Taxi have been using realtime data stream networks to their advantage and collectively are now starting to dwarf more traditional transportation models that have existed for literally hundreds of years.

In the financial sector, firms now make money by observing market fluctuations on timescales as short as a few milliseconds. Trading houses now require a continuous source of instantaneous market data with as close to zero latency as possible. This is a stream of data that represents the pulse of market conditions at any given moment. Trading houses typically position their offices so that this data stream travels the quickest path from the exchange to the quant’s financial algorithms. Every millisecond of latency saved is a competitive advantage and equates to an opportunity to react to earn more revenue.

Data Stream Network in Action

Data Stream Network Analytics

Delivering this type of data at scale requires a special type of infrastructure. Although real time data stream delivery has a similar set of requirements to that of a traditional CDN: global scale, 99.999% reliability and quality of experience it also comes with a set of unique requirements that include:

Data Stream Networks have been designed to address exactly this set of challenges for today’s real time application developers. As the world quickly embraces the benefits of real time, Data Stream Networks are becoming the de-facto choice to make these type of applications possible. It simply doesn’t make sense for developers to spend time and resources re-inventing the wheel, when it is much faster, easier and cheaper to out-source delivery to a proven solution.

Asynchronous Updates: milliseconds timescale and unpredictable in nature.

There is another type of dynamic realtime data that doesn’t necessarily produce a continual or periodic stream of data but also has the potential to be highly variable. In this case the information maybe updated at some point in time that is not predetermined but the response to the update needs to happen as quickly as possible.

Sensors that test whether a certain condition is true or not, for exmaple, send a trigger the very moment an event registers. The world of the Internet Of Things is filled with all types of devices that require realtime signaling of this sort. Data Stream Networks are perfect fit for delivering real-time asynchronous updates to IoT applications on millisecond timescales including:

This video demonstrates a serious example: an airbag for motorcyclists worn within the rider’s jacket that is triggered remotely by a sensor on the motorcycle. The rider’s airbag takes 25ms to deploy once triggered, so the sensor needs to send the the trigger event within 60ms of the motorcycle impacting an object for the rider to be sufficiently protected. There is very little room for error and the rider’s is relying on a real time data stream to prevent serious injury or death.

Real Time Data Stream Examples

As you can see, the idea of what realtime is has changed over time. Looking back over centuries, we can see that our appreciation of up-to-date information has been colored by the age in which we live. Today, by necessity, a real time data stream amounts to more information delivered faster than either you or I can physiologically grasp.

Realtime Data Streams can be used in all sorts of creative ways. Below are a few examples:

PubNub Global Network Traffic

PubNub’s Data Stream Network delivers millions of real time data streams and we have our very own data stream that we use to visualize our global traffic as it streams across our network. The application below creates a 3D visualization of the data stream rendered in your web browser in real time:

Twitter Stream

Twitter Stream contains a stream of the latest Tweets submitted to Twitter sent at a rate of no faster than 50 Tweets per second. This is more Tweets than any human could ever read in one second, delivered every second, forever.

PubNub’s Data Stream Network makes it very easy to access this data from application code so that you can make it meaningful in some way.

For example, the Twitter Emotion application, shown here takes the Tweeter Stream data feed and produces a map indicating where Tweets originated geographically and how the users were feeling based on the content of the Tweet:

Here’s a wide variety of examples of realtime data streams and realtime applications that leverage the data. In future posts we will see how different sorts of applications are latency sensitive and where latency comes from.