Realtime Data Streaming is Changing Everything

Data Streams are changing everything. The power to stream data at realtime speed, to anywhere on Earth, has become a staple of applications for every industry, and has found itself a fundamental feature for end users. As a result, we’re rethinking how we architect and build software and hardware, and has led to an explosion of new platforms, standards, and innovations.

In his keynote from Stream Conf 2016, PubNub CEO Todd Greene charts the growth of data streams from inception, and explains why realtime will be apart of every web, mobile, and Internet of Things app going forward.

Emerging Trends: How Data Streams are Changing Everything (Stream Conf 2016 Keynote) from PubNub on Vimeo.

Rest-based Architecture is Yesterday, Serverless is Today (and the future)

For years, ‘realtime’ was a device sitting idle, sending a request, waiting, and receiving a response. But it’s obvious that this isn’t really realtime. And not to mention, it’s inefficient, bulky, and doesn’t scale (imagine having millions of devices wasting bandwidth sending requests for no reason).

As a result, data streaming was born. Devices now constantly emit and receive data only when they have to, and that data is processed while it’s in-motion. In essence, the network is now programmable, and you don’t have to push data to remote servers or clouds for processing. It’s all done while your realtime data is in-transit.

This has opened up a ton of doors for what developers can build, that stretches (and continues to grow), across every industry and use-case. So what’s driving this realtime revolution?

What’s Driving the Realtime Revolution?

There are three major forces that continue to drive growth, innovation, and adoption of realtime data streaming.

Online Everywhere

Simply put, there are more devices, more people, and more accessibility into the connected world. To start, there are 2.08 billion smartphones in the wild in 2016. Key word is smartphone. These aren’t just cellphones, but full compute platforms in a third of the world’s populations’ pocket, with an always-on connection. With that, data is more affordable and more accessible than ever before.

More people = more data = more consumption.

User Expectations

Hollywood has had the technology for decades, but only recently has realtime data streaming become a reality, and a fundamental part of how users interact with apps. Shopping, playing games, navigation, and analytics — the list goes on, of apps that have at least a part of them relying on realtime interactivity. And as a result, users have come to expect that functionality.

And it’s not just Hollywood. In looking at the investment theses of the biggest VCs across the board, the entire list is in one way or another tied to massive amounts of devices sending and receiving realtime data, and having to process that data in the middle. Smart cities, virtual reality, and the ‘Uber-ization’ of manufacturing will all rely on realtime data streaming for synchronization, triggering of actions, and mission critical tracking and monitoring as core functionalities of their applications.

Software Infrastructure

If ‘software is eating the world’…software infrastructure is digesting it.

The infrastructure under the covers is what makes this all possible. Collecting and streaming data, making it easy for software developers build apps, that’s what good software infrastructure is.

And it solves three key problems for three distinct constituencies. In product management, software infrastructure is what makes a product manager’s vision and roadmap a reality. For those writing the code, it allows software developers to get a product out the door, without worrying about bits and bytes. And the last one comes down to cost, with the massive scale ideas that continue to grow, it can’t be done cost-effectively without new infrastructure.  

The Days of Monolithic Infrastructure are Over

Microservices. Serverless. PaaS. You may be thinking, haven’t we been here before? Hasn’t there been a constant move over the last couple decades between monolithic architecture and microservice-oriented architectures? Is this a typical pendulum swing back to the other side?

This time, it’s not. Monolithic architecture’s days are done, and that’s because infrastructure is everywhere. We’re in a place now where infrastructure doesn’t sit in one place anymore, because by the nature of the problems we’re solving, it has to be distributed.

Traditionally, our infrastructure ran in one place, on-premise, but more recently, there’s been the massive shift to cloud, with a pre-deployed, distributed, heterogeneous infrastructure. In a fraction of the time, companies small and large are now able get off the ground quickly, with minimal SysOps resources. But with all the innovation of the cloud, it hasn’t really changed how data moves, and software is architected. It’s mainly taken the financial and management burden of the servers, and moved it elsewhere.

In order to deal with these massive amounts of data streams, the processing needs to happen in a lot of places. In essence, the network needs to be programmable. In order to build the next generation of realtime apps, you can’t design your application to bring all the data down to one place, like a cloud or a data center, process it, and send it off. Rather, you need to be able to inject the code and execute functions in the network, while the data is in motion. That’ll be required for building and scaling realtime apps going forward.   

Beyond the programmable network, leaders in technology continue to push infrastructure forward ever more. Cisco and Ericsson, among others, have started architecting and deploying logic into hubs and cell towers, for example, making hubs in buildings programmable and deploying microservices directly into a cell tower. It’s programmatic deployment of software – not just to servers or cloud, but out to the network and the individual devices. And though this architecture isn’t widespread quite yet, these are the kinds of things that infrastructure providers will solve.

From on-premise to cloud, to the network, to edge, to the individual devices, there’s a huge opportunity for innovation and invention into how to process and stream data to the billions on always-on connected devices. New architectures will change the way that infrastructure companies, and software developers, scale and deploy realtime apps.