The Hard Thing About Streaming Things

Doing meaningful things with streaming data, that enable us to make decisions when we want to (now)—trending, statistical analysis, dynamic thresholding, ad-hoc exploration, etc—is hard, because of time. Having many sources of data (emitters) streaming a ton of data, every little bit of irregularity, skew, lag, jitter, and burstiness can throw off everything.

In his talk from Stream Conf 2016, Rajesh Ramen, Software Engineer at SignalFX discusses what happens when we try to do computations against streaming data because of the inevitable timing issues encountered in the real world, and how to engineer for them.

He touches on topics such as:

• Time series in theory and practice
• Batch vs streaming vs batch+streaming
• Strategies for managing timing issues
• How to making decisions about resolution
• Using constraint solvers