Apache Spark and OSS technologies by Paco Nathan @ Databricks

How does Apache Spark fit within the landscape of Big Data technologies? These technologies have been changing abruptly, and this talk explores where that appears to be headed.

On the one hand, the economics of datacenter technologies has shifted toward warehouse scale with commodity hardware that implies multicore and large memory spaces. Incumbent technologies do not embrace those changes, while Spark and related OSS projects work in concert to leverage them. On the other hand the shape of the data requirements are changing abruptly with sensor data, microsatellites, and other IoT use cases boosting data rates by orders of magnitude. We have decades-old advanced math techniques available to address these industrial needs, but how will our software frameworks keep pace?

This talk addresses the effective integration of newer OSS technologies for the technical audience, while providing guidance to the business audience to understand fundamental drivers for these changes — how the use cases for Big Data are extending. We will consider the roles played by functional programming (making complex workflows tractable), by cloud-based notebooks (a new wave of flexibility and collaboration), as well as how some of the advanced math has enormous implications on real-time analytics at scale.