Title of class/project By Juan Martín Pampliega Implementing Stream Processing Systems Juan Martín Pampliega

Juan Pampliega Information Engineering @ ITBA Co-Founder @ Mutt Data Professor @ ITBA - Especialización en Ciencia de Datos ● Working in Data Projects since 2010 at ITBA, Globant (Google), Despegar, Socialmetrix, Jampp, Claro, etc. ● Co Founder @ Mutt Data a company specialized in developing projects using Big Data and Data Science. ● Developed first production ready stream processing system in 2015 @juanpampliega | [email protected]

Contents Why Stream Processing? Fundamental Concepts Technology Landscape Summary Implementing Stream Processing Systems 01 02 03 04

01 Why Stream Processing ?

Event-Command vs Event-Driven For organizations to become real time, they must become event driven In Event-Command pattern (like REST applications) the endpoint is known, the method being called is also known and lastly the calls tended to return a value. In Event-Driven pattern services communicate only by generating events that can be reused by any service in the system which leads to less coupling.

Service Oriented

Event Driven: Unified Log

Event Streams Data is generally born as continuous event streams Batch processing => bounded datasets Stream processing => unbounded datasets Stream processing means computing on data directly as it is produced or received.

Batch Processing Traditional data processing storing streams of data in databases and later processing it using ETLs. Still needed when you need to explore data an when you are not yet sure what you want to do with it.

Stream Processing In stream processing the application logic, analytics, and queries exist continuously, and data flows through them continuously.

To produce results in real time a system must continuously compute and update results with each new event. Modern applications and microservices should operate in an event-driven fashion. Their logic and computation is triggered by events. Unified event-driven applications and real-time analytics

02 Fundamental Concepts

Operations over Streams FILTER JOIN

Stateful Streaming Processing Stateful stream processing is a subset of stream processing in which the computation maintains contextual state. This state is used to store information derived from the previously-seen events.

Windows Partition a stream of data into discrete batches. Needed to compute metrics that require context (average by minute, partial counts, etc.)

Event Time vs Processing Time ● Processing and transport technologies. ● Characteristics of the data source. (distribution, throughput, burstiness, etc.) ● Hardware. (network, RAM, CPUs, etc.) ● FAILURES & RETRIES

Event Time vs Processing Time

Operations over Streams PROCESSING TIME EVENT TIME

Streaming State and Windows

Fault Tolerant Local State Local state is a fundamental primitive in stream processing. It can be indexed and accessed in a variety of rich ways. Local, in-process data access is much faster. It’s easier to isolate. Implemented with in memory hash table, bloom filters, bit maps, RocksDB like systems, etc.

Critical Questions in Stream Processing Tyler Akidau (tech lead for internal streaming data processing systems @ Google) defined 4 critical questions any stream processing system should be able to answer: What results are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?

Watermarks and Triggers

03 Technology Landscape

Ververica Platform

Apache Flink

Companies that use Apache Flink

Apache Beam Beam Model: Fn Runners Apache Flink Apache Spark Beam Model: Pipeline Construction Other Languages Beam Java Beam Python Execution Execution Cloud Dataflow Execution 1. The Beam Model: What / Where / When / How 2. SDKs for writing Beam pipelines -- starting with Java 3. Runners for Existing Distributed Processing Backends ○ Apache Apex ○ Apache Flink ○ Apache Spark ○ Google Cloud Dataflow ○ Local (in-process) runner for testing

Google Cloud Dataflow

Spark Streaming Platform

Confluent Platform

04 Resources

Ben Stopford, The Data Dichotomy: Rethinking the Way We Treat Data and Services https://www.confluent.io/blog/data-dichotomy-rethinking-the-way-we-treat-data-and- services/ Tyler Akidau, Streaming 101: The world beyond batch https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101 Tyler Akidau, Streaming 102: The world beyond batch https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102 Ververica, What is Stream Processing? https://www.ververica.com/what-is-stream-processing Apache Flink Documentation https://ci.apache.org/projects/flink/flink-docs-release-1.8/ Apache Beam https://beam.apache.org/

Title of class/project By Juan Martín Pampliega Thank you ! Implementing Stream Processing Systems By Juan Martín Pampliega We are hiring Data Engineers! [email protected]