Scalable stream processing with Apache Kafka and Apache Samza

Scalable stream processing with Apache Kafka and Apache Samza

I have given this talk (with minor variations) at the following venues:

• ApacheCon EU, Budapest, Hungary, 18 November 2014.
• Unified Log London Meetup, London, UK, 2 December 2014.
• Jfokus, Stockholm, Sweden, 4 February 2015.


Samza, an Apache Incubator project, is a framework for processing and analysing high-volume data streams. It is built upon Apache Kafka and YARN (Hadoop 2.0). You can think of Samza as a real-time, continuously running version of MapReduce.

In this talk, Martin will show why stream processing is becoming an important part of the architecture of data-intensive applications, alongside storage and batch processing. We will explore how Samza works, and show how it reliably processes millions of messages per second. We will also examine what kinds of applications would benefit from using Samza.


Martin Kleppmann

November 18, 2014