Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Large scale stream processing with Apache Flink

Large scale stream processing with Apache Flink

In today’s world it’s no longer enough to build systems that process big volumes of information. We now need applications that can handle large continuous streams of data with very low latency so we can react to the ever-changing environment around us. To efficiently handle such problems we need to deploy a stream processing solution. During the talk we’ll explore one of the most popular frameworks for stream processing – Apache Flink. We’ll see what unique capabilities it provides and how they apply to some real world problems. And we’ll also explore how it works under the hood and how to get the scalable and fault-tolerant stream processing that Flink provides.

Nikolay Stoitsev

November 28, 2018
Tweet

More Decks by Nikolay Stoitsev

Other Decks in Programming

Transcript

  1. Metrics we want to track Net payout Daily items sold

    Weekly items sold Order acceptance rate Order preparation speed Item rating
  2. Flink Program Optimizer Graph Builder Client Job Manager Task Manager

    Task Manager Snapshot Store Job Manager Job Manager Zookeeper
  3. Flink Program Optimizer Graph Builder Client Job Manager Task Manager

    Task Manager Snapshot Store Job Manager Job Manager Zookeeper
  4. Flink Program Optimizer Graph Builder Client Task Manager Task Manager

    Snapshot Store Job Manager Job Manager Zookeeper
  5. Apache Flink • Can join streams • Fault tolerant •

    Exactly Once Processing • Combines stream and batch processing