Large scale stream processing with Apache Flink

Large scale stream processing with Apache Flink

In today’s world it’s no longer enough to build systems that process big volumes of information. We now need applications that can handle large continuous streams of data with very low latency so we can react to the ever-changing environment around us. To efficiently handle such problems we need to deploy a stream processing solution. During the talk we’ll explore one of the most popular frameworks for stream processing – Apache Flink. We’ll see what unique capabilities it provides and how they apply to some real world problems. And we’ll also explore how it works under the hood and how to get the scalable and fault-tolerant stream processing that Flink provides.

Eb44761e0fb3a5ec8e23ec28048dd7a5?s=128

Nikolay Stoitsev

November 28, 2018
Tweet

Transcript

  1. Large scale stream processing with Apache Flink Nikolay Stoitsev Sr.

    Software Engineer at Uber Tech Sofia
  2. Stream Processing?

  3. Stream Processing? User Interaction Logs

  4. Stream Processing? User Interaction Logs Application Logs

  5. Stream Processing? User Interaction Logs Application Logs Sensor Data

  6. Stream Processing? User Interaction Logs Application Logs Sensor Data Database

    Commit Logs
  7. Infinite Dataset

  8. Producer Stream

  9. Producer Stream HDFS

  10. Producer Stream HDFS Hive

  11. Producer Stream HDFS Hive Big Latency

  12. Producer Stream HDFS Real-time service

  13. Apache Storm storm.apache.org

  14. High-latency & accurate vs. Low-latency & approximation

  15. Lambda architecture

  16. https://www.oreilly.com/ideas/questioning-the-lambda-architecture

  17. Kappa Architecture

  18. Use Apache Kafka Durable, scalable, fault-tolerant

  19. Producer Kafka Stream Processor

  20. None
  21. None
  22. Metrics we want to track Net payout Daily items sold

    Weekly items sold Order acceptance rate Order preparation speed Item rating
  23. Real time

  24. Scalable

  25. Granular

  26. Highly available

  27. Order Stream Payment Stream User Rating Stream

  28. Order Stream Payment Stream User Rating Stream Stream Processor OLAP

  29. samza.apache.org

  30. Apache Flink flink.apache.org

  31. Everything is a batch vs. Everything is a stream

  32. Single JVM Cluster Cloud Runtime DataSet API DataStream API

  33. Dataflow graph

  34. Source Source Operator Operator Operator Sinc OLAP

  35. https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/programming-model.html

  36. https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/programming-model.html

  37. https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/programming-model.html

  38. Flink Program Optimizer Graph Builder Client

  39. Flink Program Optimizer Graph Builder Client Job Manager Task Manager

    Task Manager
  40. Flink Program Optimizer Graph Builder Client Job Manager Task Manager

    Task Manager Snapshot Store
  41. Fault tolerant

  42. Flink Program Optimizer Graph Builder Client Job Manager Task Manager

    Task Manager Snapshot Store
  43. Lightweight Asynchronous Snapshots for Distributed Dataflows Paris Carbone, Gyula Fóra,

    Stephan Ewen Seif Haridi Kostas Tzoumas
  44. Barrier Msg Msg Barrier Msg Msg Barrier Operator

  45. Barrier Msg Msg Barrier Msg Msg Operator Msg Snapshot Store

  46. Exactly Once Processing

  47. Can handle very large state

  48. Flink Program Optimizer Graph Builder Client Job Manager Task Manager

    Task Manager Snapshot Store
  49. Flink Program Optimizer Graph Builder Client Job Manager Task Manager

    Task Manager Snapshot Store Job Manager Job Manager Zookeeper
  50. Flink Program Optimizer Graph Builder Client Job Manager Task Manager

    Task Manager Snapshot Store Job Manager Job Manager Zookeeper
  51. Flink Program Optimizer Graph Builder Client Task Manager Task Manager

    Snapshot Store Job Manager Job Manager Zookeeper
  52. Joining Streams

  53. Order Stream User Rating Stream

  54. Order Stream User Rating Stream

  55. Order Stream User Rating Stream Local Join Local Join

  56. Order Stream User Rating Stream Local Join Local Join

  57. Apache Flink • Can join streams • Fault tolerant •

    Exactly Once Processing • Combines stream and batch processing
  58. … but it requires Java/Scala code

  59. Scalable, efficient and robust

  60. github.com/uber/AthenaX

  61. SQL → what data to analyze Flink → how to

    analyze it
  62. None
  63. None
  64. None
  65. None
  66. None
  67. None
  68. None
  69. Resource estimation and auto scaling

  70. Monitoring and automatic failure recovery

  71. eng.uber.com/athenax

  72. Thanks! Nikolay Stoitsev @ Uber

  73. None