$30 off During Our Annual Pro Sale. View Details »

Kafka and Samza: Distributed stream processing in practice

Kafka and Samza: Distributed stream processing in practice

Presented at the University of Cambridge Computer Laboratory on 12 November 2014 (http://www.talks.cam.ac.uk/talk/index/54973) and at the Imperial College London Large-Scale Distributed Systems Group on 13 November 2014 (http://lsds.doc.ic.ac.uk/node/194).

Abstract:

Stream processing is an old idea, but it is currently being rediscovered in industry due to pressures from increasing data volumes (throughput), increasingly diverse data sources (complexity) and increasing impatience (latency).

Apache Samza and Apache Kafka, two open source projects that originated at LinkedIn, are being successfully used at scale in production. Kafka is a fault-tolerant message broker, and Samza provides a scalable processing model on top of it. They have an interesting “back to basics” approach which questions many assumptions from the last few decades of data management practice.

In particular, their design is informed by the experience of operating large-scale systems under heavy load, and the challenges that arise in a large organisation with hundreds or even thousands of software engineers. This talk will introduce the architecture of Samza and Kafka, and explain some of the reasoning behind their underlying design decisions.

Martin Kleppmann

November 12, 2014
Tweet

More Decks by Martin Kleppmann

Other Decks in Programming

Transcript

  1. View Slide

  2. View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. View Slide

  14. View Slide

  15. View Slide

  16. Kafka at LinkedIn

    •  350+ commodity machines

    •  8,000+ topics

    •  140,000+ partitions

    •  278 Billion messages/day

    •  49 TB/day in

    •  176 TB/day out

    •  Peak Load

    –  4.4 Million messages per second

    –  6 Gigabits/sec Inbound

    –  21 Gigabits/sec Outbound

    View Slide

  17. View Slide

  18. View Slide

  19. View Slide

  20. View Slide

  21. View Slide

  22. View Slide

  23. View Slide

  24. View Slide

  25. View Slide

  26. View Slide

  27. View Slide

  28. View Slide

  29. View Slide

  30. View Slide

  31. View Slide

  32. View Slide

  33. View Slide

  34. View Slide

  35. View Slide

  36. View Slide

  37. View Slide

  38. View Slide

  39. View Slide

  40. View Slide

  41. References

    1.  Jay Kreps: “I ♥︎ Logs.” O'Reilly Media, September 2014. http://shop.oreilly.com/product/0636920034339.do

    2.  Jay Kreps: “Why local state is a fundamental primitive in stream processing.” 31 July 2014. http://radar.oreilly.com/
    2014/07/why-local-state-is-a-fundamental-primitive-in-stream-processing.html

    3.  Martin Kleppmann: “Designing data-intensive applications.” O’Reilly Media, to appear in 2015. http://dataintensive.net

    4.  Martin Kleppmann: “Rethinking caching in web apps.” 1 October 2012. http://martin.kleppmann.com/2012/10/01/
    rethinking-caching-in-web-apps.html

    5.  Martin Kleppmann: “Moving faster with data streams: The rise of Samza at LinkedIn.” 14 July 2014. http://
    engineering.linkedin.com/stream-processing/moving-faster-data-streams-rise-samza-linkedin

    6.  Jakob Homan: “Real time insights into LinkedIn's performance using Apache Samza.” 18 Aug 2014. http://
    engineering.linkedin.com/samza/real-time-insights-linkedins-performance-using-apache-samza

    7.  Nathan Marz and James Warren: “Big Data: Principles and best practices of scalable realtime data systems.” Manning
    MEAP, to appear January 2015. http://manning.com/marz/

    8.  Shirshanka Das, Chavdar Botev, Kapil Surlaker, et al.: “All Aboard the Databus!,” at ACM Symposium on Cloud
    Computing (SoCC), October 2012. http://www.socc2012.org/s18-das.pdf

    9.  Mahesh Balakrishnan, Dahlia Malkhi, Ted Wobber, et al.: “Tango: Distributed Data Structures over a Shared Log,” at
    24th ACM Symposium on Operating Systems Principles (SOSP), pages 325–340, November 2013. http://
    research.microsoft.com/pubs/199947/Tango.pdf

    10.  Peter Bailis, Alan Fekete, Ali Ghodsi, Joseph M Hellerstein, and Ion Stoica: “Scalable Atomic Visibility with RAMP
    Transactions,” at ACM International Conference on Management of Data (SIGMOD), June 2014. http://
    www.bailis.org/papers/ramp-sigmod2014.pdf

    11.  Roshan Sumbaly, Jay Kreps, and Sam Shah: “The ‘Big Data’ Ecosystem at LinkedIn,” at ACM International Conference
    on Management of Data (SIGMOD), July 2013. http://www.slideshare.net/s_shah/the-big-data-ecosystem-at-
    linkedin-23512853

    12.  Apache Samza documentation. http://samza.incubator.apache.org

    View Slide

  42. View Slide