$30 off During Our Annual Pro Sale. View Details »

Staying agile in the face of the data deluge

Staying agile in the face of the data deluge

Slides from my talk at Span conference, London on 28 October 2014. http://london-2014.spanconf.io/martin-kleppmann/

As our applications need to process ever more data in ever shorter time, it's difficult to stay sane. The architecture of our applications quickly becomes a monstrosity of different databases, queues and servers held together by string and sellotape. That may work at first, but soon gets ugly. If something goes wrong, it's hard to recover. If features of the application need to change, it's hard to adapt.

Stream processing gives us a route towards building data systems that are scalable, robust, and easy to adapt to changing requirements. In this talk, we will discuss how you can bring sanity to your own application architecture with Apache Samza, an open source framework for distributed stream processing applications.

Apache Samza is used in production at LinkedIn, building upon years of hard-won experience in building large-scale data systems. Even if you're not processing millions of messages per second, like LinkedIn is, you can still pick up useful tips on how to structure your data processing for scale and agility.

Martin Kleppmann

October 28, 2014
Tweet

More Decks by Martin Kleppmann

Other Decks in Programming

Transcript

  1. View Slide

  2. View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. View Slide

  14. View Slide

  15. View Slide

  16. View Slide

  17. View Slide

  18. View Slide

  19. View Slide

  20. View Slide

  21. View Slide

  22. View Slide

  23. View Slide

  24. View Slide

  25. View Slide

  26. View Slide

  27. View Slide

  28. View Slide

  29. View Slide

  30. View Slide

  31. View Slide

  32. View Slide

  33. View Slide

  34. View Slide

  35. View Slide

  36. View Slide

  37. View Slide

  38. View Slide

  39. View Slide

  40. View Slide

  41. View Slide

  42. View Slide

  43. View Slide

  44. View Slide

  45. View Slide

  46. View Slide

  47. View Slide

  48. View Slide

  49. View Slide

  50. View Slide

  51. References (fun stuff to read)

    1.  Jay Kreps: “I ♥︎ Logs.” O'Reilly Media, September 2014. http://shop.oreilly.com/product/0636920034339.do

    2.  Jay Kreps: “Why local state is a fundamental primitive in stream processing.” 31 July 2014. http://radar.oreilly.com/
    2014/07/why-local-state-is-a-fundamental-primitive-in-stream-processing.html

    3.  Martin Kleppmann: “Designing data-intensive applications.” O’Reilly Media, to appear in 2015. http://dataintensive.net

    4.  Martin Kleppmann: “Rethinking caching in web apps.” 1 October 2012. http://martin.kleppmann.com/2012/10/01/
    rethinking-caching-in-web-apps.html

    5.  Martin Kleppmann: “Moving faster with data streams: The rise of Samza at LinkedIn.” 14 July 2014. http://
    engineering.linkedin.com/stream-processing/moving-faster-data-streams-rise-samza-linkedin

    6.  Nathan Marz and James Warren: “Big Data: Principles and best practices of scalable realtime data systems.” Manning
    MEAP, to appear January 2015. http://manning.com/marz/

    7.  Shirshanka Das, Chavdar Botev, Kapil Surlaker, et al.: “All Aboard the Databus!,” at ACM Symposium on Cloud
    Computing (SoCC), October 2012. http://www.socc2012.org/s18-das.pdf

    8.  Mahesh Balakrishnan, Dahlia Malkhi, Ted Wobber, et al.: “Tango: Distributed Data Structures over a Shared Log,” at
    24th ACM Symposium on Operating Systems Principles (SOSP), pages 325–340, November 2013. http://
    research.microsoft.com/pubs/199947/Tango.pdf

    9.  Roshan Sumbaly, Jay Kreps, and Sam Shah: “The ‘Big Data’ Ecosystem at LinkedIn,” at ACM International Conference
    on Management of Data (SIGMOD), July 2013. http://www.slideshare.net/s_shah/the-big-data-ecosystem-at-
    linkedin-23512853

    10.  Apache Samza documentation. http://samza.incubator.apache.org

    View Slide

  52. View Slide