Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streams as the team interface

Streams as the team interface

Slides from a talk at Øredev, Malmö, Sweden, on 6 November 2015.


How do you remain agile when working on a large application? How do you keep feedback cycles short and ship features quickly? How do you adapt rapidly to changing requirements?

Somehow, you need to break the big application into smaller, more manageable chunks. Microservices are a growing trend for splitting up monolithic applications into independently deployable services. Although there are many good things about microservices, it’s not all roses: keeping them reliable and consistently performant can take serious effort.

In this talk, we’ll discuss a different approach to breaking down a big application into smaller chunks: stream processing. Many applications can be designed as a cascade of jobs that consume and produce real-time data streams. When you build applications this way, streams are not just an implementation detail — they become the interface from one team to another.

Building on the experience of building large-scale stream processing systems with Apache Kafka, this talk will explore how streams can help make our software development more agile, and our systems faster and more robust at the same time.


Martin Kleppmann

November 06, 2015

More Decks by Martin Kleppmann

Other Decks in Programming


  1. None
  2. None
  3. None
  4. None
  5. None
  6. None
  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. None
  14. None
  15. None
  16. None
  17. None
  18. None
  19. None
  20. None
  21. None
  22. None
  23. None
  24. None
  25. None
  26. None
  27. None
  28. None
  29. None
  30. None
  31. None
  32. None
  33. None
  34. None
  35. None
  36. None
  37. None
  38. None
  39. None
  40. None
  41. None
  42. None
  43. None
  44. None
  45. None
  46. None
  47. None
  48. None
  49. None
  50. None
  51. None
  52. None
  53. None
  54. None
  55. None
  56. None
  57. None
  58. None
  59. None
  60. None
  61. None
  62. None
  63. None
  64. None
  65. None
  66. None
  67. None
  68. None
  69. None
  70. None
  71. None
  72. None
  73. None
  74. None
  75. None
  76. None
  77. None
  78. References 1.  Jay Kreps: “I ♥︎ Logs.” O'Reilly Media, September

    2014. http://shop.oreilly.com/product/0636920034339.do 2.  Jay Kreps: “Putting Apache Kafka to use: A practical guide to building a stream data platform.” 25 February 2015. http://blog.confluent.io/2015/02/25/stream-data-platform-1/ 3.  Martin Kleppmann: “Designing data-intensive applications.” O’Reilly Media, to appear. http://dataintensive.net 4.  Martin Kleppmann: “Turning the database inside-out with Apache Samza.” 4 March 2015. http://blog.confluent.io/ 2015/03/04/turning-the-database-inside-out-with-apache-samza/ 5.  Jakob Homan: “Real time insights into LinkedIn's performance using Apache Samza.” 18 Aug 2014. http:// engineering.linkedin.com/samza/real-time-insights-linkedins-performance-using-apache-samza 6.  Nathan Marz and James Warren: “Big Data: Principles and best practices of scalable realtime data systems.” Manning, April 2015. ISBN 9781617290343, https://www.manning.com/books/big-data 7.  Shirshanka Das, Chavdar Botev, Kapil Surlaker, et al.: “All Aboard the Databus!,” at ACM Symposium on Cloud Computing (SoCC), October 2012. http://www.socc2012.org/s18-das.pdf 8.  Mahesh Balakrishnan, Dahlia Malkhi, Ted Wobber, et al.: “Tango: Distributed Data Structures over a Shared Log,” at 24th ACM Symposium on Operating Systems Principles (SOSP), pages 325–340, November 2013. http:// research.microsoft.com/pubs/199947/Tango.pdf 9.  Peter Bailis, Alan Fekete, Ali Ghodsi, Joseph M Hellerstein, and Ion Stoica: “Scalable Atomic Visibility with RAMP Transactions,” at ACM International Conference on Management of Data (SIGMOD), June 2014. http:// www.bailis.org/papers/ramp-sigmod2014.pdf 10.  Roshan Sumbaly, Jay Kreps, and Sam Shah: “The ‘Big Data’ Ecosystem at LinkedIn,” at ACM International Conference on Management of Data (SIGMOD), July 2013. http://www.slideshare.net/s_shah/the-big-data-ecosystem-at- linkedin-23512853 11.  Pat Helland: “Immutability Changes Everything,” at 7th Biennial Conference on InnovaBve Data Systems Research (CIDR), January 2015. hLp://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf 12.  Apache Kafka documentation. http://kafka.apache.org/documentation.html
  79. URL for free ebook in your goodie bag!