The Rise of Real-Time

The Rise of Real-Time

These are slides from my talk at the dotScale conference in Paris on April 24th, 2017.

There is a tectonic shift happening in how data powers the core of a company's business. This shift is about the rise of real-time. Apache Kafka was built with the vision to help companies navigate this change and become the central nervous system that makes data available in real-time to all the applications that need to use it.

This talk is about how you can put Apache Kafka to practice to help your company make this shift to real-time. And how the Connect and Streams API in Apache Kafka capture the entire scope of what it means to put streams into practice.

C7f59de0d5062b4d704a47f9dbe91b66?s=128

nehanarkhede

April 25, 2017
Tweet

Transcript

  1. the rise of real time 1 Neha Narkhede, Co-founder &

    CTO, Confluent
  2. THEN 2

  3. NOW 3

  4. What does transitioning to a digital business look like? 4

  5. not just a better or faster version of what has

    been done before 5
  6. but a fundamentally different solution fit for modern digital needs

    6
  7. STREAMS 7

  8. STREAMS 7

  9. 8 “all your data is event streams” bold claim

  10. 9 sales shipments inventory adjustments price adjustments analytics fraud reordering

    retail today
  11. 10 event streams are relevant to every business

  12. 11 business events logs sensor s monitoring databases event streams

    are relevant to every business
  13. 12 two uses for streams build streaming data pipelines react

    to, process, transform streams
  14. 13 how do companies actually do this?

  15. 14 A giant mess! App App App App search Hadoop

    DWH monitoring security MQ MQ cache cache
  16. 15 “data pipeline == event stream” key insight

  17. Confidential 16 16 streaming platform DWH Hadoop security App App

    App App search NoSQL monitoring request-response messaging OR stream processing streaming data pipelines changelogs
  18. app be it one app …

  19. Hadoop ! { Real- time analysis }Stream processing apps Data

    Warehouse DBs Apps … or an entire company
  20. how do you being the transition to a streaming-first enterprise?

  21. make a fundamental transition to event-centric thinking

  22. 21 event-centric thinking Streaming Platform Event: “A product was viewed”

    Hadoop Web app
  23. 22 event-centric thinking Hadoop Web app mobile app APIs Streaming

    Platform Event: “A product was viewed”
  24. 23 event-centric thinking mobile app web app APIs Streaming Platform

    Hadoop Security Monitoring Rec engine Event: “A product was viewed”
  25. Confidential 24 event-centric thinking at a company-wide scale! 24

  26. scalability of a filesystem • hundreds of MB/s • many

    TBs per server • commodity hardware guarantees of a database • persistence • ordering • replication & fault tolerance • sharding & horizontal scaling distributed by design apache kafka: a distributed streaming platform
  27. Confidential 26 26 apache kafka 7 years ago

  28. 27 > 1,400,000,000,000 messages processed per day

  29. 28 kafka is adopted at 1000s of companies Financial Services

    Enterprise Tech Consumer Tech Entertainment & Media Telecom Retail Travel & Leisure
  30. how does Kafka put streams into practice?

  31. kafka for the two uses for streams build streaming data

    pipelines react to, process, transform streams
  32. NoSQL rdbms hadoop dwh search monitoring rt analytics apps apps

    apps 31 kafka's connect api = streaming data pipelines made easy!
  33. 32 connect API connect API source sink pull push kafka's

    connect api = streaming data pipelines made easy!
  34. 33 Kafka’s connect API kafka ALL the things!

  35. 34 kafka for the two uses for streams build streaming

    data pipelines react to, process, transform streams
  36. Confidential 35 stream processing 35

  37. Confidential 36 kafka’s streams api = stream processing made easy!

    36
  38. 37 2 visions for stream processing real-time mapreduce event-driven microservices

  39. vision 1: real-time mapreduce 38

  40. 39 vision 2: event-driven microservices streams api microservice stream processing

  41. vision 2: event-driven microservices using kafka’s streams api • simple

    but powerful Java library • convenient DSL • event-at-a-time processing; No micro batching • local state • automatic scaling streams api microservice stream processing
  42. 41 example: real-time dashboard app for security monitoring

  43. 42 vision 1 vision 2 kafka’s streams api: build apps,

    not clusters
  44. kafka’s connect api + kafka’s streams api messaging api streams

    api apps app s connect api connect api source sink pull push stream processing = streaming-first enterprise
  45. Confidential 44 44 streaming platform DWH Hadoop security App App

    App App search NoSQL monitoring request-response messaging OR stream processing streaming data pipelines changelogs vision: all your data … everywhere … now
  46. Confidential 45 45 streaming platform DWH Hadoop security App App

    App App search NoSQL monitor ing request-response messaging OR stream processing streaming data pipelines changelogs vision: all your data … everywhere … now
  47. confluent.io/download 46 @nehanarkhede