Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Rise of Real-Time

The Rise of Real-Time

These are slides from my talk at the dotScale conference in Paris on April 24th, 2017.

There is a tectonic shift happening in how data powers the core of a company's business. This shift is about the rise of real-time. Apache Kafka was built with the vision to help companies navigate this change and become the central nervous system that makes data available in real-time to all the applications that need to use it.

This talk is about how you can put Apache Kafka to practice to help your company make this shift to real-time. And how the Connect and Streams API in Apache Kafka capture the entire scope of what it means to put streams into practice.

nehanarkhede

April 25, 2017
Tweet

More Decks by nehanarkhede

Other Decks in Technology

Transcript

  1. 14 A giant mess! App App App App search Hadoop

    DWH monitoring security MQ MQ cache cache
  2. Confidential 16 16 streaming platform DWH Hadoop security App App

    App App search NoSQL monitoring request-response messaging OR stream processing streaming data pipelines changelogs
  3. Hadoop ! { Real- time analysis }Stream processing apps Data

    Warehouse DBs Apps … or an entire company
  4. 22 event-centric thinking Hadoop Web app mobile app APIs Streaming

    Platform Event: “A product was viewed”
  5. 23 event-centric thinking mobile app web app APIs Streaming Platform

    Hadoop Security Monitoring Rec engine Event: “A product was viewed”
  6. scalability of a filesystem • hundreds of MB/s • many

    TBs per server • commodity hardware guarantees of a database • persistence • ordering • replication & fault tolerance • sharding & horizontal scaling distributed by design apache kafka: a distributed streaming platform
  7. 28 kafka is adopted at 1000s of companies Financial Services

    Enterprise Tech Consumer Tech Entertainment & Media Telecom Retail Travel & Leisure
  8. kafka for the two uses for streams build streaming data

    pipelines react to, process, transform streams
  9. NoSQL rdbms hadoop dwh search monitoring rt analytics apps apps

    apps 31 kafka's connect api = streaming data pipelines made easy!
  10. 32 connect API connect API source sink pull push kafka's

    connect api = streaming data pipelines made easy!
  11. 34 kafka for the two uses for streams build streaming

    data pipelines react to, process, transform streams
  12. vision 2: event-driven microservices using kafka’s streams api • simple

    but powerful Java library • convenient DSL • event-at-a-time processing; No micro batching • local state • automatic scaling streams api microservice stream processing
  13. kafka’s connect api + kafka’s streams api messaging api streams

    api apps app s connect api connect api source sink pull push stream processing = streaming-first enterprise
  14. Confidential 44 44 streaming platform DWH Hadoop security App App

    App App search NoSQL monitoring request-response messaging OR stream processing streaming data pipelines changelogs vision: all your data … everywhere … now
  15. Confidential 45 45 streaming platform DWH Hadoop security App App

    App App search NoSQL monitor ing request-response messaging OR stream processing streaming data pipelines changelogs vision: all your data … everywhere … now