The Rise of Real-Time

These are slides from my talk at the dotScale conference in Paris on April 24th, 2017.

There is a tectonic shift happening in how data powers the core of a company's business. This shift is about the rise of real-time. Apache Kafka was built with the vision to help companies navigate this change and become the central nervous system that makes data available in real-time to all the applications that need to use it.

This talk is about how you can put Apache Kafka to practice to help your company make this shift to real-time. And how the Connect and Streams API in Apache Kafka capture the entire scope of what it means to put streams into practice.

Transcript

1. the rise of real time 1 Neha Narkhede, Co-founder & CTO, Confluent
2. THEN 2
3. NOW 3
4. What does transitioning to a digital business look like? 4
5. not just a better or faster version of what has been done before 5
6. but a fundamentally different solution fit for modern digital needs 6
7. STREAMS 7
8. STREAMS 7
9. 8 “all your data is event streams” bold claim
10. 9 sales shipments inventory adjustments price adjustments analytics fraud reordering retail today
11. 10 event streams are relevant to every business
12. 11 business events logs sensor s monitoring databases event streams are relevant to every business
13. 12 two uses for streams build streaming data pipelines react to, process, transform streams
14. 13 how do companies actually do this?
15. 14 A giant mess! App App App App search Hadoop DWH monitoring security MQ MQ cache cache
16. 15 “data pipeline == event stream” key insight
17. Confidential 16 16 streaming platform DWH Hadoop security App App App App search NoSQL monitoring request-response messaging OR stream processing streaming data pipelines changelogs
18. app be it one app …
19. Hadoop ! { Real- time analysis }Stream processing apps Data Warehouse DBs Apps … or an entire company
20. how do you being the transition to a streaming-first enterprise?
21. make a fundamental transition to event-centric thinking
22. 21 event-centric thinking Streaming Platform Event: “A product was viewed” Hadoop Web app
23. 22 event-centric thinking Hadoop Web app mobile app APIs Streaming Platform Event: “A product was viewed”
24. 23 event-centric thinking mobile app web app APIs Streaming Platform Hadoop Security Monitoring Rec engine Event: “A product was viewed”
25. Confidential 24 event-centric thinking at a company-wide scale! 24
26. scalability of a filesystem • hundreds of MB/s • many TBs per server • commodity hardware guarantees of a database • persistence • ordering • replication & fault tolerance • sharding & horizontal scaling distributed by design apache kafka: a distributed streaming platform
27. Confidential 26 26 apache kafka 7 years ago
28. 27 > 1,400,000,000,000 messages processed per day
29. 28 kafka is adopted at 1000s of companies Financial Services Enterprise Tech Consumer Tech Entertainment & Media Telecom Retail Travel & Leisure
30. how does Kafka put streams into practice?
31. kafka for the two uses for streams build streaming data pipelines react to, process, transform streams
32. NoSQL rdbms hadoop dwh search monitoring rt analytics apps apps apps 31 kafka's connect api = streaming data pipelines made easy!
33. 32 connect API connect API source sink pull push kafka's connect api = streaming data pipelines made easy!
34. 33 Kafka’s connect API kafka ALL the things!
35. 34 kafka for the two uses for streams build streaming data pipelines react to, process, transform streams
36. Confidential 35 stream processing 35
37. Confidential 36 kafka’s streams api = stream processing made easy! 36
38. 37 2 visions for stream processing real-time mapreduce event-driven microservices
39. vision 1: real-time mapreduce 38
40. 39 vision 2: event-driven microservices streams api microservice stream processing
41. vision 2: event-driven microservices using kafka’s streams api • simple but powerful Java library • convenient DSL • event-at-a-time processing; No micro batching • local state • automatic scaling streams api microservice stream processing
42. 41 example: real-time dashboard app for security monitoring
43. 42 vision 1 vision 2 kafka’s streams api: build apps, not clusters
44. kafka’s connect api + kafka’s streams api messaging api streams api apps app s connect api connect api source sink pull push stream processing = streaming-first enterprise
45. Confidential 44 44 streaming platform DWH Hadoop security App App App App search NoSQL monitoring request-response messaging OR stream processing streaming data pipelines changelogs vision: all your data … everywhere … now
46. Confidential 45 45 streaming platform DWH Hadoop security App App App App search NoSQL monitor ing request-response messaging OR stream processing streaming data pipelines changelogs vision: all your data … everywhere … now
47. confluent.io/download 46 @nehanarkhede