The Rise of Real-Time - Speaker Deck

Slide 1

Slide 1 text

the rise of real time 1 Neha Narkhede, Co-founder & CTO, Confluent

Slide 2

Slide 2 text

THEN 2

Slide 3

Slide 3 text

NOW 3

Slide 4

Slide 4 text

What does transitioning to a digital business look like? 4

Slide 5

Slide 5 text

not just a better or faster version of what has been done before 5

Slide 6

Slide 6 text

but a fundamentally different solution fit for modern digital needs 6

Slide 7

Slide 7 text

STREAMS 7

Slide 8

Slide 8 text

STREAMS 7

Slide 9

Slide 9 text

8 “all your data is event streams” bold claim

Slide 10

Slide 10 text

9 sales shipments inventory adjustments price adjustments analytics fraud reordering retail today

Slide 11

Slide 11 text

10 event streams are relevant to every business

Slide 12

Slide 12 text

11 business events logs sensor s monitoring databases event streams are relevant to every business

Slide 13

Slide 13 text

12 two uses for streams build streaming data pipelines react to, process, transform streams

Slide 14

Slide 14 text

13 how do companies actually do this?

Slide 15

Slide 15 text

14 A giant mess! App App App App search Hadoop DWH monitoring security MQ MQ cache cache

Slide 16

Slide 16 text

15 “data pipeline == event stream” key insight

Slide 17

Slide 17 text

Confidential 16 16 streaming platform DWH Hadoop security App App App App search NoSQL monitoring request-response messaging OR stream processing streaming data pipelines changelogs

Slide 18

Slide 18 text

app be it one app …

Slide 19

Slide 19 text

Hadoop ! { Real- time analysis }Stream processing apps Data Warehouse DBs Apps … or an entire company

Slide 20

Slide 20 text

how do you being the transition to a streaming-first enterprise?

Slide 21

Slide 21 text

make a fundamental transition to event-centric thinking

Slide 22

Slide 22 text

21 event-centric thinking Streaming Platform Event: “A product was viewed” Hadoop Web app

Slide 23

Slide 23 text

22 event-centric thinking Hadoop Web app mobile app APIs Streaming Platform Event: “A product was viewed”

Slide 24

Slide 24 text

23 event-centric thinking mobile app web app APIs Streaming Platform Hadoop Security Monitoring Rec engine Event: “A product was viewed”

Slide 25

Slide 25 text

Confidential 24 event-centric thinking at a company-wide scale! 24

Slide 26

Slide 26 text

scalability of a filesystem ● hundreds of MB/s ● many TBs per server ● commodity hardware guarantees of a database ● persistence ● ordering ● replication & fault tolerance ● sharding & horizontal scaling distributed by design apache kafka: a distributed streaming platform

Slide 27

Slide 27 text

Confidential 26 26 apache kafka 7 years ago

Slide 28

Slide 28 text

27 > 1,400,000,000,000 messages processed per day

Slide 29

Slide 29 text

28 kafka is adopted at 1000s of companies Financial Services Enterprise Tech Consumer Tech Entertainment & Media Telecom Retail Travel & Leisure

Slide 30

Slide 30 text

how does Kafka put streams into practice?

Slide 31

Slide 31 text

kafka for the two uses for streams build streaming data pipelines react to, process, transform streams

Slide 32

Slide 32 text

NoSQL rdbms hadoop dwh search monitoring rt analytics apps apps apps 31 kafka's connect api = streaming data pipelines made easy!

Slide 33

Slide 33 text

32 connect API connect API source sink pull push kafka's connect api = streaming data pipelines made easy!

Slide 34

Slide 34 text

33 Kafka’s connect API kafka ALL the things!

Slide 35

Slide 35 text

34 kafka for the two uses for streams build streaming data pipelines react to, process, transform streams

Slide 36

Slide 36 text

Confidential 35 stream processing 35

Slide 37

Slide 37 text

Confidential 36 kafka’s streams api = stream processing made easy! 36

Slide 38

Slide 38 text

37 2 visions for stream processing real-time mapreduce event-driven microservices

Slide 39

Slide 39 text

vision 1: real-time mapreduce 38

Slide 40

Slide 40 text

39 vision 2: event-driven microservices streams api microservice stream processing

Slide 41

Slide 41 text

vision 2: event-driven microservices using kafka’s streams api ● simple but powerful Java library ● convenient DSL ● event-at-a-time processing; No micro batching ● local state ● automatic scaling streams api microservice stream processing

Slide 42

Slide 42 text

41 example: real-time dashboard app for security monitoring

Slide 43

Slide 43 text

42 vision 1 vision 2 kafka’s streams api: build apps, not clusters

Slide 44

Slide 44 text

kafka’s connect api + kafka’s streams api messaging api streams api apps app s connect api connect api source sink pull push stream processing = streaming-first enterprise

Slide 45

Slide 45 text

Confidential 44 44 streaming platform DWH Hadoop security App App App App search NoSQL monitoring request-response messaging OR stream processing streaming data pipelines changelogs vision: all your data … everywhere … now

Slide 46

Slide 46 text

Confidential 45 45 streaming platform DWH Hadoop security App App App App search NoSQL monitor ing request-response messaging OR stream processing streaming data pipelines changelogs vision: all your data … everywhere … now

Slide 47

Slide 47 text

confluent.io/download 46 @nehanarkhede