Boston Meetup, Fev 2020

Transcript

1. Rediscovering the value of apache kafka® in modern data architectures @riferrei | #kafkameetup | @CONFLUENTINC
2. About me @riferrei | @kafkameetup | @CONFLUENTINC • RICARDO FERREIRA • Works for confluent • Developer advocate • Ricardo@confluent.iO • HTTPS://RIFERREI.NET
5. Origins of apache kafka @riferrei | @kafkameetup | @CONFLUENTINC ”there were lots of databases and other systems built to store data, but what was missing in our architecture was something that would help us to handle continuous flows of data.” – jay kreps
6. @riferrei | @kafkameetup | @CONFLUENTINC
7. @riferrei | @kafkameetup | @CONFLUENTINC First realization > I changed my job from oracle to confluent I work at confluent event state
8. @riferrei | @kafkameetup | @CONFLUENTINC Events are both notification State transfer +
9. @riferrei | @kafkameetup | @CONFLUENTINC Event-driven application Job change recommendation engine Search engine Email service
10. @riferrei | @kafkameetup | @CONFLUENTINC SQL SQL SQL Recommendation engine Search engine Email service database LOG Let’s implement this!
11. @riferrei | @kafkameetup | @CONFLUENTINC second realization database 1000x more volume Non-transactional events Transactional events LOG
12. Databases, 30 years ago...
13. Developer Databases, these days...
14. @riferrei | @kafkameetup | @CONFLUENTINC Databases are limited
15. Limited? Are you kidding me?
16. @riferrei | @kafkameetup | @CONFLUENTINC ARE DATABASES LIMITED? YES, THEY ARE. WHY DO WE HAVE TO MOVE DATA FROM ONE DB TO ANOTHER JUST TO DO ANALYTICS?
17. @riferrei | @kafkameetup | @CONFLUENTINC SHARED STATE = MORE DB’S Business line 1 Business line 2 Business line 3
18. @riferrei | @kafkameetup | @CONFLUENTINC THIRD REALIZATION User tracking Historical data Operational metrics Nosql database Graph database Sql database microservices ... HADOOP Elastic search grafana Machine learning REC. ENGINE SEARCH SECURITY EMAIL SOCIAL GRAPH
19. “The truth is the log. The database is a cache of a subset of the log.” — pat helland Immutability changes everything http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
20. @riferrei | @kafkameetup | @CONFLUENTINC log as first-class citizen database LOG 0 1 2 3 4 5 6 7 8 LOG reads writes Destination System a (time = 1) Destination System b (time = 3)
21. @riferrei | @kafkameetup | @CONFLUENTINC SOLUTION: BUILD A COMMIT LOG Commit LOG User tracking Historical data Operational metrics Nosql database Graph database Sql database microservices ... HADOOP Elastic search grafana Machine learning REC. ENGINE SEARCH SECURITY EMAIL SOCIAL GRAPH
22. http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf Do you THINK is a table that you RUN QUERIES?
23. @riferrei | @kafkameetup | @CONFLUENTINC STREAMS AND TABLES DUALITY {"user":"riferrei","score":"1001"} {"user":"riferrei","score":"1002"} {"user":"riferrei","score":"1003"} {"user":"riferrei","score":"1004"} {"user":"riferrei","score":"1005"} {"user":"riferrei","score":"1005"} stream table
24. Origins of apache kafka @riferrei | @kafkameetup | @CONFLUENTINC ”WE’VE COME TO THINK OF KAFKA AS A STREAMING PLATFORM: A SYSTEM THAT LETS YOU PUBLISH AND SUBSCRIBE TO STREAMS OF DATA, STORE THEM, AND PROCESS THEM, AND THAT IS EXACTLY WHAT APACHE KAFKA IS BUILT TO BE.” – jay kreps
25. @riferrei | @kafkameetup | @CONFLUENTINC ORIGINS OF APACHE KAFKA Databases Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Fast (Low Latency)
26. @riferrei | @kafkameetup | @CONFLUENTINC ORIGINS OF APACHE KAFKA Databases Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Fast (Low Latency) Highly Scalable Durable Persistent Ordered Fast (Low Latency) Distributed Commit log
27. @riferrei | @kafkameetup | @CONFLUENTINC ORIGINS OF APACHE KAFKA Databases Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Fast (Low Latency) Highly Scalable Durable Persistent Ordered Fast (Low Latency) Stream processing Continuous flows Scalable integration Distributed Streaming platform
28. @riferrei | @confluentinc | @itau
29. Origins of apache kafka @riferrei | @kafkameetup | @CONFLUENTINC ”the ability to combine these three areas – to bring all the streams of data together across all the use cases – is what makes the idea of a streaming platform so appealing to people” – jay kreps
30. 01 Well done messaging 02 Durable storage 03 Stream processing WHAT IS APACHE KAFKA?
31. @riferrei | @kafkameetup | @CONFLUENTINC Time for some fun 1. Get the game 2. Name yourself
32. @riferrei | @KAFKAMEETUP | @CONFLUENTINC https://github.com/confluentinc/demo-scene <<Pacman-ccloud>> Source-code
33. @riferrei | @kafkameetup | @CONFLUENTINC Source: USER_GAME TOPIC
34. @riferrei | @kafkameetup | @CONFLUENTINC Creating User_game stream
35. @riferrei | @kafkameetup | @CONFLUENTINC Querying USER_GAME STREAM
36. @riferrei | @kafkameetup | @CONFLUENTINC Creating Stats_per_user table
37. @riferrei | @kafkameetup | @CONFLUENTINC Querying STATS_PER_USER TABLE
38. @riferrei | @kafkameetup | @CONFLUENTINC Low latency Pull queries
39. @riferrei | @kafkameetup | @CONFLUENTINC Source: User_losses topic
40. @riferrei | @kafkameetup | @CONFLUENTINC Creating USER_LOSSES STREAM
41. @riferrei | @kafkameetup | @CONFLUENTINC querying USER_LOSSES STREAM
42. @riferrei | @kafkameetup | @CONFLUENTINC Creating LOSSES_PER_USER TABLE
43. @riferrei | @kafkameetup | @CONFLUENTINC Querying LOSSES_PER_USER TABLE
44. @riferrei | @kafkameetup | @CONFLUENTINC Creating SCOREBOARD TABLE
45. @riferrei | @kafkameetup | @CONFLUENTINC Querying SCOREBOARD TABLE
46. @riferrei | @kafkameetup | @CONFLUENTINC Complete scoreboard USER_GAME USER_losses Stats_per_user losses_per_user SCOREBOARD storage process storage process storage
47. @riferrei | @kafkameetup | @CONFLUENTINC how can I learn more?
48. @riferrei | @kafkameetup | @CONFLUENTINC Get kafka: confluent cloud Try free: https://cnfl.io/confluent-cloud
49. @riferrei | @kafkameetup | @CONFLUENTINC https://cnfl.io/tutorials Get examples: kafka tutorials
50. @riferrei | @kafkameetup | @CONFLUENTINC https://cnfl.io/books Get books: o’reilly bundle
51. @riferrei | @kafkameetup | @CONFLUENTINC https://kafka-summit.org/events/kafka-summit-austin-2020 join kafka summit https://myeventi.events/kafka20/aus Use 25% discount code: KSL20Meetup
52. @riferrei | @kafkameetup | @CONFLUENTINC Thank you