Boston Meetup, Fev 2020

Transcript

1. Rediscovering the value of
   apache kafka® in modern
   data architectures
   @riferrei | #kafkameetup | @CONFLUENTINC
   

   View Slide

2. About me
   @riferrei | @kafkameetup | @CONFLUENTINC
   • RICARDO FERREIRA
   • Works for confluent
   • Developer advocate
   • Ricardo@confluent.iO
   • HTTPS://RIFERREI.NET
   

   View Slide

3. View Slide

4. View Slide

5. Origins of apache kafka
   @riferrei | @kafkameetup | @CONFLUENTINC
   ”there were lots of databases and
   other systems built to store data,
   but what was missing in our
   architecture was something that
   would help us to handle continuous
   flows of data.” – jay kreps
   

   View Slide

6. @riferrei | @kafkameetup | @CONFLUENTINC
   

   View Slide

7. @riferrei | @kafkameetup | @CONFLUENTINC
   First realization
   >
   I changed my job
   from oracle to
   confluent
   I work at
   confluent
   event state
   

   View Slide

8. @riferrei | @kafkameetup | @CONFLUENTINC
   Events are both
   notification State transfer
   +
   

   View Slide

9. @riferrei | @kafkameetup | @CONFLUENTINC
   Event-driven application
   Job change recommendation engine
   Search engine
   Email service
   

   View Slide

10. @riferrei | @kafkameetup | @CONFLUENTINC
    SQL
    SQL
    SQL
    Recommendation engine
    Search engine
    Email service
    database
    LOG
    Let’s implement this!
    

    View Slide

11. @riferrei | @kafkameetup | @CONFLUENTINC
    second realization
    database
    1000x more volume
    Non-transactional events
    Transactional events
    LOG
    

    View Slide

12. Databases, 30
    years ago...
    

    View Slide

13. Developer
    Databases,
    these days...
    

    View Slide

14. @riferrei | @kafkameetup | @CONFLUENTINC
    Databases
    are limited
    

    View Slide

15. Limited?
    Are you
    kidding me?
    

    View Slide

16. @riferrei | @kafkameetup | @CONFLUENTINC
    ARE DATABASES LIMITED?
    YES, THEY ARE. WHY
    DO WE HAVE TO MOVE
    DATA FROM ONE DB TO
    ANOTHER JUST TO DO
    ANALYTICS?
    

    View Slide

17. @riferrei | @kafkameetup | @CONFLUENTINC
    SHARED STATE = MORE DB’S
    Business line 1 Business line 2 Business line 3
    

    View Slide

18. @riferrei | @kafkameetup | @CONFLUENTINC
    THIRD REALIZATION
    User
    tracking
    Historical
    data
    Operational
    metrics
    Nosql
    database
    Graph
    database
    Sql
    database
    microservices
    ...
    HADOOP
    Elastic
    search
    grafana
    Machine
    learning
    REC.
    ENGINE SEARCH SECURITY EMAIL
    SOCIAL
    GRAPH
    

    View Slide

19. “The truth is the log.
    The database is a cache
    of a subset of the log.”
    — pat helland
    Immutability changes everything
    http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
    

    View Slide

20. @riferrei | @kafkameetup | @CONFLUENTINC
    log as first-class citizen
    database
    LOG
    0 1 2 3 4 5 6 7 8
    LOG
    reads
    writes
    Destination System a
    (time = 1)
    Destination System b
    (time = 3)
    

    View Slide

21. @riferrei | @kafkameetup | @CONFLUENTINC
    SOLUTION: BUILD A COMMIT LOG
    Commit LOG
    User
    tracking
    Historical
    data
    Operational
    metrics
    Nosql
    database
    Graph
    database
    Sql
    database
    microservices
    ...
    HADOOP
    Elastic
    search
    grafana
    Machine
    learning
    REC.
    ENGINE SEARCH SECURITY EMAIL
    SOCIAL
    GRAPH
    

    View Slide

22. http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
    Do you THINK is a table
    that you RUN QUERIES?
    

    View Slide

23. @riferrei | @kafkameetup | @CONFLUENTINC
    STREAMS AND TABLES DUALITY
    {"user":"riferrei","score":"1001"}
    {"user":"riferrei","score":"1002"}
    {"user":"riferrei","score":"1003"}
    {"user":"riferrei","score":"1004"}
    {"user":"riferrei","score":"1005"}
    {"user":"riferrei","score":"1005"}
    stream
    table
    

    View Slide

24. Origins of apache kafka
    @riferrei | @kafkameetup | @CONFLUENTINC
    ”WE’VE COME TO THINK OF KAFKA AS A
    STREAMING PLATFORM: A SYSTEM THAT
    LETS YOU PUBLISH AND SUBSCRIBE TO
    STREAMS OF DATA, STORE THEM, AND
    PROCESS THEM, AND THAT IS EXACTLY
    WHAT APACHE KAFKA IS BUILT TO BE.”
    – jay kreps
    

    View Slide

25. @riferrei | @kafkameetup | @CONFLUENTINC
    ORIGINS OF APACHE KAFKA
    Databases Messaging
    Batch
    Expensive
    Time Consuming
    Difficult to Scale
    No Persistence After
    Consumption
    No Replay
    Highly Scalable
    Durable
    Persistent
    Ordered
    Fast (Low Latency)
    

    View Slide

26. @riferrei | @kafkameetup | @CONFLUENTINC
    ORIGINS OF APACHE KAFKA
    Databases Messaging
    Batch
    Expensive
    Time Consuming
    Difficult to Scale
    No Persistence After
    Consumption
    No Replay
    Highly Scalable
    Durable
    Persistent
    Ordered
    Fast (Low Latency)
    Highly Scalable
    Durable
    Persistent
    Ordered
    Fast (Low Latency)
    Distributed
    Commit log
    

    View Slide

27. @riferrei | @kafkameetup | @CONFLUENTINC
    ORIGINS OF APACHE KAFKA
    Databases Messaging
    Batch
    Expensive
    Time Consuming
    Difficult to Scale
    No Persistence After
    Consumption
    No Replay
    Highly Scalable
    Durable
    Persistent
    Ordered
    Fast (Low Latency)
    Highly Scalable
    Durable
    Persistent
    Ordered
    Fast (Low Latency)
    Stream processing
    Continuous flows
    Scalable integration
    Distributed
    Streaming platform
    

    View Slide

28. @riferrei | @confluentinc | @itau
    

    View Slide

29. Origins of apache kafka
    @riferrei | @kafkameetup | @CONFLUENTINC
    ”the ability to combine these three
    areas – to bring all the streams of
    data together across all the use
    cases – is what makes the idea of a
    streaming platform so appealing
    to people” – jay kreps
    

    View Slide

30. 01
    Well done
    messaging
    02
    Durable
    storage
    03
    Stream
    processing
    WHAT IS APACHE KAFKA?
    

    View Slide

31. @riferrei | @kafkameetup | @CONFLUENTINC
    Time for some fun
    1. Get the game 2. Name yourself
    

    View Slide

32. @riferrei | @KAFKAMEETUP | @CONFLUENTINC
    https://github.com/confluentinc/demo-scene
    <>
    Source-code
    

    View Slide

33. @riferrei | @kafkameetup | @CONFLUENTINC
    Source: USER_GAME TOPIC
    

    View Slide

34. @riferrei | @kafkameetup | @CONFLUENTINC
    Creating User_game stream
    

    View Slide

35. @riferrei | @kafkameetup | @CONFLUENTINC
    Querying USER_GAME STREAM
    

    View Slide

36. @riferrei | @kafkameetup | @CONFLUENTINC
    Creating Stats_per_user table
    

    View Slide

37. @riferrei | @kafkameetup | @CONFLUENTINC
    Querying STATS_PER_USER TABLE
    

    View Slide

38. @riferrei | @kafkameetup | @CONFLUENTINC
    Low latency Pull queries
    

    View Slide

39. @riferrei | @kafkameetup | @CONFLUENTINC
    Source: User_losses topic
    

    View Slide

40. @riferrei | @kafkameetup | @CONFLUENTINC
    Creating USER_LOSSES STREAM
    

    View Slide

41. @riferrei | @kafkameetup | @CONFLUENTINC
    querying USER_LOSSES STREAM
    

    View Slide

42. @riferrei | @kafkameetup | @CONFLUENTINC
    Creating LOSSES_PER_USER TABLE
    

    View Slide

43. @riferrei | @kafkameetup | @CONFLUENTINC
    Querying LOSSES_PER_USER TABLE
    

    View Slide

44. @riferrei | @kafkameetup | @CONFLUENTINC
    Creating SCOREBOARD TABLE
    

    View Slide

45. @riferrei | @kafkameetup | @CONFLUENTINC
    Querying SCOREBOARD TABLE
    

    View Slide

46. @riferrei | @kafkameetup | @CONFLUENTINC
    Complete scoreboard
    USER_GAME
    USER_losses
    Stats_per_user
    losses_per_user
    SCOREBOARD
    storage process storage process storage
    

    View Slide

47. @riferrei | @kafkameetup | @CONFLUENTINC
    how can I
    learn more?
    

    View Slide

48. @riferrei | @kafkameetup | @CONFLUENTINC
    Get kafka: confluent cloud
    Try free:
    https://cnfl.io/confluent-cloud
    

    View Slide

49. @riferrei | @kafkameetup | @CONFLUENTINC
    https://cnfl.io/tutorials
    Get examples: kafka tutorials
    

    View Slide

50. @riferrei | @kafkameetup | @CONFLUENTINC
    https://cnfl.io/books
    Get books: o’reilly bundle
    

    View Slide

51. @riferrei | @kafkameetup | @CONFLUENTINC
    https://kafka-summit.org/events/kafka-summit-austin-2020
    join kafka summit
    https://myeventi.events/kafka20/aus
    Use 25% discount code: KSL20Meetup
    

    View Slide

52. @riferrei | @kafkameetup | @CONFLUENTINC
    Thank you
    

    View Slide