Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Boston Meetup, Fev 2020

Boston Meetup, Fev 2020

Ricardo Ferreira

February 19, 2020
Tweet

More Decks by Ricardo Ferreira

Other Decks in Programming

Transcript

  1. Rediscovering the value of
    apache kafka® in modern
    data architectures
    @riferrei | #kafkameetup | @CONFLUENTINC

    View Slide

  2. About me
    @riferrei | @kafkameetup | @CONFLUENTINC
    • RICARDO FERREIRA
    • Works for confluent
    • Developer advocate
    [email protected]
    • HTTPS://RIFERREI.NET

    View Slide

  3. View Slide

  4. View Slide

  5. Origins of apache kafka
    @riferrei | @kafkameetup | @CONFLUENTINC
    ”there were lots of databases and
    other systems built to store data,
    but what was missing in our
    architecture was something that
    would help us to handle continuous
    flows of data.” – jay kreps

    View Slide

  6. @riferrei | @kafkameetup | @CONFLUENTINC

    View Slide

  7. @riferrei | @kafkameetup | @CONFLUENTINC
    First realization
    >
    I changed my job
    from oracle to
    confluent
    I work at
    confluent
    event state

    View Slide

  8. @riferrei | @kafkameetup | @CONFLUENTINC
    Events are both
    notification State transfer
    +

    View Slide

  9. @riferrei | @kafkameetup | @CONFLUENTINC
    Event-driven application
    Job change recommendation engine
    Search engine
    Email service

    View Slide

  10. @riferrei | @kafkameetup | @CONFLUENTINC
    SQL
    SQL
    SQL
    Recommendation engine
    Search engine
    Email service
    database
    LOG
    Let’s implement this!

    View Slide

  11. @riferrei | @kafkameetup | @CONFLUENTINC
    second realization
    database
    1000x more volume
    Non-transactional events
    Transactional events
    LOG

    View Slide

  12. Databases, 30
    years ago...

    View Slide

  13. Developer
    Databases,
    these days...

    View Slide

  14. @riferrei | @kafkameetup | @CONFLUENTINC
    Databases
    are limited

    View Slide

  15. Limited?
    Are you
    kidding me?

    View Slide

  16. @riferrei | @kafkameetup | @CONFLUENTINC
    ARE DATABASES LIMITED?
    YES, THEY ARE. WHY
    DO WE HAVE TO MOVE
    DATA FROM ONE DB TO
    ANOTHER JUST TO DO
    ANALYTICS?

    View Slide

  17. @riferrei | @kafkameetup | @CONFLUENTINC
    SHARED STATE = MORE DB’S
    Business line 1 Business line 2 Business line 3

    View Slide

  18. @riferrei | @kafkameetup | @CONFLUENTINC
    THIRD REALIZATION
    User
    tracking
    Historical
    data
    Operational
    metrics
    Nosql
    database
    Graph
    database
    Sql
    database
    microservices
    ...
    HADOOP
    Elastic
    search
    grafana
    Machine
    learning
    REC.
    ENGINE SEARCH SECURITY EMAIL
    SOCIAL
    GRAPH

    View Slide

  19. “The truth is the log.
    The database is a cache
    of a subset of the log.”
    — pat helland
    Immutability changes everything
    http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf

    View Slide

  20. @riferrei | @kafkameetup | @CONFLUENTINC
    log as first-class citizen
    database
    LOG
    0 1 2 3 4 5 6 7 8
    LOG
    reads
    writes
    Destination System a
    (time = 1)
    Destination System b
    (time = 3)

    View Slide

  21. @riferrei | @kafkameetup | @CONFLUENTINC
    SOLUTION: BUILD A COMMIT LOG
    Commit LOG
    User
    tracking
    Historical
    data
    Operational
    metrics
    Nosql
    database
    Graph
    database
    Sql
    database
    microservices
    ...
    HADOOP
    Elastic
    search
    grafana
    Machine
    learning
    REC.
    ENGINE SEARCH SECURITY EMAIL
    SOCIAL
    GRAPH

    View Slide

  22. http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
    Do you THINK is a table
    that you RUN QUERIES?

    View Slide

  23. @riferrei | @kafkameetup | @CONFLUENTINC
    STREAMS AND TABLES DUALITY
    {"user":"riferrei","score":"1001"}
    {"user":"riferrei","score":"1002"}
    {"user":"riferrei","score":"1003"}
    {"user":"riferrei","score":"1004"}
    {"user":"riferrei","score":"1005"}
    {"user":"riferrei","score":"1005"}
    stream
    table

    View Slide

  24. Origins of apache kafka
    @riferrei | @kafkameetup | @CONFLUENTINC
    ”WE’VE COME TO THINK OF KAFKA AS A
    STREAMING PLATFORM: A SYSTEM THAT
    LETS YOU PUBLISH AND SUBSCRIBE TO
    STREAMS OF DATA, STORE THEM, AND
    PROCESS THEM, AND THAT IS EXACTLY
    WHAT APACHE KAFKA IS BUILT TO BE.”
    – jay kreps

    View Slide

  25. @riferrei | @kafkameetup | @CONFLUENTINC
    ORIGINS OF APACHE KAFKA
    Databases Messaging
    Batch
    Expensive
    Time Consuming
    Difficult to Scale
    No Persistence After
    Consumption
    No Replay
    Highly Scalable
    Durable
    Persistent
    Ordered
    Fast (Low Latency)

    View Slide

  26. @riferrei | @kafkameetup | @CONFLUENTINC
    ORIGINS OF APACHE KAFKA
    Databases Messaging
    Batch
    Expensive
    Time Consuming
    Difficult to Scale
    No Persistence After
    Consumption
    No Replay
    Highly Scalable
    Durable
    Persistent
    Ordered
    Fast (Low Latency)
    Highly Scalable
    Durable
    Persistent
    Ordered
    Fast (Low Latency)
    Distributed
    Commit log

    View Slide

  27. @riferrei | @kafkameetup | @CONFLUENTINC
    ORIGINS OF APACHE KAFKA
    Databases Messaging
    Batch
    Expensive
    Time Consuming
    Difficult to Scale
    No Persistence After
    Consumption
    No Replay
    Highly Scalable
    Durable
    Persistent
    Ordered
    Fast (Low Latency)
    Highly Scalable
    Durable
    Persistent
    Ordered
    Fast (Low Latency)
    Stream processing
    Continuous flows
    Scalable integration
    Distributed
    Streaming platform

    View Slide

  28. @riferrei | @confluentinc | @itau

    View Slide

  29. Origins of apache kafka
    @riferrei | @kafkameetup | @CONFLUENTINC
    ”the ability to combine these three
    areas – to bring all the streams of
    data together across all the use
    cases – is what makes the idea of a
    streaming platform so appealing
    to people” – jay kreps

    View Slide

  30. 01
    Well done
    messaging
    02
    Durable
    storage
    03
    Stream
    processing
    WHAT IS APACHE KAFKA?

    View Slide

  31. @riferrei | @kafkameetup | @CONFLUENTINC
    Time for some fun
    1. Get the game 2. Name yourself

    View Slide

  32. @riferrei | @KAFKAMEETUP | @CONFLUENTINC
    https://github.com/confluentinc/demo-scene
    <>
    Source-code

    View Slide

  33. @riferrei | @kafkameetup | @CONFLUENTINC
    Source: USER_GAME TOPIC

    View Slide

  34. @riferrei | @kafkameetup | @CONFLUENTINC
    Creating User_game stream

    View Slide

  35. @riferrei | @kafkameetup | @CONFLUENTINC
    Querying USER_GAME STREAM

    View Slide

  36. @riferrei | @kafkameetup | @CONFLUENTINC
    Creating Stats_per_user table

    View Slide

  37. @riferrei | @kafkameetup | @CONFLUENTINC
    Querying STATS_PER_USER TABLE

    View Slide

  38. @riferrei | @kafkameetup | @CONFLUENTINC
    Low latency Pull queries

    View Slide

  39. @riferrei | @kafkameetup | @CONFLUENTINC
    Source: User_losses topic

    View Slide

  40. @riferrei | @kafkameetup | @CONFLUENTINC
    Creating USER_LOSSES STREAM

    View Slide

  41. @riferrei | @kafkameetup | @CONFLUENTINC
    querying USER_LOSSES STREAM

    View Slide

  42. @riferrei | @kafkameetup | @CONFLUENTINC
    Creating LOSSES_PER_USER TABLE

    View Slide

  43. @riferrei | @kafkameetup | @CONFLUENTINC
    Querying LOSSES_PER_USER TABLE

    View Slide

  44. @riferrei | @kafkameetup | @CONFLUENTINC
    Creating SCOREBOARD TABLE

    View Slide

  45. @riferrei | @kafkameetup | @CONFLUENTINC
    Querying SCOREBOARD TABLE

    View Slide

  46. @riferrei | @kafkameetup | @CONFLUENTINC
    Complete scoreboard
    USER_GAME
    USER_losses
    Stats_per_user
    losses_per_user
    SCOREBOARD
    storage process storage process storage

    View Slide

  47. @riferrei | @kafkameetup | @CONFLUENTINC
    how can I
    learn more?

    View Slide

  48. @riferrei | @kafkameetup | @CONFLUENTINC
    Get kafka: confluent cloud
    Try free:
    https://cnfl.io/confluent-cloud

    View Slide

  49. @riferrei | @kafkameetup | @CONFLUENTINC
    https://cnfl.io/tutorials
    Get examples: kafka tutorials

    View Slide

  50. @riferrei | @kafkameetup | @CONFLUENTINC
    https://cnfl.io/books
    Get books: o’reilly bundle

    View Slide

  51. @riferrei | @kafkameetup | @CONFLUENTINC
    https://kafka-summit.org/events/kafka-summit-austin-2020
    join kafka summit
    https://myeventi.events/kafka20/aus
    Use 25% discount code: KSL20Meetup

    View Slide

  52. @riferrei | @kafkameetup | @CONFLUENTINC
    Thank you

    View Slide