Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kafka and Debezium at trivago | Code.Talks 2017 edition

Mario Mueller
September 29, 2017

Kafka and Debezium at trivago | Code.Talks 2017 edition

Video available: https://www.youtube.com/watch?v=cU0BCVl4bjo

Nowadays everyone solves the separation and modularisation of big monolithic legacy applications and databases with HTTP microservices. We at trivago take another approach using Apache Kafka, Debezium and Stream processing. This talk shows how we do this and why we are convinced, that this is a better way to enhance your architecture, why this exceeds HTTP microservices and is more then 1 million times faster.

Mario Mueller

September 29, 2017
Tweet

More Decks by Mario Mueller

Other Decks in Technology

Transcript

  1. Apache Kafka and Debezium
    at

    View full-size slide

  2. Hello!
    René Kerner
    Software-Architect
    Stream Platform Services
    @rk3n3r
    Mario Müller
    Team Lead
    Stream Platform Services
    @xenji
    http://muppethub.com/wp-content/uploads/2014/02/Statler-and-Waldorf-1.png

    View full-size slide

  3. Who of you uses Apache Kafka
    in production?
    From: http://www.inwardconsulting.com/how-we-think/blog/taking-the-vote-to-the-office/

    View full-size slide

  4. Who of you uses Debezium in production?
    From: http://www.inwardconsulting.com/how-we-think/blog/taking-the-vote-to-the-office/

    View full-size slide

  5. A little bit of technological context
    • We heavily make use of Percona Server
    • We use Redis, Cassandra, Memcached, Elasticsearch, ...
    • Four on-premise datacenters (DUS, SFO, DCA, HGK)
    • Some projects in public cloud environments (AWS & GCP)
    • We use PHP, Java, Go, Python, Kotlin, JavaScript, ...

    View full-size slide

  6. Kafka at trivago
    • Applications
    • Clickstream and application log transport
    • Change set Streams, Transport + Storage
    • Stream Processing
    • Numbers
    • 28 production servers, spread over four datacenters
    • 9 TB permanent data
    • 200 Mbit/s peak
    • 240k msg/s peak

    View full-size slide

  7. Debezium at trivago
    • Debezium
    • Is a RedHat sponsored OSS project
    • Supports MySQL, PostgreSQL, Oracle and MongoDB
    • Ingests the change events into Kafka topics
    • Use-cases for us
    • Cross-technology and routed replication
    • Decouple writer (producer) and reader (consumer)
    • Stream Processing

    View full-size slide

  8. Where do we come from?
    • Company Structure
    • Matrix organisation
    • Horizontal value chain through multiple teams
    (e. g. Advertiser → Content → Tech → Marketing → User)
    • Tech
    • One big PHP app, two big Java apps
    • Central MySQL schema for all with an unknown amount of readers and
    writers
    • ~120 employees (~20 tech) 2011, now >1300 employees (> 300 tech)
    • 1 dev location until 2012, 4 dev locations now

    View full-size slide

  9. “Any organization that designs a system (defined broadly) will produce a design whose structure
    is a copy of the organization's communication structure” - Conway’s law

    View full-size slide

  10. https://fsmedia.imgix.net/45/3f/ed/06/84b7/4d42/b51a/40681c42eac3/picard-facepalmjpg.jpeg

    View full-size slide

  11. HTTP Microservices
    Are they really solving the problem?

    View full-size slide

  12. What is so hard about HTTP APIs?
    RyW - Read your own writes MR - Monotonic Reads
    MW - Monotonic Writes WfR - Write follows Read

    View full-size slide

  13. What is the result?
    ● Invalid caches due to race
    conditions
    ● How to solve?
    • Global Locks
    => Moves latency to the client
    • (Software) Transactions
    => Tons of complexity
    from: confluent.io

    View full-size slide

  14. https://pbs.twimg.com/profile_images/1927177125/image.jpg

    View full-size slide

  15. Apache Kafka
    Reactive Microservice Architecture

    View full-size slide

  16. Kafka Glossary
    ● CDC = Change Data Capturing
    ● Kafka Cluster, Broker, Producer, Consumer
    ● Source = Datastore to Kafka Message producer
    ● Sink = Kafka Message to Datastore writer
    ● SP = Stream Processor (Kafka Streams API)

    View full-size slide

  17. Kafka Basic Internals
    ● Stream = Topic (a stream of
    records)
    ● Record = Message = Event
    ● One record consists of a key,
    a value and a timestamp
    ● 1 Topic can have “n”
    partitions (scaling)

    View full-size slide

  18. Kafka Characteristics
    ● Distributed Streaming Platform: Publish & subscribe, process and store
    ● Fault-tolerant and scalable (partitioning, replication)
    ● Read: At-Least-Once-Delivery, Exactly-Once-Delivery (v0.11)
    ● Tuneable write consistency (no messages/events get lost)

    View full-size slide

  19. Kafka Characteristics
    ● Reactive: Change Events, Long-Polling consumers, Kafka Streams
    API for joining, filtering, aggregation
    ● Unidirectional data flows => no race conditions (or at least easy
    manageable)
    ● Strictly ordered per partition
    ● Log compaction by: Time, Size or Key Compaction

    View full-size slide

  20. Kafka as write API
    MySQL Kafka Topic
    Debezium
    Protobuf
    Converter
    App 1
    App 2
    App 3
    Source

    View full-size slide

  21. Kafka as read API
    Kafka Topic
    MySQL Redis
    MySQL
    Elastic-
    Search
    Debezium
    Protobuf
    Converter
    App 1
    App 2
    App 3
    Cassandra
    Sink
    Adapter
    Protobuf
    Converter
    App 1
    App 4
    Sink

    View full-size slide

  22. Our solution approach

    View full-size slide

  23. Change Data Capturing (CDC) with Debezium
    ● Change Capture the big central MySQL
    database
    ● Externalise the schema using Google
    Protocol Buffers
    ● Debezium reads from MySQL
    replication binlog (row replication)
    => Make each table available as stream of
    changesets
    from: confluent.io

    View full-size slide

  24. Stream Processors
    ● Generate efficient views
    ● Join different streams
    ● Filter, map and aggregate data
    ● Optimize data + structure for your
    application needs
    ● Create pre-calculated data before
    hitting the datastore/s
    ● Performance benefit (unfair):
    >1 mio. times faster
    from: confluent.io

    View full-size slide

  25. ● Ownership of data
    ● Focus on expertise for data inside the
    bounded context
    ● Deliver best data quality
    ● “Hunter not hunted”
    ● Independence (data usage/ownership/scaling)
    ● Focus on freely combining data
    • Gain new knowledge
    ● Filter data
    ● Join and aggregate
    ● Focus on consumer / application needs

    View full-size slide

  26. http://i0.kym-cdn.com/photos/images/original/000/049/657/23icsb8.jpg

    View full-size slide

  27. Q & A
    https://muppetmindset.files.wordpress.com/2016/05/question-mark.jpg

    View full-size slide

  28. Links and notes
    http://www.dbms2.com/2010/05/01/ryw-read-your-writes-consistency/
    http://www.allthingsdistributed.com/2007/12/eventually_consistent.html
    http://www.cs.cornell.edu/courses/cs734/2000FA/cached%20papers/SessionGuaranteesPDIS_1.html
    https://www.slideshare.net/ConfluentInc/capture-the-streams-of-database-changes
    http://www.melconway.com/Home/Conways_Law.html

    View full-size slide