Kafka and Debezium at trivago | Code.Talks 2017 edition

Video available: https://www.youtube.com/watch?v=cU0BCVl4bjo

Nowadays everyone solves the separation and modularisation of big monolithic legacy applications and databases with HTTP microservices. We at trivago take another approach using Apache Kafka, Debezium and Stream processing. This talk shows how we do this and why we are convinced, that this is a better way to enhance your architecture, why this exceeds HTTP microservices and is more then 1 million times faster.

Transcript

1. Apache Kafka and Debezium
   at
   

   View Slide

2. Hello!
   René Kerner
   Software-Architect
   Stream Platform Services
   @rk3n3r
   Mario Müller
   Team Lead
   Stream Platform Services
   @xenji
   http://muppethub.com/wp-content/uploads/2014/02/Statler-and-Waldorf-1.png
   

   View Slide

3. Who of you uses Apache Kafka
   in production?
   From: http://www.inwardconsulting.com/how-we-think/blog/taking-the-vote-to-the-office/
   

   View Slide

4. Who of you uses Debezium in production?
   From: http://www.inwardconsulting.com/how-we-think/blog/taking-the-vote-to-the-office/
   

   View Slide

5. A little bit of technological context
   • We heavily make use of Percona Server
   • We use Redis, Cassandra, Memcached, Elasticsearch, ...
   • Four on-premise datacenters (DUS, SFO, DCA, HGK)
   • Some projects in public cloud environments (AWS & GCP)
   • We use PHP, Java, Go, Python, Kotlin, JavaScript, ...
   

   View Slide

6. Kafka at trivago
   • Applications
   • Clickstream and application log transport
   • Change set Streams, Transport + Storage
   • Stream Processing
   • Numbers
   • 28 production servers, spread over four datacenters
   • 9 TB permanent data
   • 200 Mbit/s peak
   • 240k msg/s peak
   

   View Slide

7. Debezium at trivago
   • Debezium
   • Is a RedHat sponsored OSS project
   • Supports MySQL, PostgreSQL, Oracle and MongoDB
   • Ingests the change events into Kafka topics
   • Use-cases for us
   • Cross-technology and routed replication
   • Decouple writer (producer) and reader (consumer)
   • Stream Processing
   

   View Slide

8. History
   

   View Slide

9. Where do we come from?
   • Company Structure
   • Matrix organisation
   • Horizontal value chain through multiple teams
   (e. g. Advertiser → Content → Tech → Marketing → User)
   • Tech
   • One big PHP app, two big Java apps
   • Central MySQL schema for all with an unknown amount of readers and
   writers
   • ~120 employees (~20 tech) 2011, now >1300 employees (> 300 tech)
   • 1 dev location until 2012, 4 dev locations now
   

   View Slide

10. “Any organization that designs a system (defined broadly) will produce a design whose structure
    is a copy of the organization's communication structure” - Conway’s law
    

    View Slide

11. View Slide

12. https://fsmedia.imgix.net/45/3f/ed/06/84b7/4d42/b51a/40681c42eac3/picard-facepalmjpg.jpeg
    

    View Slide

13. HTTP Microservices
    Are they really solving the problem?
    

    View Slide

14. What is so hard about HTTP APIs?
    RyW - Read your own writes MR - Monotonic Reads
    MW - Monotonic Writes WfR - Write follows Read
    

    View Slide

15. What is the result?
    ● Invalid caches due to race
    conditions
    ● How to solve?
    • Global Locks
    => Moves latency to the client
    • (Software) Transactions
    => Tons of complexity
    from: confluent.io
    

    View Slide

16. https://pbs.twimg.com/profile_images/1927177125/image.jpg
    

    View Slide

17. Apache Kafka
    Reactive Microservice Architecture
    

    View Slide

18. Kafka Glossary
    ● CDC = Change Data Capturing
    ● Kafka Cluster, Broker, Producer, Consumer
    ● Source = Datastore to Kafka Message producer
    ● Sink = Kafka Message to Datastore writer
    ● SP = Stream Processor (Kafka Streams API)
    

    View Slide

19. Kafka Basic Internals
    ● Stream = Topic (a stream of
    records)
    ● Record = Message = Event
    ● One record consists of a key,
    a value and a timestamp
    ● 1 Topic can have “n”
    partitions (scaling)
    

    View Slide

20. Kafka Characteristics
    ● Distributed Streaming Platform: Publish & subscribe, process and store
    ● Fault-tolerant and scalable (partitioning, replication)
    ● Read: At-Least-Once-Delivery, Exactly-Once-Delivery (v0.11)
    ● Tuneable write consistency (no messages/events get lost)
    

    View Slide

21. Kafka Characteristics
    ● Reactive: Change Events, Long-Polling consumers, Kafka Streams
    API for joining, filtering, aggregation
    ● Unidirectional data flows => no race conditions (or at least easy
    manageable)
    ● Strictly ordered per partition
    ● Log compaction by: Time, Size or Key Compaction
    

    View Slide

22. Kafka as write API
    MySQL Kafka Topic
    Debezium
    Protobuf
    Converter
    App 1
    App 2
    App 3
    Source
    

    View Slide

23. Kafka as read API
    Kafka Topic
    MySQL Redis
    MySQL
    Elastic-
    Search
    Debezium
    Protobuf
    Converter
    App 1
    App 2
    App 3
    Cassandra
    Sink
    Adapter
    Protobuf
    Converter
    App 1
    App 4
    Sink
    

    View Slide

24. Our solution approach
    

    View Slide

25. Change Data Capturing (CDC) with Debezium
    ● Change Capture the big central MySQL
    database
    ● Externalise the schema using Google
    Protocol Buffers
    ● Debezium reads from MySQL
    replication binlog (row replication)
    => Make each table available as stream of
    changesets
    from: confluent.io
    

    View Slide

26. Stream Processors
    ● Generate efficient views
    ● Join different streams
    ● Filter, map and aggregate data
    ● Optimize data + structure for your
    application needs
    ● Create pre-calculated data before
    hitting the datastore/s
    ● Performance benefit (unfair):
    >1 mio. times faster
    from: confluent.io
    

    View Slide

27. ● Ownership of data
    ● Focus on expertise for data inside the
    bounded context
    ● Deliver best data quality
    ● “Hunter not hunted”
    ● Independence (data usage/ownership/scaling)
    ● Focus on freely combining data
    • Gain new knowledge
    ● Filter data
    ● Join and aggregate
    ● Focus on consumer / application needs
    

    View Slide

28. http://i0.kym-cdn.com/photos/images/original/000/049/657/23icsb8.jpg
    

    View Slide

29. Thank you!
    

    View Slide

30. Q & A
    https://muppetmindset.files.wordpress.com/2016/05/question-mark.jpg
    

    View Slide

31. Links and notes
    http://www.dbms2.com/2010/05/01/ryw-read-your-writes-consistency/
    http://www.allthingsdistributed.com/2007/12/eventually_consistent.html
    http://www.cs.cornell.edu/courses/cs734/2000FA/cached%20papers/SessionGuaranteesPDIS_1.html
    https://www.slideshare.net/ConfluentInc/capture-the-streams-of-database-changes
    http://www.melconway.com/Home/Conways_Law.html
    

    View Slide