Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kafka and Debezium at trivago | Code.Talks 2017 edition

Mario Mueller
September 29, 2017

Kafka and Debezium at trivago | Code.Talks 2017 edition

Video available: https://www.youtube.com/watch?v=cU0BCVl4bjo

Nowadays everyone solves the separation and modularisation of big monolithic legacy applications and databases with HTTP microservices. We at trivago take another approach using Apache Kafka, Debezium and Stream processing. This talk shows how we do this and why we are convinced, that this is a better way to enhance your architecture, why this exceeds HTTP microservices and is more then 1 million times faster.

Mario Mueller

September 29, 2017

More Decks by Mario Mueller

Other Decks in Technology


  1. Hello! René Kerner Software-Architect Stream Platform Services @rk3n3r Mario Müller

    Team Lead Stream Platform Services @xenji http://muppethub.com/wp-content/uploads/2014/02/Statler-and-Waldorf-1.png
  2. A little bit of technological context • We heavily make

    use of Percona Server • We use Redis, Cassandra, Memcached, Elasticsearch, ... • Four on-premise datacenters (DUS, SFO, DCA, HGK) • Some projects in public cloud environments (AWS & GCP) • We use PHP, Java, Go, Python, Kotlin, JavaScript, ...
  3. Kafka at trivago • Applications • Clickstream and application log

    transport • Change set Streams, Transport + Storage • Stream Processing • Numbers • 28 production servers, spread over four datacenters • 9 TB permanent data • 200 Mbit/s peak • 240k msg/s peak
  4. Debezium at trivago • Debezium • Is a RedHat sponsored

    OSS project • Supports MySQL, PostgreSQL, Oracle and MongoDB • Ingests the change events into Kafka topics • Use-cases for us • Cross-technology and routed replication • Decouple writer (producer) and reader (consumer) • Stream Processing
  5. Where do we come from? • Company Structure • Matrix

    organisation • Horizontal value chain through multiple teams (e. g. Advertiser → Content → Tech → Marketing → User) • Tech • One big PHP app, two big Java apps • Central MySQL schema for all with an unknown amount of readers and writers • ~120 employees (~20 tech) 2011, now >1300 employees (> 300 tech) • 1 dev location until 2012, 4 dev locations now
  6. “Any organization that designs a system (defined broadly) will produce

    a design whose structure is a copy of the organization's communication structure” - Conway’s law
  7. What is so hard about HTTP APIs? RyW - Read

    your own writes MR - Monotonic Reads MW - Monotonic Writes WfR - Write follows Read
  8. What is the result? • Invalid caches due to race

    conditions • How to solve? • Global Locks => Moves latency to the client • (Software) Transactions => Tons of complexity from: confluent.io
  9. Kafka Glossary • CDC = Change Data Capturing • Kafka

    Cluster, Broker, Producer, Consumer • Source = Datastore to Kafka Message producer • Sink = Kafka Message to Datastore writer • SP = Stream Processor (Kafka Streams API)
  10. Kafka Basic Internals • Stream = Topic (a stream of

    records) • Record = Message = Event • One record consists of a key, a value and a timestamp • 1 Topic can have “n” partitions (scaling)
  11. Kafka Characteristics • Distributed Streaming Platform: Publish & subscribe, process

    and store • Fault-tolerant and scalable (partitioning, replication) • Read: At-Least-Once-Delivery, Exactly-Once-Delivery (v0.11) • Tuneable write consistency (no messages/events get lost)
  12. Kafka Characteristics • Reactive: Change Events, Long-Polling consumers, Kafka Streams

    API for joining, filtering, aggregation • Unidirectional data flows => no race conditions (or at least easy manageable) • Strictly ordered per partition • Log compaction by: Time, Size or Key Compaction
  13. Kafka as read API Kafka Topic MySQL Redis MySQL Elastic-

    Search Debezium Protobuf Converter App 1 App 2 App 3 Cassandra Sink Adapter Protobuf Converter App 1 App 4 Sink
  14. Change Data Capturing (CDC) with Debezium • Change Capture the

    big central MySQL database • Externalise the schema using Google Protocol Buffers • Debezium reads from MySQL replication binlog (row replication) => Make each table available as stream of changesets from: confluent.io
  15. Stream Processors • Generate efficient views • Join different streams

    • Filter, map and aggregate data • Optimize data + structure for your application needs • Create pre-calculated data before hitting the datastore/s • Performance benefit (unfair): >1 mio. times faster from: confluent.io
  16. • Ownership of data • Focus on expertise for data

    inside the bounded context • Deliver best data quality • “Hunter not hunted” • Independence (data usage/ownership/scaling) • Focus on freely combining data • Gain new knowledge • Filter data • Join and aggregate • Focus on consumer / application needs