Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Syncing data between microservices using Debezium and Apache Kafka

Syncing data between microservices using Debezium and Apache Kafka

In this talk, I explained a bit our challenges with data syncing between microservices and how Debezium is helping us to achieve our goal.

Unfortunately, PDFs cannot contain GIFs =/
But you can see the original presentation here:
http://bit.ly/tdc-2018-sync-data-debezium

81f903d136119b188f0fd62d4f180764?s=128

Eizo Nishime

July 20, 2018
Tweet

Transcript

  1. Syncing data between microservices using Debezium and Apache Kafka

  2. Eizo Nishime Software Engineer @ QuintoAndar linkedin.com/in/eizonishime eizo.nishime@gmail.com github.com/japoneizo speakerdeck.com/japoneizo

    quin.to/carreiras #joinus
  3. Schedule • How to scale? • The journey to microservices.

    • Databases (and services) @ QuintoAndar. • Our takes on microservices data integration. • Alternatives. • Debezium might help you.
  4. How to scale? • It's also about how to scale

    your organization.
  5. How to scale? • It's also about how to scale

    your organization. • Encourage ownership.
  6. The journey to microservices

  7. The journey to microservices • Trade-offs everywhere.

  8. The journey to microservices • Trade-offs everywhere. • A big

    part of the problem is your data.
  9. The journey to microservices • Trade-offs everywhere. • A big

    part of the problem is your data. • The boundaries aren't always clear and well defined.
  10. The journey to microservices • Trade-offs everywhere. • A big

    part of the problem is your data. • The boundaries aren't always clear and well defined. • If you won't be careful, you'll just have more problems instead one.
  11. Monolith

  12. Some (micro)services @ QuintoAndar Main (view) Planner Mobile API Tasks

    Us-Emails Accounts Properties Owners M. Control Popis Cyclops Seu barriga Daemons Sherlock Markito Godfather Jaiminho Cidade alerta Charizard
  13. A lot of databases

  14. What if you need to sync data between them?

  15. Our first take

  16. Our first take on microservices data integration • Http Pooling.

  17. Our first take on microservices data integration • Http Pooling.

    • Hibernate event listeners + AWS SQS.
  18. Lessons from our first take • Http pooling had performance

    problems, but allowed to seek through history.
  19. Lessons from our first take • Http pooling had performance

    problems, but allowed to seek through history. • SQS worked fine for some time because we had a few services.
  20. Lessons from our first take • Http pooling had performance

    problems, but allowed to seek through history. • SQS worked fine for some time because we had a few services. • How to scale to more services consume the same message?
  21. Our second take

  22. Our second take • Hibernate event listeners + SNS +

    multiple queues. • Hibernate event listeners + Kafka topics.
  23. We still have some struggling...

  24. More lessons learned • Hibernate entity listeners are application level

    and can fail for many reasons and handling that can be painful.
  25. More lessons learned • Hibernate entity listeners are application level

    and can fail for many reasons and handling that can be painful. • We started to have more sources of truth of different microservices.
  26. More lessons learned • Hibernate entity listeners are application level

    and can fail for many reasons and handling that can be painful. • We started to have more sources of truth of different microservices. • Sometimes when you add a new microservice to the mix, you need to resync and warm up the sync.
  27. More lessons learned • Hibernate entity listeners are application level

    and can fail for many reasons and handling that can be painful. • We started to have more sources of truth of different microservices. • Sometimes when you add a new microservice to the mix, you need to resync and warm up the sync. • Make your consumers idempotent (imagine your consumers like PUT and DELETE http methods).
  28. Sometimes, batch updates directly in database happens.

  29. Our third take • Capture database changes and publish them

    to Kafka topics automatically.
  30. Debezium

  31. Other alternatives • Raw handling bin-log (for MySQL, but Aurora,

    PostgreSQL, MongoDB and MSSQL have something similar). • MySQL-Streamer - Yelp open source tool.
  32. Debezium (debezium.io)

  33. Debezium (debezium.io) • Apache Kafka • Kafka Connect • Docker

    ready
  34. Kafka Connect • confluent.io/produ ct/connectors

  35. MySQL BinLog events and message format

  36. MySQL How Debezium works

  37. MySQL How Debezium works binlog Write Update Delete

  38. MySQL How Debezium works binlog Topics Write Update Delete

  39. MySQL How Debezium works binlog Write Update Delete Topics

  40. How Debezium works • Kafka topics • Debezium will create

    one topic for each table (or collections in MongoDB) in your db.
  41. MySQL How Debezium works binlog Write Update Delete Topics

  42. How Debezium works • You can have workers in your

    microservices to handle the changes.
  43. Debezium connectors • The core of Debezium is about Kafka

    Connect connectors.
  44. Debezium connectors • MySQL • MongoDB • PostgreSQL • Oracle

    • MSSQL (in progress)
  45. MySQL connector example

  46. Links • https://engineeringblog.yelp.com/2016/08/streaming-mysql-tables-in-real-time-to-kafka.html • https://www.wix.engineering/single-post/2017/09/18/The-Cost-of-Streaming-Data-from-MySQL • https://wecode.wepay.com/posts/streaming-databases-in-realtime-with-mysql-debezium-kafka • https://jinyuwang.weebly.com/for-mysql/how-to-enable-binary-logging-for-mysql •

    https://aws.amazon.com/blogs/database/streaming-changes-in-a-database-with-amazon-kinesis/ • https://www.youtube.com/watch?v=IOZ2Um6e430 (Streaming Database Changes with Debezium by Gunnar Morling) • http://blog.christianposta.com/microservices/the-hardest-part-about-microservices-data/
  47. Thanks =)

  48. Eizo Nishime Software Engineer @ QuintoAndar linkedin.com/in/eizonishime eizo.nishime@gmail.com github.com/japoneizo speakerdeck.com/japoneizo

    quin.to/carreiras #joinus