Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Data Integration with Apache Kafka Connect @ Graz Kafka Meetup #1

Streaming Data Integration with Apache Kafka Connect @ Graz Kafka Meetup #1

Abstract:

Stream processing gained a lot of traction in recent times and more often than not, we find Apache Kafka - the de facto standard event streaming platform - as the central nervous system of company-wide data architectures. This notwithstanding, many real-world uses cases simply need operational data stores as complementary components to live up to various application-related requirements and challenges. Join this session to learn about how Kafka Connect enables robust integration paths between both worlds. The focus lies on configuration-based data-in-motion scenarios leveraging turn-key ready connector implementations in order to lay out streaming data pipelines without writing a single line of code.

Bio:

Hans-Peter Grahsl is working as a technical trainer at NETCONOMY in Graz, Austria. As an independent engineer & consultant he is working with customers to build on-premise or cloud-based data architectures using NoSQL data stores and event streaming platforms such as Apache Kafka. Hans-Peter is also an associate lecturer for Software Engineering at CAMPUS 02 and occasionally speaks at developer conferences.

Event Page:
https://www.meetup.com/Graz-Kafka/events/265837901/

Recording:
There was no recording of this session.

Hans-Peter Grahsl

November 27, 2019
Tweet

More Decks by Hans-Peter Grahsl

Other Decks in Programming

Transcript

  1. Hans-Peter Grahsl • technical trainer at • independent consultant &

    engineer • associate lecturer • ! occasional conference speaker @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 2
  2. Apache Kafka • pub / sub to event streams •

    (permanently) store event streams • process streams in near real-time ➔ horizontal scalability ➔ high availability @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 6
  3. Apache Kafka Connect • often about data stores @hpgrahsl |

    Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 27
  4. MongoDB Connector • officially supported by MongoDB • developed open-source

    on GitHub • verified Gold by Confluent @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 31
  5. MongoDB Connector • instead of my community sink https://confluent.io/hub/hpgrahsl/kafka-connect-mongodb •

    recommendation: use the official one https://confluent.io/hub/mongodb/kafka-connect-mongodb @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 32
  6. Change-Data-Capture • react to database changes • INSERTs and UPDATEs

    • DELETEs (if applicable) @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 35
  7. Change-Data-Capture "...is one giant enabler [...] ultimately, it's liberation for

    your data." — Gunnar Morling @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 36
  8. Change-Data-Capture query-based log-based no changes missed ! ✔ low delay

    & low polling overhead ! ✔ data model agnostic ! ✔ captures deletions & previous state ! ✔ installation & configuration ✔ ! ! Debezium Blog https://bit.ly/2CRUvxo @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 38
  9. Single Message Transforms • cast types • drop key /

    value • mask fields • blacklist / whitelist fields • convert timestamps • topic routing • ... @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 44
  10. Converters • Kafka only stores bytes • converters do (de)serializations

    • e.g. String, JSON, Avro, ... @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 46
  11. ❗ Schema Governance "If your dev process doesn't validate schema

    compatibility somewhere between your IDE and production - you are screwed and don't know it." — Gwen Shapira @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 52
  12. "Best bet" currently together with Confluent's Schema Registry @hpgrahsl |

    Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 53