Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Data Integration with Apache Kafka Connect @ Graz Kafka Meetup #1

Streaming Data Integration with Apache Kafka Connect @ Graz Kafka Meetup #1

Abstract:

Stream processing gained a lot of traction in recent times and more often than not, we find Apache Kafka - the de facto standard event streaming platform - as the central nervous system of company-wide data architectures. This notwithstanding, many real-world uses cases simply need operational data stores as complementary components to live up to various application-related requirements and challenges. Join this session to learn about how Kafka Connect enables robust integration paths between both worlds. The focus lies on configuration-based data-in-motion scenarios leveraging turn-key ready connector implementations in order to lay out streaming data pipelines without writing a single line of code.

Bio:

Hans-Peter Grahsl is working as a technical trainer at NETCONOMY in Graz, Austria. As an independent engineer & consultant he is working with customers to build on-premise or cloud-based data architectures using NoSQL data stores and event streaming platforms such as Apache Kafka. Hans-Peter is also an associate lecturer for Software Engineering at CAMPUS 02 and occasionally speaks at developer conferences.

Event Page:
https://www.meetup.com/Graz-Kafka/events/265837901/

Recording:
There was no recording of this session.

744f1c2c6cbea2ff5104b0ac512936bd?s=128

Hans-Peter Grahsl

November 27, 2019
Tweet

Transcript

  1. Streaming Data Integration with Apache Kafka Connect

  2. Hans-Peter Grahsl • technical trainer at • independent consultant &

    engineer • associate lecturer • ! occasional conference speaker @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 2
  3. Apache Kafka @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 3
  4. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 4
  5. STREAMING Platform @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 5
  6. Apache Kafka • pub / sub to event streams •

    (permanently) store event streams • process streams in near real-time ➔ horizontal scalability ➔ high availability @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 6
  7. EVENTS ...events everywhere @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 7
  8. APIs for "Everything" @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 8
  9. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 9
  10. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 10
  11. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 11
  12. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 12
  13. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 13
  14. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 14
  15. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 15
  16. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 16
  17. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 17
  18. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 18
  19. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 19
  20. Apache Kafka Connect @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 20
  21. Apache Kafka Connect @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 21
  22. Apache Kafka Connect @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 22
  23. Apache Kafka Connect @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 23
  24. Apache Kafka Connect @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 24
  25. Apache Kafka Connect @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 25
  26. disentangling spaghetti architectures @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 26
  27. Apache Kafka Connect • often about data stores @hpgrahsl |

    Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 27
  28. Apache Kafka Connect • concrete examples @hpgrahsl | Apache Kafka®

    Meetup, 27th Nov. 2019, Graz - Austria 28
  29. Apache Kafka Connect • concrete examples @hpgrahsl | Apache Kafka®

    Meetup, 27th Nov. 2019, Graz - Austria 29
  30. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 30
  31. MongoDB Connector • officially supported by MongoDB • developed open-source

    on GitHub • verified Gold by Confluent @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 31
  32. MongoDB Connector • instead of my community sink https://confluent.io/hub/hpgrahsl/kafka-connect-mongodb •

    recommendation: use the official one https://confluent.io/hub/mongodb/kafka-connect-mongodb @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 32
  33. Source Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 33
  34. Source Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 34
  35. Change-Data-Capture • react to database changes • INSERTs and UPDATEs

    • DELETEs (if applicable) @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 35
  36. Change-Data-Capture "...is one giant enabler [...] ultimately, it's liberation for

    your data." — Gunnar Morling @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 36
  37. Change-Data-Capture @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz

    - Austria 37
  38. Change-Data-Capture query-based log-based no changes missed ! ✔ low delay

    & low polling overhead ! ✔ data model agnostic ! ✔ captures deletions & previous state ! ✔ installation & configuration ✔ ! ! Debezium Blog https://bit.ly/2CRUvxo @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 38
  39. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz -

    Austria 39
  40. Apache Kafka Source Connectors Demo @hpgrahsl | Apache Kafka® Meetup,

    27th Nov. 2019, Graz - Austria 40
  41. ...break apart the silos @hpgrahsl | Apache Kafka® Meetup, 27th

    Nov. 2019, Graz - Austria 41
  42. to unleash your data @hpgrahsl | Apache Kafka® Meetup, 27th

    Nov. 2019, Graz - Austria 42
  43. Source Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 43
  44. Single Message Transforms • cast types • drop key /

    value • mask fields • blacklist / whitelist fields • convert timestamps • topic routing • ... @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 44
  45. Source Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 45
  46. Converters • Kafka only stores bytes • converters do (de)serializations

    • e.g. String, JSON, Avro, ... @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 46
  47. Serialization Formats @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 47
  48. Serialization Formats @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 48
  49. Serialization Formats @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 49
  50. Serialization Formats @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 50
  51. Serialization Formats @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 51
  52. ❗ Schema Governance "If your dev process doesn't validate schema

    compatibility somewhere between your IDE and production - you are screwed and don't know it." — Gwen Shapira @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 52
  53. "Best bet" currently together with Confluent's Schema Registry @hpgrahsl |

    Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 53
  54. Sink Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 54
  55. Sink Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 55
  56. Sink Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 56
  57. Sink Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019,

    Graz - Austria 57
  58. Apache Kafka Sink Connectors Demo @hpgrahsl | Apache Kafka® Meetup,

    27th Nov. 2019, Graz - Austria 58
  59. Exemplary Use Cases @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 59
  60. Customer 360° View @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 60
  61. Customer 360° View @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 61
  62. Customer 360° View @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 62
  63. Customer 360° View @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 63
  64. Customer 360° View @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 64
  65. Synchronization across Services @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 65
  66. Synchronization across Services @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 66
  67. Synchronization across Services @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 67
  68. Synchronization across Services @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 68
  69. Synchronization across Services @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 69
  70. Near Real-Time Recommendations @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 70
  71. Near Real-Time Recommendations @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 71
  72. Near Real-Time Recommendations @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 72
  73. Near Real-Time Recommendations @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 73
  74. Near Real-Time Recommendations @hpgrahsl | Apache Kafka® Meetup, 27th Nov.

    2019, Graz - Austria 74
  75. None
  76. None