Streaming Data Integration with Apache Kafka Connect @ Graz Kafka Meetup #1

Abstract:

Stream processing gained a lot of traction in recent times and more often than not, we find Apache Kafka - the de facto standard event streaming platform - as the central nervous system of company-wide data architectures. This notwithstanding, many real-world uses cases simply need operational data stores as complementary components to live up to various application-related requirements and challenges. Join this session to learn about how Kafka Connect enables robust integration paths between both worlds. The focus lies on configuration-based data-in-motion scenarios leveraging turn-key ready connector implementations in order to lay out streaming data pipelines without writing a single line of code.

Bio:

Hans-Peter Grahsl is working as a technical trainer at NETCONOMY in Graz, Austria. As an independent engineer & consultant he is working with customers to build on-premise or cloud-based data architectures using NoSQL data stores and event streaming platforms such as Apache Kafka. Hans-Peter is also an associate lecturer for Software Engineering at CAMPUS 02 and occasionally speaks at developer conferences.

Event Page:
https://www.meetup.com/Graz-Kafka/events/265837901/

Recording:
There was no recording of this session.

Transcript

1. Streaming Data Integration with Apache Kafka Connect
2. Hans-Peter Grahsl • technical trainer at • independent consultant & engineer • associate lecturer • ! occasional conference speaker @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 2
3. Apache Kafka @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 3
4. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 4
5. STREAMING Platform @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 5
6. Apache Kafka • pub / sub to event streams • (permanently) store event streams • process streams in near real-time ➔ horizontal scalability ➔ high availability @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 6
7. EVENTS ...events everywhere @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 7
8. APIs for "Everything" @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 8
9. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 9
10. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 10
11. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 11
12. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 12
13. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 13
14. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 14
15. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 15
16. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 16
17. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 17
18. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 18
19. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 19
20. Apache Kafka Connect @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 20
21. Apache Kafka Connect @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 21
22. Apache Kafka Connect @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 22
23. Apache Kafka Connect @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 23
24. Apache Kafka Connect @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 24
25. Apache Kafka Connect @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 25
26. disentangling spaghetti architectures @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 26
27. Apache Kafka Connect • often about data stores @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 27
28. Apache Kafka Connect • concrete examples @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 28
29. Apache Kafka Connect • concrete examples @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 29
30. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 30
31. MongoDB Connector • officially supported by MongoDB • developed open-source on GitHub • verified Gold by Confluent @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 31
32. MongoDB Connector • instead of my community sink https://confluent.io/hub/hpgrahsl/kafka-connect-mongodb • recommendation: use the official one https://confluent.io/hub/mongodb/kafka-connect-mongodb @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 32
33. Source Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 33
34. Source Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 34
35. Change-Data-Capture • react to database changes • INSERTs and UPDATEs • DELETEs (if applicable) @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 35
36. Change-Data-Capture "...is one giant enabler [...] ultimately, it's liberation for your data." — Gunnar Morling @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 36
37. Change-Data-Capture @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 37
38. Change-Data-Capture query-based log-based no changes missed ! ✔ low delay & low polling overhead ! ✔ data model agnostic ! ✔ captures deletions & previous state ! ✔ installation & configuration ✔ ! ! Debezium Blog https://bit.ly/2CRUvxo @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 38
39. @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 39
40. Apache Kafka Source Connectors Demo @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 40
41. ...break apart the silos @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 41
42. to unleash your data @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 42
43. Source Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 43
44. Single Message Transforms • cast types • drop key / value • mask fields • blacklist / whitelist fields • convert timestamps • topic routing • ... @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 44
45. Source Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 45
46. Converters • Kafka only stores bytes • converters do (de)serializations • e.g. String, JSON, Avro, ... @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 46
47. Serialization Formats @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 47
48. Serialization Formats @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 48
49. Serialization Formats @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 49
50. Serialization Formats @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 50
51. Serialization Formats @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 51
52. ❗ Schema Governance "If your dev process doesn't validate schema compatibility somewhere between your IDE and production - you are screwed and don't know it." — Gwen Shapira @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 52
53. "Best bet" currently together with Confluent's Schema Registry @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 53
54. Sink Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 54
55. Sink Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 55
56. Sink Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 56
57. Sink Connectors @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 57
58. Apache Kafka Sink Connectors Demo @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 58
59. Exemplary Use Cases @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 59
60. Customer 360° View @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 60
61. Customer 360° View @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 61
62. Customer 360° View @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 62
63. Customer 360° View @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 63
64. Customer 360° View @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 64
65. Synchronization across Services @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 65
66. Synchronization across Services @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 66
67. Synchronization across Services @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 67
68. Synchronization across Services @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 68
69. Synchronization across Services @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 69
70. Near Real-Time Recommendations @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 70
71. Near Real-Time Recommendations @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 71
72. Near Real-Time Recommendations @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 72
73. Near Real-Time Recommendations @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 73
74. Near Real-Time Recommendations @hpgrahsl | Apache Kafka® Meetup, 27th Nov. 2019, Graz - Austria 74