Real-time Data Integration at Scale with Kafka Connect - Dublin Apache Kafka Meetup 04 Jul 2017

Apache Kafka is a streaming data platform. It enables integration of data across the enterprise, and ships with its own stream processing capabilities. But how do we get data in and out of Kafka in an easy, scalable, and standardised manner? Enter Kafka Connect.
Part of Apache Kafka since 0.9, Kafka Connect defines an API that enables the integration of data from multiple sources, including MQTT, common NoSQL stores, and CDC from relational databases such as Oracle. By "turning the database inside out" we can enable an event-driven architecture in our business that reacts to changes made by applications writing to a database, without having to modify those applications themselves. As well as ingest, Kafka Connect has connectors with support for numerous targets, including HDFS, S3, and Elasticsearch.
This presentation will briefly recap the purpose of Kafka, and then dive into Kafka Connect, with practical examples of data pipelines that can be built with it and are in production at companies around the world already. We'll also look at the Single Message Transform (SMT) capabilities introduced with Kafka 0.10.2 and how they can make Kafka Connect even more flexible and powerful.

Transcript

1. 1
   Real-time Data Integration at
   Scale with Kafka Connect
   Robin Moffatt
   Partner Technology Evangelist, EMEA @ Confluent
   @rmoff robin@confluent.io
   

   View Slide

2. View Slide

3. View Slide

4. View Slide

5. 5
   Kafka Connect in the Apache Kafka ecosystem
   

   View Slide

6. 6
   Kafka Connect : Separation of Concerns
   

   View Slide

7. View Slide

8. 8
   Single Message Transform (SMT) -- Extract, TRANSFORM, Load…
   • Modify events before storing in Kafka:
   • Mask/drop sensitive information
   • Set partitioning key
   • Store lineage
   • Modify events going out of Kafka:
   • Route high priority events to faster
   data stores
   • Direct events to different
   ElasticSearch indexes
   • Cast data types to match destination
   

   View Slide

9. 9
   Kafka Connect API Library of Connectors
   Databases
   Analytics Applications / Other
   Datastore/File Store
   https://www.confluent.io/product/connectors/
   

   View Slide

10. 10
    Streaming Application Data to Kafka
    • Applications are rich source of events
    • Modifying applications is not always possible or
    desirable
    • And what if the data gets changed within the
    database or by other apps?
    • JDBC is one option for extracting data
    

    View Slide

11. 11
    Liberate Application Data into Kafka with CDC
    • Relational databases use transaction logs to
    ensure Durability of data
    • Change-Data-Capture (CDC) mines the log to get
    raw events from the database
    • CDC tools that integrate with Kafka Connect
    include:
    • Debezium
    • DBVisit
    • GoldenGate
    • Attunity
    • + more
    

    View Slide

12. 12
    Kafka Connect
    Common Patterns – Data Integration into Data Lake for batch analytics
    Oracle
    DB2
    MS SQL
    Postgres
    MySQL
    Cassandra
    MongoDB
    Couchbase
    HBase
    S3 / Athena
    HDFS
    BigQuery
    Elasticsearch
    Solr
    CRM ERP
    WebApp
    Twitter
    IRC
    Bloomberg
    …
    Kafka Connect
    Mainframe
    (e.g. VSAM)
    

    View Slide

13. 13
    Common Patterns – Event-Driven microservices
    CRM WebApp
    Orders Service
    Stock Service
    Cassandra
    MongoDB
    Couchbase
    HBase
    S3 / Athena
    HDFS
    BigQuery
    Elasticsearch
    Solr
    Kafka Connect
    Oracle
    DB2
    MS SQL
    Postgres
    MySQL
    Twitter
    IRC
    Bloomberg
    …
    Kafka Connect
    Mainframe
    (e.g. VSAM)
    ERP
    

    View Slide

14. 14
    Common Patterns – Event-Driven microservices & audit/search/storage
    CRM WebApp
    Orders Service
    Stock Service
    Cassandra
    MongoDB
    Couchbase
    HBase
    S3 / Athena
    HDFS
    BigQuery
    Elasticsearch
    Solr
    Kafka Connect
    Oracle
    DB2
    MS SQL
    Postgres
    MySQL
    Twitter
    IRC
    Bloomberg
    …
    Kafka Connect
    Mainframe
    (e.g. VSAM)
    ERP
    

    View Slide

15. 15
    The Numerous Benefits of Kafka Connect
    • Restart capabilities (offset management)
    • Distributed workers
    • Parallelism (for throughput)
    • Load balancing
    • Fault tolerance
    • Schema preservation
    • Data serialisation
    • Centralised management and configuration
    

    View Slide

16. 16
    Kafka Connect – under the covers
    • Each Kafka Connect node is a worker
    • Each worker executes one or more
    tasks
    • Tasks do the actual work of pulling
    data from sources / landing it to sinks
    • Kafka Connect manages the
    distribution and execution of tasks
    • Parallelism, fault-tolerance, load
    balancing all handled automatically
    

    View Slide

17. 17
    Kafka Connect – under the covers
    • Each Kafka Connect node is a worker
    • Each worker executes one or more
    tasks
    • Tasks do the actual work of pulling
    data from sources / landing it to sinks
    • Kafka Connect manages the
    distribution and execution of tasks
    • Parallelism, fault-tolerance, load
    balancing all handled automatically
    

    View Slide

18. 18
    Kafka Connect – under the covers
    • Each Kafka Connect node is a worker
    • Each worker executes one or more
    tasks
    • Tasks do the actual work of pulling
    data from sources / landing it to sinks
    • Kafka Connect manages the
    distribution and execution of tasks
    • Parallelism, fault-tolerance, load
    balancing all handled automatically
    

    View Slide

19. 19
    Kafka Connect – Standalone vs Distributed
    • Kafka Connect has two modes:
    standalone or distributed
    • Distributed - Scaleout & fault
    tolerance easy – just add more
    workers
    • Can run on one node!
    • Standalone - Useful for where
    data source is machine-specific
    (e.g. single-node log files)
    

    View Slide

20. 20
    Kafka Connect - Converters
    • Data from source system is in its own format (e.g. RecordSet from JDBC)
    • Kafka Connect’s Converters provide reusable functionality to serialise data into JSON or Avro
    • The Confluent Schema Registry is used to stores schemas of ingested data
    http://docs.confluent.io/current/connect/concepts.html#converters
    

    View Slide

21. 21
    Configuring Kafka Connect - REST API
    • Configure & control Kafka Connect through REST API
    • Validate connector configuration
    • Create connectors
    • List available plugins
    • Query connector & task state
    • Pause, resume, restart connectors + tasks
    • Configuration is persisted through a Kafka topic
    • Reference :
    http://docs.confluent.io/current/connect/restapi.html
    

    View Slide

22. 22
    Configure Kafka Connect with Confluent Control Center
    

    View Slide

23. 23
    Monitor Your Data Pipeline from End to End with Confluent Control Center
    

    View Slide

24. View Slide

25. 26
    Confluent: a Streaming Platform based on Apache Kafka™
    Database
    Changes
    Log Events loT Data Web Events …
    CRM
    Data Warehouse
    Database
    Hadoop
    Data
    Integration
    …
    Monitoring
    Analytics
    Custom Apps
    Transformations
    Real-time Applications
    …
    Apache Open Source Confluent Open Source Confluent Enterprise
    Confluent Platform
    Confluent Platform
    Apache Kafka™
    Core | Connect| Streams
    Data Compatibility
    Schema Registry
    Monitoring & Administration
    Confluent Control Center
    Operations
    Replicator | Auto Data Balancing
    Development and Connectivity
    Clients | Connectors | REST Proxy
    

    View Slide

26. 27
    Kafka Connect – Getting Started
    • Docs : http://docs.confluent.io/current/connect/
    • Includes Quickstart and full Connect
    documentation including Architecture + Internals
    • Official Confluent Platform Docker images
    available
    • http://docs.confluent.io/current/cp-docker-
    images/docs/quickstart.html#kafka-connect
    • List of connectors
    • https://www.confluent.io/product/connectors/
    • Also search on github
    https://github.com/search?q=kafka-connect
    https://www.confluent.io/download/
    @rmoff robin@confluent.io
    

    View Slide