Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tips & Tricks about Apache Kafka in the Cloud for Java Developers

Tips & Tricks about Apache Kafka in the Cloud for Java Developers

This talk has been delivered at JEE Conf on April 26th, 2019.

02ff2dde723b6e26f4ef03ee6b3f6eb9?s=128

Ricardo Ferreira

April 26, 2019
Tweet

Transcript

  1. 1 Tips & Tricks about Apache Kafka in the Cloud

    for Java Developers Ricardo Ferreira, Developer Advocate @riferrei
  2. 2

  3. 3 About Us: • Ricardo Ferreira ◦ Developer Advocate @

    Confluent ◦ Ex-Oracle, Red Hat, IONA Tech ◦ Currently ~10% Smart, ~90 Dumb ◦ https://riferrei.net • Echo Dot (Alexa) ◦ The voice behind Amazon ◦ Ex-Raspberry Pi, Arduino, T3? ◦ Currently ~90% Smart, ~10 Dumb @riferrei @alexa99
  4. 4 #1 Managed Kafka Services • Kafka is a fine

    proven technology • People don’t know/care internals • Human labor is often too costly • Install-patch-monitor-scale repeat • Build applications, not clusters! https://www.confluent.io/confluent-cloud @riferrei @jeeconf
  5. 5 Let's Create a Cluster 1. Log in to the

    Confluent Cloud. 2. Create a new cluster on-the-fly. 3. Connect your Java applications. https://confluent.cloud @riferrei @jeeconf
  6. 6 http://kafka.apache.org/documentation.ht ml#connect https://docs.confluent.io/current/connect /quickstart.html https://debezium.io #2 Apache Kafka Connect

    • Moving data in-and-out is hard • +75 connectors available now • Integrated with Schema Registry • CDC via Red Hat's Debezium • Connectivity code is you, let the Serialization, Partitioning, Fault- Tolerance, Schema Mgmt on us. @riferrei @jeeconf
  7. 7 Pushing Data to Redis Cache using Apache Kafka Connect

    @riferrei @jeeconf
  8. 8 Connecting the Layers 1. Please scan the barcode below.

    2. Listen the song and try to guess. 3. Enter with your name and send. @riferrei @jeeconf
  9. 9 #3 Kafka Streams and KSQL • Create stream processing

    apps • Data computation outside Kafka • Same partitioning/scaling model • Complex pipelines using filters, aggregations, windows & more. • Flavors: Java, Scala and KSQL https://kafka.apache.org/documentation/ streams https://www.confluent.io/product/ksql @riferrei @jeeconf
  10. 10 @riferrei @jeeconf Processing the Data In-Motion and Creating a

    New Data Set
  11. 11 #4 Distributed Tracing • Monitor distributed transactions • Glue

    your Java code with Kafka • Trace producers and consumers • Decorators and Interceptors API • Trace Kafka services such as REST Proxy, Connect and KSQL https://github.com/opentracing-contrib/java-kafka- client https://github.com/riferrei/jaeger-tracing-support @riferrei @jeeconf
  12. 12 https://github.com/opentracing-contrib/java-kafka- client https://github.com/riferrei/jaeger-tracing-support @riferrei @jeeconf Tomorrow: 11:05 – 11:50

    Implementing Distributed Tracing 'Like a Boss' in your Apache Kafka Deployments
  13. 13 #5 Infrastructure-as-Code • Create immutable infrastructure • Handle dependencies

    of the code • Build your Java code on-the-fly • Connect with Cloud automatically • Blend Kafka services with infra https://github.com/confluentinc/cp-docker-images https://github.com/confluentinc/cp-helm-charts https://github.com/confluentinc/ccloud-tools https://github.com/confluentinc/cp-ansible @riferrei @jeeconf
  14. 14 14 Free Books by Confluent: https://www.confluent.io/apache-kafka-stream-processing-book-bundle

  15. 15 15 Let’s Keep in Touch: @riferrei riferrei riferrei ricardo@confluent.io

    https://riferrei.net https://cnfl.io/slack Confluent is Hiring: https://www.confluent.io/careers
  16. 16