Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tips & Tricks about Apache Kafka in the Cloud f...

Tips & Tricks about Apache Kafka in the Cloud for Java Developers

This talk has been delivered at JEE Conf on April 26th, 2019.

Ricardo Ferreira

April 26, 2019
Tweet

More Decks by Ricardo Ferreira

Other Decks in Programming

Transcript

  1. 1 Tips & Tricks about Apache Kafka in the Cloud

    for Java Developers Ricardo Ferreira, Developer Advocate @riferrei
  2. 2

  3. 3 About Us: • Ricardo Ferreira ◦ Developer Advocate @

    Confluent ◦ Ex-Oracle, Red Hat, IONA Tech ◦ Currently ~10% Smart, ~90 Dumb ◦ https://riferrei.net • Echo Dot (Alexa) ◦ The voice behind Amazon ◦ Ex-Raspberry Pi, Arduino, T3? ◦ Currently ~90% Smart, ~10 Dumb @riferrei @alexa99
  4. 4 #1 Managed Kafka Services • Kafka is a fine

    proven technology • People don’t know/care internals • Human labor is often too costly • Install-patch-monitor-scale repeat • Build applications, not clusters! https://www.confluent.io/confluent-cloud @riferrei @jeeconf
  5. 5 Let's Create a Cluster 1. Log in to the

    Confluent Cloud. 2. Create a new cluster on-the-fly. 3. Connect your Java applications. https://confluent.cloud @riferrei @jeeconf
  6. 6 http://kafka.apache.org/documentation.ht ml#connect https://docs.confluent.io/current/connect /quickstart.html https://debezium.io #2 Apache Kafka Connect

    • Moving data in-and-out is hard • +75 connectors available now • Integrated with Schema Registry • CDC via Red Hat's Debezium • Connectivity code is you, let the Serialization, Partitioning, Fault- Tolerance, Schema Mgmt on us. @riferrei @jeeconf
  7. 8 Connecting the Layers 1. Please scan the barcode below.

    2. Listen the song and try to guess. 3. Enter with your name and send. @riferrei @jeeconf
  8. 9 #3 Kafka Streams and KSQL • Create stream processing

    apps • Data computation outside Kafka • Same partitioning/scaling model • Complex pipelines using filters, aggregations, windows & more. • Flavors: Java, Scala and KSQL https://kafka.apache.org/documentation/ streams https://www.confluent.io/product/ksql @riferrei @jeeconf
  9. 11 #4 Distributed Tracing • Monitor distributed transactions • Glue

    your Java code with Kafka • Trace producers and consumers • Decorators and Interceptors API • Trace Kafka services such as REST Proxy, Connect and KSQL https://github.com/opentracing-contrib/java-kafka- client https://github.com/riferrei/jaeger-tracing-support @riferrei @jeeconf
  10. 13 #5 Infrastructure-as-Code • Create immutable infrastructure • Handle dependencies

    of the code • Build your Java code on-the-fly • Connect with Cloud automatically • Blend Kafka services with infra https://github.com/confluentinc/cp-docker-images https://github.com/confluentinc/cp-helm-charts https://github.com/confluentinc/ccloud-tools https://github.com/confluentinc/cp-ansible @riferrei @jeeconf
  11. 15 15 Let’s Keep in Touch: @riferrei riferrei riferrei [email protected]

    https://riferrei.net https://cnfl.io/slack Confluent is Hiring: https://www.confluent.io/careers
  12. 16