$30 off During Our Annual Pro Sale. View Details »

Tips & Tricks about Apache Kafka in the Cloud for Java Developers

Tips & Tricks about Apache Kafka in the Cloud for Java Developers

This talk has been delivered at JEE Conf on April 26th, 2019.

Ricardo Ferreira

April 26, 2019
Tweet

More Decks by Ricardo Ferreira

Other Decks in Programming

Transcript

  1. 1 Tips & Tricks about Apache Kafka in the Cloud

    for Java Developers Ricardo Ferreira, Developer Advocate @riferrei
  2. 2

  3. 3 About Us: • Ricardo Ferreira ◦ Developer Advocate @

    Confluent ◦ Ex-Oracle, Red Hat, IONA Tech ◦ Currently ~10% Smart, ~90 Dumb ◦ https://riferrei.net • Echo Dot (Alexa) ◦ The voice behind Amazon ◦ Ex-Raspberry Pi, Arduino, T3? ◦ Currently ~90% Smart, ~10 Dumb @riferrei @alexa99
  4. 4 #1 Managed Kafka Services • Kafka is a fine

    proven technology • People don’t know/care internals • Human labor is often too costly • Install-patch-monitor-scale repeat • Build applications, not clusters! https://www.confluent.io/confluent-cloud @riferrei @jeeconf
  5. 5 Let's Create a Cluster 1. Log in to the

    Confluent Cloud. 2. Create a new cluster on-the-fly. 3. Connect your Java applications. https://confluent.cloud @riferrei @jeeconf
  6. 6 http://kafka.apache.org/documentation.ht ml#connect https://docs.confluent.io/current/connect /quickstart.html https://debezium.io #2 Apache Kafka Connect

    • Moving data in-and-out is hard • +75 connectors available now • Integrated with Schema Registry • CDC via Red Hat's Debezium • Connectivity code is you, let the Serialization, Partitioning, Fault- Tolerance, Schema Mgmt on us. @riferrei @jeeconf
  7. 7 Pushing Data to Redis Cache using Apache Kafka Connect

    @riferrei @jeeconf
  8. 8 Connecting the Layers 1. Please scan the barcode below.

    2. Listen the song and try to guess. 3. Enter with your name and send. @riferrei @jeeconf
  9. 9 #3 Kafka Streams and KSQL • Create stream processing

    apps • Data computation outside Kafka • Same partitioning/scaling model • Complex pipelines using filters, aggregations, windows & more. • Flavors: Java, Scala and KSQL https://kafka.apache.org/documentation/ streams https://www.confluent.io/product/ksql @riferrei @jeeconf
  10. 10 @riferrei @jeeconf Processing the Data In-Motion and Creating a

    New Data Set
  11. 11 #4 Distributed Tracing • Monitor distributed transactions • Glue

    your Java code with Kafka • Trace producers and consumers • Decorators and Interceptors API • Trace Kafka services such as REST Proxy, Connect and KSQL https://github.com/opentracing-contrib/java-kafka- client https://github.com/riferrei/jaeger-tracing-support @riferrei @jeeconf
  12. 12 https://github.com/opentracing-contrib/java-kafka- client https://github.com/riferrei/jaeger-tracing-support @riferrei @jeeconf Tomorrow: 11:05 – 11:50

    Implementing Distributed Tracing 'Like a Boss' in your Apache Kafka Deployments
  13. 13 #5 Infrastructure-as-Code • Create immutable infrastructure • Handle dependencies

    of the code • Build your Java code on-the-fly • Connect with Cloud automatically • Blend Kafka services with infra https://github.com/confluentinc/cp-docker-images https://github.com/confluentinc/cp-helm-charts https://github.com/confluentinc/ccloud-tools https://github.com/confluentinc/cp-ansible @riferrei @jeeconf
  14. 14 14 Free Books by Confluent: https://www.confluent.io/apache-kafka-stream-processing-book-bundle

  15. 15 15 Let’s Keep in Touch: @riferrei riferrei riferrei ricardo@confluent.io

    https://riferrei.net https://cnfl.io/slack Confluent is Hiring: https://www.confluent.io/careers
  16. 16