Implementing Distributed Tracing ‘Like a Boss’ in your Apache Kafka Deployments

Transcript

1. Join the Confluent
   Community Slack
   Subscribe to the
   Confluent blog
   cnfl.io/community-slack cnfl.io/read
   Welcome to the DC Apache Kafka® Meetup!
   6:30pm
   Doors open
   6:30pm - 7:00pm
   Pizza, Drinks and Networking
   7:00pm - 7:45pm
   Ricardo Ferreira, Confluent
   7:45pm - 8:00pm
   Additional Q&A & Networking
   Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. The Apache Software Foundation has no
   affiliation with and does not endorse the materials provided at this event.
   

   View Slide

2. @riferrei | @confluentinc | #dckafkameetup
   Implementing distributed
   tracing ‘like a boss’
   IN YOUR Apache Kafka DEPLOYMENTS
   @riferrei | @confluentinc | #dckafkameetup
   

   View Slide

3. About Us:
   ● Ricardo Ferreira
   ❑ Developer Advocate @ Confluent
   ❑ Currently into Cloud & Serverless
   ❑ Ex-Oracle, Red Hat, and IONA Tech
   ❑ https://riferrei.net
   ● Echo Dot (Alexa)
   ❑ The voice behind Amazon
   ❑ Ex-Raspberry Pi, Arduino
   @riferrei
   @alexa99
   

   View Slide

4. 4
   Let´s go back
   in time…
   

   View Slide

5. How did we do monitoring?
   ● Each system would have their
   very own monitoring system.
   ● Developers would not worry about
   monitoring. This was supposed to
   be “IT stuff”.
   ● Application was deemed OK if all
   systems were showing green.
   ● Different monitoring approaches
   were used throughout the entire
   IT stack portfolio, requiring many
   teams to be involved.
   

   View Slide

6. @riferrei | @confluentinc | #dckafkameetup
   6
   And?
   

   View Slide

7. @riferrei | @confluentinc | #dckafkameetup
   System IS
   NOT Working!
   System IS
   working!
   Time spent with troubleshooting. Days, weeks, months!
   

   View Slide

8. @riferrei | @confluentinc | #dckafkameetup
   8
   Introduction to
   observability
   

   View Slide

9. @riferrei | @confluentinc | #dckafkameetup
   

   View Slide

10. Pillars of observability
    ● Logs: raw sequence of events
    from a single instance of service.
    ● Metrics: numerical measures
    aggregated in given point of time.
    ● Distributed Tracing: detailed
    execution of the causality-related
    activities performed by a given
    transaction. It answers:
    ○ Which services were involved?
    ○ If it was slow, who caused that?
    ○ If failed, who actually failed?
    

    View Slide

11. Evolution of concurrency
    ● No Concurrency at all
    ○ Ex: Apache HTTP Server
    ● Basic Concurrency
    ○ Ex: Multi-threaded Applications
    ● Async Concurrency
    ○ Ex: Actor-based Programming
    ● Distributed Concurrency
    ○ Ex: μServices Architecture Style
    

    View Slide

12. Distributed tracing today
    ● There are many distributed
    tracing technologies available.
    ● Standards are getting created to
    ensure a single programming
    model for each μService.
    ● Deployment mechanisms such as
    Kubernetes are also taking care
    of that automatically.
    ● Network proxies such as Service
    Meshes are also handling this.
    ● OSS and proprietary options.
    

    View Slide

13. Why trace apache kafka?
    ● Apache Kafka is becoming the
    de-facto standard to handle data.
    ● Prediction? It will be the central
    nervous system of any company.
    ● μServices in general already use
    Kafka to exchange messages and
    keep their data stores in-sync.
    ● With event streaming becoming
    even more popular, the Kafka
    adoption tend to grow even more.
    ● Because it is so freaking cool!
    

    View Slide

14. @riferrei | @confluentinc | #dckafkameetup
    14
    Distributed
    tracing in kafka
    

    View Slide

15. Opentracing java api
    ● Library written in Java to handle
    distributed tracing via OpenTracing
    compatible APIs.
    ● Requires the creation of specific
    tracer using the distributed tracing
    technology API.
    ● Uses the GlobalTracer utility class
    to handle the tracer throughout the
    JVM application.
    ● Supports: Apache Kafka Clients,
    Kafka Streams, and Spring Kafka.
    https://github.com/opentracing-contrib/java-kafka-
    client
    

    View Slide

16. Support for bundled jvms
    ● Library written in Java that does
    the automatic creation of the tracer.
    ● Implements the tracing logic using
    the Kafka Interceptors API.
    ● Allows different tracers to be used,
    by using the TracerResolver class.
    ● Provides OOTB support for Jaeger.
    ● Allows multiple services in the JVM
    use their own tracer by specifying a
    configuration properties file.
    ○ export INTERCEPTORS_CONFIG_FILE=
    https://github.com/riferrei/kafka-tracing-support
    

    View Slide

17. @riferrei | @confluentinc | #dckafkameetup
    17
    DEMO
    

    View Slide

18. @riferrei | @confluentinc | #dckafkameetup
    

    View Slide

19. Join
    The
    fun
    

    View Slide

20. @riferrei | @confluentinc | #dckafkameetup
    20
    announcements
    

    View Slide

21. NOMINATE YOURSELF OR A PEER AT
    CONFLUENT.IO/NOMINATE
    

    View Slide

22. KS19Meetup.
    CONFLUENT COMMUNITY DISCOUNT CODE
    25% OFF*
    *Standard Priced Conference pass
    

    View Slide

23. @riferrei | @confluentinc | #dckafkameetup
    

    View Slide

24. View Slide