Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Getting Superman X-Ray Vision: Bringing Observa...
Search
Ricardo Ferreira
March 26, 2019
Programming
0
510
Getting Superman X-Ray Vision: Bringing Observability to your Stream Processing
Ricardo Ferreira
March 26, 2019
Tweet
Share
More Decks by Ricardo Ferreira
See All by Ricardo Ferreira
Elastic Community Conference CfP Clinic
riferrei
0
65
Criando Provedores Customizados para Terraform
riferrei
0
81
From Zero to Hero with Observability in Microservices
riferrei
0
110
Construíndo seu Primeiro Provider para Terraform
riferrei
2
530
Understanding Streaming Data and Analytics with Apache Kafka®
riferrei
1
550
Kafka Meetup from March 2020
riferrei
1
380
BU Talk Fev 2020
riferrei
0
160
Boston Meetup, Fev 2020
riferrei
0
57
Building Event Streaming Applications with Serverless Kafka
riferrei
1
470
Other Decks in Programming
See All in Programming
イベントストーミング図からコードへの変換手順 / Procedure for Converting Event Storming Diagrams to Code
nrslib
2
940
Advanced Micro Frontends: Multi Version/ Framework Scenarios @WAD 2025, Berlin
manfredsteyer
PRO
0
340
AIと”コードの評価関数”を共有する / Share the "code evaluation function" with AI
euglena1215
1
170
Result型で“失敗”を型にするPHPコードの書き方
kajitack
5
970
PicoRuby on Rails
makicamel
2
140
チームで開発し事業を加速するための"良い"設計の考え方 @ サポーターズCoLab 2025-07-08
agatan
1
450
明示と暗黙 ー PHPとGoの インターフェイスの違いを知る
shimabox
2
580
VS Code Update for GitHub Copilot
74th
2
670
AIともっと楽するE2Eテスト
myohei
7
2.9k
Flutterで備える!Accessibility Nutrition Labels完全ガイド
yuukiw00w
0
170
おやつのお供はお決まりですか?@WWDC25 Recap -Japan-\(region).swift
shingangan
0
140
Rubyでやりたい駆動開発 / Ruby driven development
chobishiba
1
750
Featured
See All Featured
Code Review Best Practice
trishagee
69
19k
Faster Mobile Websites
deanohume
307
31k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Reflections from 52 weeks, 52 projects
jeffersonlam
351
21k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
Practical Orchestrator
shlominoach
189
11k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
YesSQL, Process and Tooling at Scale
rocio
173
14k
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.5k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.9k
Designing for Performance
lara
610
69k
Transcript
Getting Superman X-Ray Vision: Bringing Observability to your Stream Processing
Ricardo Ferreira, Developer Advocate
About Me: • Hi, my name is Ricardo Ferreira •
Developer Advocate @ Confluent • Ex-Oracle, Red Hat, IONA Tech • Currently ~70% Dev, ~30% Ops • https://riferrei.net @riferrei
Stream Processing Observability Challenges Best Practices for Kafka • Part
1: • Part 2: Agenda • Part 3:
Stream Processing Observability Challenges Best Practices for Kafka • Part
1: • Part 2: Agenda • Part 3:
Let’s Tweet the Song! 1. Access your Twitter account. 2.
Use #dotcnz in your tweet for correct tracking of your guess. 3. The song name must be within brackets as shown below.
Apache Kafka and Stream Processing O'Reilly Books
Application details: • AWS and Terraform • Confluent Cloud Cluster
• Spring Boot Application • Apache Kafka Connect • Kafka Streams / KSQL • Redis Cache • AWS Lambda • Amazon Alexa
Application details: • AWS and Terraform • Confluent Cloud Cluster
• Spring Boot Application • Apache Kafka Connect • Kafka Streams / KSQL • Redis Cache • AWS Lambda • Amazon Alexa You can find the source-code of this application here:
Stream Processing Observability Challenges Best Practices for Kafka • Part
1: • Part 2: Agenda • Part 3:
Why Observability is Important for Modern Software Architectures? Because it
allows us to focus on solving problems instead of finding where the problem is…
OpenTracing has the elements to implement tracing transparently • Multiple
programming languages • Support for multiple tracers • Large community supporting it
Does it Works with Software Built using Microservices? Of course!
A B
So… What are the Problems? • Collaboration between teams •
Microservices and Serverless • Binary Protocols with no Headers • Pub/Sub based Architectures • Source-Code Unavailability
So… What are the Problems? • Collaboration between teams •
Microservices and Serverless • Binary Protocols with no Headers • Pub/Sub based Architectures • Source-Code Unavailability Stream Processing Arena
Stream Processing Observability Challenges Best Practices for Kafka • Part
1: • Part 2: Agenda • Part 3:
Headers Support on Kafka 0.11.X • Span context can be
set there • All headers are persistent • Headers are agnostic of schema
OpenTracing Contributions • Apache Kafka Instrumentation • Provides support for:
◦ Decorators ◦ Interceptors ◦ Spring Framework ◦ Kafka Streams You can find the source-code of this application here:
Tracing Support for Closed JVMs • Jaeger Tracing support for
REST Proxy, Connect and KSQL • Based on the interceptors model • Instantiate its own tracers based on JSON configuration file set as a environment variable You can find the source-code of this application here:
19 Please Stay in Touch: @riferrei riferrei riferrei
[email protected]
https://riferrei.net
https://cnfl.io/slack
None