Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Getting Superman X-Ray Vision: Bringing Observa...
Search
Ricardo Ferreira
March 26, 2019
Programming
0
510
Getting Superman X-Ray Vision: Bringing Observability to your Stream Processing
Ricardo Ferreira
March 26, 2019
Tweet
Share
More Decks by Ricardo Ferreira
See All by Ricardo Ferreira
Elastic Community Conference CfP Clinic
riferrei
0
74
Criando Provedores Customizados para Terraform
riferrei
0
87
From Zero to Hero with Observability in Microservices
riferrei
0
110
Construíndo seu Primeiro Provider para Terraform
riferrei
2
530
Understanding Streaming Data and Analytics with Apache Kafka®
riferrei
1
560
Kafka Meetup from March 2020
riferrei
1
390
BU Talk Fev 2020
riferrei
0
160
Boston Meetup, Fev 2020
riferrei
0
61
Building Event Streaming Applications with Serverless Kafka
riferrei
1
480
Other Decks in Programming
See All in Programming
Kiroで始めるAI-DLC
kaonash
2
630
基礎から学ぶ大画面対応(Learning Large-Screen Support from the Ground Up)
tomoya0x00
0
4.3k
go test -json そして testing.T.Attr / Kyoto.go #63
utgwkk
3
320
請來的 AI Agent 同事們在寫程式時,怎麼用 pytest 去除各種幻想與盲點
keitheis
0
130
AI時代のUIはどこへ行く?
yusukebe
18
9.1k
ぬるぬる動かせ! Riveでアニメーション実装🐾
kno3a87
1
230
AI Agents: How Do They Work and How to Build Them @ Shift 2025
slobodan
0
110
Putting The Genie in the Bottle - A Crash Course on running LLMs on Android
iurysza
0
140
私の後悔をAWS DMSで解決した話
hiramax
4
210
Amazon RDS 向けに提供されている MCP Server と仕組みを調べてみた/jawsug-okayama-2025-aurora-mcp
takahashiikki
1
120
奥深くて厄介な「改行」と仲良くなる20分
oguemon
1
570
Platformに“ちょうどいい”責務ってどこ? 関心の熱さにあわせて考える、責務分担のプラクティス
estie
1
170
Featured
See All Featured
Scaling GitHub
holman
463
140k
The World Runs on Bad Software
bkeepers
PRO
70
11k
Art, The Web, and Tiny UX
lynnandtonic
303
21k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
113
20k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
30
9.7k
Music & Morning Musume
bryan
46
6.8k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
188
55k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
1.1k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Transcript
Getting Superman X-Ray Vision: Bringing Observability to your Stream Processing
Ricardo Ferreira, Developer Advocate
About Me: • Hi, my name is Ricardo Ferreira •
Developer Advocate @ Confluent • Ex-Oracle, Red Hat, IONA Tech • Currently ~70% Dev, ~30% Ops • https://riferrei.net @riferrei
Stream Processing Observability Challenges Best Practices for Kafka • Part
1: • Part 2: Agenda • Part 3:
Stream Processing Observability Challenges Best Practices for Kafka • Part
1: • Part 2: Agenda • Part 3:
Let’s Tweet the Song! 1. Access your Twitter account. 2.
Use #dotcnz in your tweet for correct tracking of your guess. 3. The song name must be within brackets as shown below.
Apache Kafka and Stream Processing O'Reilly Books
Application details: • AWS and Terraform • Confluent Cloud Cluster
• Spring Boot Application • Apache Kafka Connect • Kafka Streams / KSQL • Redis Cache • AWS Lambda • Amazon Alexa
Application details: • AWS and Terraform • Confluent Cloud Cluster
• Spring Boot Application • Apache Kafka Connect • Kafka Streams / KSQL • Redis Cache • AWS Lambda • Amazon Alexa You can find the source-code of this application here:
Stream Processing Observability Challenges Best Practices for Kafka • Part
1: • Part 2: Agenda • Part 3:
Why Observability is Important for Modern Software Architectures? Because it
allows us to focus on solving problems instead of finding where the problem is…
OpenTracing has the elements to implement tracing transparently • Multiple
programming languages • Support for multiple tracers • Large community supporting it
Does it Works with Software Built using Microservices? Of course!
A B
So… What are the Problems? • Collaboration between teams •
Microservices and Serverless • Binary Protocols with no Headers • Pub/Sub based Architectures • Source-Code Unavailability
So… What are the Problems? • Collaboration between teams •
Microservices and Serverless • Binary Protocols with no Headers • Pub/Sub based Architectures • Source-Code Unavailability Stream Processing Arena
Stream Processing Observability Challenges Best Practices for Kafka • Part
1: • Part 2: Agenda • Part 3:
Headers Support on Kafka 0.11.X • Span context can be
set there • All headers are persistent • Headers are agnostic of schema
OpenTracing Contributions • Apache Kafka Instrumentation • Provides support for:
◦ Decorators ◦ Interceptors ◦ Spring Framework ◦ Kafka Streams You can find the source-code of this application here:
Tracing Support for Closed JVMs • Jaeger Tracing support for
REST Proxy, Connect and KSQL • Based on the interceptors model • Instantiate its own tracers based on JSON configuration file set as a environment variable You can find the source-code of this application here:
19 Please Stay in Touch: @riferrei riferrei riferrei
[email protected]
https://riferrei.net
https://cnfl.io/slack
None