Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Getting Superman X-Ray Vision: Bringing Observa...
Search
Ricardo Ferreira
March 26, 2019
Programming
0
510
Getting Superman X-Ray Vision: Bringing Observability to your Stream Processing
Ricardo Ferreira
March 26, 2019
Tweet
Share
More Decks by Ricardo Ferreira
See All by Ricardo Ferreira
Elastic Community Conference CfP Clinic
riferrei
0
73
Criando Provedores Customizados para Terraform
riferrei
0
86
From Zero to Hero with Observability in Microservices
riferrei
0
110
Construíndo seu Primeiro Provider para Terraform
riferrei
2
530
Understanding Streaming Data and Analytics with Apache Kafka®
riferrei
1
560
Kafka Meetup from March 2020
riferrei
1
380
BU Talk Fev 2020
riferrei
0
160
Boston Meetup, Fev 2020
riferrei
0
60
Building Event Streaming Applications with Serverless Kafka
riferrei
1
480
Other Decks in Programming
See All in Programming
私の後悔をAWS DMSで解決した話
hiramax
4
190
Claude Codeで実装以外の開発フロー、どこまで自動化できるか?失敗と成功
ndadayo
4
1.9k
Improving my own Ruby thereafter
sisshiki1969
1
150
実用的なGOCACHEPROG実装をするために / golang.tokyo #40
mazrean
1
210
AWS発のAIエディタKiroを使ってみた
iriikeita
1
160
More Approvers for Greater OSS and Japan Community
tkikuc
1
110
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
330
ソフトウェアテスト徹底指南書の紹介
goyoki
1
140
複雑なドメインに挑む.pdf
yukisakai1225
5
970
UbieのAIパートナーを支えるコンテキストエンジニアリング実践
syucream
2
820
Kiroの仕様駆動開発から見えてきたAIコーディングとの正しい付き合い方
clshinji
1
200
MLH State of the League: 2026 Season
theycallmeswift
0
220
Featured
See All Featured
Music & Morning Musume
bryan
46
6.8k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Visualization
eitanlees
147
16k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
The Pragmatic Product Professional
lauravandoore
36
6.8k
Mobile First: as difficult as doing things right
swwweet
224
9.9k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Being A Developer After 40
akosma
90
590k
How to train your dragon (web standard)
notwaldorf
96
6.2k
Imperfection Machines: The Place of Print at Facebook
scottboms
268
13k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Transcript
Getting Superman X-Ray Vision: Bringing Observability to your Stream Processing
Ricardo Ferreira, Developer Advocate
About Me: • Hi, my name is Ricardo Ferreira •
Developer Advocate @ Confluent • Ex-Oracle, Red Hat, IONA Tech • Currently ~70% Dev, ~30% Ops • https://riferrei.net @riferrei
Stream Processing Observability Challenges Best Practices for Kafka • Part
1: • Part 2: Agenda • Part 3:
Stream Processing Observability Challenges Best Practices for Kafka • Part
1: • Part 2: Agenda • Part 3:
Let’s Tweet the Song! 1. Access your Twitter account. 2.
Use #dotcnz in your tweet for correct tracking of your guess. 3. The song name must be within brackets as shown below.
Apache Kafka and Stream Processing O'Reilly Books
Application details: • AWS and Terraform • Confluent Cloud Cluster
• Spring Boot Application • Apache Kafka Connect • Kafka Streams / KSQL • Redis Cache • AWS Lambda • Amazon Alexa
Application details: • AWS and Terraform • Confluent Cloud Cluster
• Spring Boot Application • Apache Kafka Connect • Kafka Streams / KSQL • Redis Cache • AWS Lambda • Amazon Alexa You can find the source-code of this application here:
Stream Processing Observability Challenges Best Practices for Kafka • Part
1: • Part 2: Agenda • Part 3:
Why Observability is Important for Modern Software Architectures? Because it
allows us to focus on solving problems instead of finding where the problem is…
OpenTracing has the elements to implement tracing transparently • Multiple
programming languages • Support for multiple tracers • Large community supporting it
Does it Works with Software Built using Microservices? Of course!
A B
So… What are the Problems? • Collaboration between teams •
Microservices and Serverless • Binary Protocols with no Headers • Pub/Sub based Architectures • Source-Code Unavailability
So… What are the Problems? • Collaboration between teams •
Microservices and Serverless • Binary Protocols with no Headers • Pub/Sub based Architectures • Source-Code Unavailability Stream Processing Arena
Stream Processing Observability Challenges Best Practices for Kafka • Part
1: • Part 2: Agenda • Part 3:
Headers Support on Kafka 0.11.X • Span context can be
set there • All headers are persistent • Headers are agnostic of schema
OpenTracing Contributions • Apache Kafka Instrumentation • Provides support for:
◦ Decorators ◦ Interceptors ◦ Spring Framework ◦ Kafka Streams You can find the source-code of this application here:
Tracing Support for Closed JVMs • Jaeger Tracing support for
REST Proxy, Connect and KSQL • Based on the interceptors model • Instantiate its own tracers based on JSON configuration file set as a environment variable You can find the source-code of this application here:
19 Please Stay in Touch: @riferrei riferrei riferrei
[email protected]
https://riferrei.net
https://cnfl.io/slack
None