Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
ペパボOpenTelemetry革命
Search
Kazuhiko Yamashita
May 09, 2024
Programming
2.8k
2
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
ペパボOpenTelemetry革命
社内のイベントでお話しました
Kazuhiko Yamashita
May 09, 2024
Transcript
48 slides
ϖύϘOpenTelemetryֵ໋ ʙ͜ͷલMOROHAΈͯେ߸ٽ͠·ͨ͠ʙ
ࢁԼ!QZBNB (.0ϖύϘٕज़ج൫νʔϜ γχΞɾϓϦϯγύϧ ɹΩϟϯϓɺཱྀߦɺώϧτϯ८ΓɺιϫχΤ८Γ 1SPYZαʔό։ൃɺ0QFO5FMFNFUSZ
45/4 -JOVY/444FSWFS TUOTKQ
None
ࠓ͢͜ͱ OpenTelemetry֓ཁ OpenTelemetryΛಋೖ͖͔͚ͨͬ͠ OpenTelemetryಋೖঢ়گ ࠓޙͷల
🍉🌴🏄 GoConference’19 🏄🌴🍉 6 0QFO5FMFNFUSZ ֓ཁ
OpenTelemetry چདྷɺ4BB4044͝ͱʹ࣮͞Ε͍ͯͨςϨϝτϦσʔλͷඪ४Խ IUUQTDMPVEOBUJWFEBZTKQPZ https://opentelemetry.connpass.com/
ςϨϝτϦσʔλ “ςϨϝτϦσʔλɺΦϒβʔόϏϦςΟΛ࣮ݱ͢ΔͨΊʹγεςϜ ιʔε͔Βऩू͞ΕΔग़ྗΛࢦ͠·͢ɻ͜ΕΒͷग़ྗΛ౷߹తʹੳ͢ Δ͜ͱʹΑΓɺࢄڥͰͷγεςϜؒͷؔґଘ͕ؔΘ͔Γ· ͢ɻओʹ༻͞ΕΔσʔλΫϥε ϩάɺϝτϦΫεɺτϨʔεͷ3ͭͰɺ͜Ε ʮΦϒβʔόϏϦςΟͷ3ͭͷபʯͱݺΕ·͢ɻ” https://www.splunk.com/ja_jp/data-insider/what-is-opentelemetry.html ΑΓҾ༻
ΦϒβʔόϏϦςΟ ΦϒβʔόϏϦςΟɾΤϯδχΞϦϯά 6PΑΓҾ༻ “؆୯ʹݴ͏ͱɺࢲ͕ͨͪߟ͑ΔιϑτΣΞγεςϜͷ ʮΦϒβʔόϏϦςΟʯͱɺγεςϜ͕ͲͷΑ͏ͳঢ়ଶʹͳͬͨͱ͠ ͯɺͦΕ͕ͲΜͳʹ৽ͰحົͳͷͰ͋ͬͯɺͲΕ͚ͩཧղ͠આ໌ Ͱ͖Δ͔Λࣔ͢ईͰ͢ɻ·ͨɺͦͷΑ͏ͳ৽Ͱحົͳঢ়ଶʹରͯ͠ ɺࣄલʹσόοάͷඞཁੑΛఆٛͨ͠Γ༧ଌͨ͠Γ͢Δ͜ͱͳ͘ɺ γεςϜͷঢ়ଶσʔλͷ͋ΒΏΔσΟϝϯγϣϯͦΕΒͷΈ߹Θͤʹ ͍ͭͯΞυϗοΫʹௐࠪ͠ɺΑΓσόοά͕ՄೳͰ͋ΔΑ͏ʹ͢Δඞཁ͕
͋Γ·͢ɻ͠ɺ৽͍͠ίʔυΛσϓϩΠ͢Δඞཁ͕ͳ͘ɺͲΜͳ৽Ͱ حົͳঢ়ଶͰཧղͰ͖ΔͳΒɺΦϒβʔόϏϦςΟ͕͋Δͱݴ͑·͢ɻ
https://opentelemetry.io/blog/2024/pro fi ling/
OpenTelemetry https://opentelemetry.io/docs/ ֤ݴޠͷSDK ࣗಈܭɺखಈܭ ϛυϧΣΞ͔Β ςϨϝτϦσʔλΛ ग़ྗ
OpenTelemetry https://opentelemetry.io/docs/ ඪ४Խ͞Εͨϓϩτίϧʹ ରԠͨ͠ΤʔδΣϯτ
OpenTelemetry https://opentelemetry.io/docs/collector/ GoͰ࣮͞ΕͨόΠφϦ͕ɺReceivers,Processors,ExportersΛ࣮ߦ
OpenTelemetry https://opentelemetry.io/docs/ 4BB4 ϗετ͞Εͨ044 ࠩ͠ସ͑Մೳ
OpenTelemetryΛ ಋೖ͖͔͚ͨͬ͠
None
͓Θ͔ΓͩΖ͏͔ʁ https:// fi nance.yahoo.co.jp/quote/USDJPY=FX?styl=cndl&frm=mnthly&trm=mnthly&ovrIndctr=sma%2Cmma%2Clma
ϖύϘͷࣄ ࢄτϨʔγϯά͕ະಋೖ ࢹαʔϏε͕MackerelͱPrometheusͰڞଘ
MackerelͱPrometheus Mackerel Prometheus ຊ൪αʔόͷVM Managed Service Kubernetes Control Plane ࣾαʔϏεͷVM
Kubernetes Node Monitor Servers OpenTelemetryਪਐલ ֎ܗࢹ ϓϥΠϕʔτΫϥυͷHW
MackerelͱPrometheus Mackerel Prometheus ຊ൪αʔόͷVM Managed Service Kubernetes Control Plane ࣾαʔϏεͷVM
Kubernetes Node Monitor Servers Otel CollecotrΛ͍ΕΔͳΒɺ͜͏ͨ͠΄͏͕ແବ͕ͳ͍ ֎ܗࢹ ϓϥΠϕʔτΫϥυͷHW ຊ൪αʔόͷVM
OpenTelemetryಋೖঢ়گ
ίϯϙʔωϯτ VM k8s Metrics ̋ ̋ Trace ̋ ̋ Log
˚ ˚ OpenTelemetryಋೖঢ়گ ̋ϓϩμΫγϣϯͰಋೖࡁΈ ˚લճ.FFUVQͰͨ͠ͱ͖ΑΓਐḿ
Metrics
Metrics ϗετͷϦιʔεhostmetricsͰऔಘ ɹVMͦΕͧΕotel collectorΛΠϯετʔϧ ɹk8sDaemonSetͰ/Λmount receivers: hostmetrics: collection_interval: 1m scrapers:
cpu: metrics: system.cpu.utilization: enabled: true load: metrics: system.cpu.load_average.1m: enabled: true system.cpu.load_average.5m: enabled: true system.cpu.load_average.15m: ...
Metrics %BTICPBSE(JU)VCͰϝτϦΫε໊Ͱൃ۷ͨ͠ https://ten-snapon.com/archives/2983
Metrics 4%,Λར༻ͯ͠ϝτϦΫεΛܭ͍ͯ͠Δ https://ten-snapon.com/archives/2983 if req.Method == http.MethodPost && strings.Contains(req.RequestURI, “cartin")){
if err == nil && res.StatusCode >= 200 && res.StatusCode < 300 { r.successCounter.Add(context.Background(), metric.WithAttributes(semconv.HostName(requestURL.Host))) } else { r.failCounter.Add(context.Background(), 1, metric.WithAttributes(semconv.HostName(requestURL.Host))) } }
Trace
Trace k8sͷμογϡϘʔυͱͯ͠GrafanaΛಋೖࡁΈ k8sͷϩάݕࡧʹGrafana Loki + S3ΛಋೖࡁΈ (SBGBOB5FNQPΛ࠾༻
Trace otelcol k8s DataCenter Tempo pod VM otelcol app VMͷTraceOtel
CollectorΛ ܦ༝ͯ͠ɺk8sͷTempo
΄ͱ͠ΔϚΠΫϩαʔϏεײ
΄ͱ͠ΔϚΠΫϩαʔϏεײ
Log
طଘͷϩάύΠϓϥΠϯ ϩάͷϥΠϑλΠϜʹ߹ΘͤͨετΞʹ֨ೲ
Apache Kafka LBGLBDMVTUFS /var/log/audit.log LBGLBCSPLFS BVEJUMPH UPQJD LBGLBCSPLFS TZTMPH UPQJD
ϩά͝ͱʹτϐοΫΛ͚͍ͯΔ /var/log/syslog
Kafka Exporter 5PQJD͕ݱঢ়ܾΊଧͪͬΆ͍ https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/kafkaexporter
Kafka Exporter ͜͏͍͏ײ͡Ͱॻ͚͍ͨ
ͳΜͱॻ͚·͢!!1
ࠓޙͷల Self Hosted APM ܭଌ͔ΒվળͷαΠΫϧΛճ͢ OpenTelemetryͷߩݙ
Self Hosted APM APMͷՄ༻ੑߴ͘ͳ͍͍ͯ͘ͷͰɺ ӡ༻Ͱ͖ΔͳΒSelf HostedͰ͍͍ ݱࡏSentryΛධՁத
Elastic APMΊͨ ҎલDockerͰىಈͯͦ͜͠ʹϝτϦΫεΛૹ Δ͚͕ͩͩͬͨɺAgentͷΠϯετʔϧ͕ඞཁʹ ͳͬͨΓɺҠߦɺESɺkibanaͷϝϯςίετ͕ ͍ͱݴ͑ͳ͍
ܭଌ͔ΒվળͷαΠΫϧΛճ͢ ܭଌͰ͖ΔΑ͏ʹͳ͕ͬͨɺܭଌͰ͖͍ͯΔ͔ผ ܭଌͰ͖ͨͱͯ͠ɺվળͰ͖Δ͔ผ πʔϧͷڭҭɺϋϯζΦϯ ϘτϧωοΫͷಛఆɺղফ
ܭଌ͔ΒվળͷαΠΫϧΛճ͢
永遠のN+1 net.ipv4.tcp_max_syn_backlog Kubernetes Scheduler 突然のYahooトピックス 特定のバージョンだけで取られる謎のロック Slow Query
ܭଌ͔ΒվળͷαΠΫϧΛճ͢ վળʹ͍Ζ͍ΖͳεΩϧɺ͕ࣝඞཁ σʔλϕʔε: ΫΤϦύϑΥʔϚϯεɺύϥϝʔλʔνϡʔχϯά OS/Kernel: ωοτϫʔΫɺύϥϝʔλʔɺCPUɺϝϞϦʔ ιϑτΣΞ։ൃ: ܭࢉྔɺϦόʔεΤϯδχΞϦϯά ใऩूೳྗ: GitHub
IssueɺThe Linux Kernel ArchivesɺX
ܭଌ͔ΒվળͷαΠΫϧΛճ͢ ͠ΐ͏ͳ͍ɺΘΕͳ͍ɺҙຯ͕ͳ͍ ͦΜͳͷͰ͍͍ɺͱʹ͔͘ॻ͘͜ͱͰ ॻ͚ΔΑ͏ʹͳΓɺؾ͕͍ͭͨΒເͷதͰ ઃܭ͢ΔΑ͏ʹͳΔ
OpenTelemetryͷߩݙ OpenTelemetry(collector)ͷ࣮ൣғͱ͍ͯ receiver,exporter,processor͝ͱʹϝϯςφ͕͍ͯɺ·ΔͰ৫ͷΑ͏ ࣮ࢀߟʹͳΔ͜ͱ͕ଟ͘ɺֶͼͷػձʹͳΔ
ͳΜͱɺ ೖࣾͨ͠Βશ෦ Ε·͢ʂʂ1
܅PUFMʹͳΒͳ͍͔ʁ ࠷৽ͷ࠾༻ใΛνΣοΫˠ !QC@SFDSVJU
More Decks by Kazuhiko Yamashita
See All by Kazuhiko Yamashita
タクシーアプリ『GO』の バックエンド開発のおける AI利活用と若者のすべて
pyama86
3
2.8k
継続的な負荷検証を目指して
pyama86
3
1.9k
成長期における、 ユーザー領域の複雑さと 整備の進め方
pyama86
1
860
Stay Hacker 〜九州で生まれ、Perlに出会い、コミュニティで育つ〜
pyama86
2
6.8k
Managing Database Migrations in Go Backend Systems
pyama86
0
620
新しい職場の CI が 20 分かかっていたらあなたならどうする?
pyama86
2
1.6k
事業を差別化する技術を生み出す技術
pyama86
4
2.5k
Re:Define 可用性を支える モニタリング、パフォーマンス最適化、そしてセキュリティ
pyama86
9
12k
AI時代におけるSRE、 あるいはエンジニアの生存戦略
pyama86
6
2.2k
Other Decks in Programming
See All in Programming
エージェントを使い続けるためのメタワーク / Meta-work for keeping using the agent
rkaga
6
2.3k
Lambda durable functionsでHuman in the Loopを楽に…楽に……
har1101
2
330
AWS CDKを理解するために公式の定義を深掘って解説してみる / Understanding AWS CDK by Digging into Its Official Definition
akihisaikeda
3
310
Engineering Biometric UX Across Android Versions | DroidKnights 2026
ryunen344
1
210
SFCで実現する病院・医療システムの動的組み立て式UI
koukimiura
1
180
仕様駆動開発入門 (何を作るかを先に決めて渡す) | 目指せ Kiro マスター 05
inariku
PRO
2
170
Augmenting AI with the Power of Jakarta EE
ivargrimstad
0
340
FactoryBotをやめればRSpecは速くなるのか?
yuta_onishi_97
0
1.7k
graphql-ruby を Spinel で速くしようとした話
mtsmfm
0
150
AIがコードを書いてくれる時代、我々は何を確かめるのか
yosuke_furukawa
PRO
13
6k
小さなアプリを作って公開するまで
onozaty
0
180
Scalaの型でマルチテナントの認可を守る
nextbeatdev
0
150
Featured
See All Featured
Highjacked: Video Game Concept Design
rkendrick25
PRO
2
570
A Tale of Four Properties
chriscoyier
163
24k
How to Talk to Developers About Accessibility
jct
2
620
Discover your Explorer Soul
emna__ayadi
2
1.3k
How to build a perfect <img>
jonoalderson
1
6.2k
Making Projects Easy
brettharned
120
6.9k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
141
35k
Redefining SEO in the New Era of Traffic Generation
szymonslowik
1
530
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
38
3.1k
Jess Joyce - The Pitfalls of Following Frameworks
techseoconnect
PRO
1
480
Paper Plane
katiecoart
PRO
6
55k
Gemini Prompt Engineering: Practical Techniques for Tangible AI Outcomes
mfonobong
2
640