ペパボOpenTelemetry革命

by Kazuhiko Yamashita

Slide 1

Slide 1 text

ϖύϘOpenTelemetryֵ໋ ʙ͜ͷલMOROHAΈͯେ߸ٽ͠·ͨ͠ʙ

Slide 2

Slide 2 text

ࢁԼ࿨඙!QZBNB (.0ϖύϘٕज़ج൫νʔϜ γχΞɾϓϦϯγύϧ ɹΩϟϯϓɺཱྀߦɺώϧτϯ८ΓɺιϫχΤ८Γ 1SPYZαʔό։ൃɺ0QFO5FMFNFUSZ

Slide 3

Slide 3 text

45/4 -JOVY/444FSWFS TUOTKQ

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

ࠓ೔࿩͢͜ͱ OpenTelemetry֓ཁ OpenTelemetryΛಋೖ͖͔͚ͨͬ͠ OpenTelemetryಋೖঢ়گ ࠓޙͷల๬

Slide 6

Slide 6 text

🍉🌴🏄 GoConference’19 🏄🌴🍉 6 0QFO5FMFNFUSZ ֓ཁ

Slide 7

Slide 7 text

OpenTelemetry چདྷɺ4BB4΍044͝ͱʹ࣮૷͞Ε͍ͯͨςϨϝτϦσʔλͷඪ४Խ IUUQTDMPVEOBUJWFEBZTKQPZ https://opentelemetry.connpass.com/

Slide 8

Slide 8 text

ςϨϝτϦσʔλ “ςϨϝτϦσʔλ͸ɺΦϒβʔόϏϦςΟΛ࣮ݱ͢ΔͨΊʹγεςϜ ιʔε͔Βऩू͞ΕΔग़ྗΛࢦ͠·͢ɻ͜ΕΒͷग़ྗΛ౷߹తʹ෼ੳ͢ Δ͜ͱʹΑΓɺ෼ࢄ؀ڥͰͷγεςϜؒͷؔ܎΍ґଘؔ܎͕Θ͔Γ· ͢ɻओʹ࢖༻͞ΕΔσʔλΫϥε͸ ϩάɺϝτϦΫεɺτϨʔεͷ3ͭͰɺ͜Ε͸ ʮΦϒβʔόϏϦςΟͷ3ͭͷபʯͱ΋ݺ͹Ε·͢ɻ” https://www.splunk.com/ja_jp/data-insider/what-is-opentelemetry.html ΑΓҾ༻

Slide 9

Slide 9 text

ΦϒβʔόϏϦςΟ ΦϒβʔόϏϦςΟɾΤϯδχΞϦϯά 6PΑΓҾ༻ “؆୯ʹݴ͏ͱɺࢲ͕ͨͪߟ͑Διϑτ΢ΣΞγεςϜͷ ʮΦϒβʔόϏϦςΟʯͱ͸ɺγεςϜ͕ͲͷΑ͏ͳঢ়ଶʹͳͬͨͱ͠ ͯ΋ɺͦΕ͕ͲΜͳʹ࢐৽Ͱحົͳ΋ͷͰ͋ͬͯ΋ɺͲΕ͚ͩཧղ͠આ໌ Ͱ͖Δ͔Λࣔ͢ई౓Ͱ͢ɻ·ͨɺͦͷΑ͏ͳ࢐৽Ͱحົͳঢ়ଶʹରͯ͠ ΋ɺࣄલʹσόοάͷඞཁੑΛఆٛͨ͠Γ༧ଌͨ͠Γ͢Δ͜ͱͳ͘ɺ γεςϜͷঢ়ଶσʔλͷ͋ΒΏΔσΟϝϯγϣϯ΍ͦΕΒͷ૊Έ߹Θͤʹ ͍ͭͯΞυϗοΫʹௐࠪ͠ɺΑΓσόοά͕ՄೳͰ͋ΔΑ͏ʹ͢Δඞཁ͕ ͋Γ·͢ɻ΋͠ɺ৽͍͠ίʔυΛσϓϩΠ͢Δඞཁ͕ͳ͘ɺͲΜͳ࢐৽Ͱ حົͳঢ়ଶͰ΋ཧղͰ͖ΔͳΒɺΦϒβʔόϏϦςΟ͕͋Δͱݴ͑·͢ɻ

Slide 10

Slide 10 text

https://opentelemetry.io/blog/2024/pro fi ling/

Slide 11

Slide 11 text

OpenTelemetry https://opentelemetry.io/docs/ ֤ݴޠͷSDK΍ ࣗಈܭ૷ɺखಈܭ૷ ΍ϛυϧ΢ΣΞ͔Β ςϨϝτϦσʔλΛ ग़ྗ

Slide 12

Slide 12 text

OpenTelemetry https://opentelemetry.io/docs/ ඪ४Խ͞Εͨϓϩτίϧʹ ରԠͨ͠ΤʔδΣϯτ

Slide 13

Slide 13 text

OpenTelemetry https://opentelemetry.io/docs/collector/ GoͰ࣮૷͞ΕͨόΠφϦ͕ɺReceivers,Processors,ExportersΛ࣮ߦ

Slide 14

Slide 14 text

OpenTelemetry https://opentelemetry.io/docs/ 4BB4΍ ϗετ͞Εͨ044 ࠩ͠ସ͑Մೳ

Slide 15

Slide 15 text

OpenTelemetryΛ ಋೖ͖͔͚ͨͬ͠

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

͓Θ͔ΓͩΖ͏͔ʁ https:// fi nance.yahoo.co.jp/quote/USDJPY=FX?styl=cndl&frm=mnthly&trm=mnthly&ovrIndctr=sma%2Cmma%2Clma

Slide 18

Slide 18 text

ϖύϘͷࣄ৘ ෼ࢄτϨʔγϯά͕ະಋೖ ؂ࢹαʔϏε͕MackerelͱPrometheusͰڞଘ

Slide 19

Slide 19 text

MackerelͱPrometheus Mackerel Prometheus ຊ൪αʔόͷVM Managed Service Kubernetes Control Plane ࣾ಺αʔϏεͷVM Kubernetes Node Monitor Servers OpenTelemetryਪਐલ ֎ܗ؂ࢹ ϓϥΠϕʔτΫϥ΢υͷHW

Slide 20

Slide 20 text

MackerelͱPrometheus Mackerel Prometheus ຊ൪αʔόͷVM Managed Service Kubernetes Control Plane ࣾ಺αʔϏεͷVM Kubernetes Node Monitor Servers Otel CollecotrΛ͍ΕΔͳΒɺ͜͏ͨ͠΄͏͕ແବ͕ͳ͍ ֎ܗ؂ࢹ ϓϥΠϕʔτΫϥ΢υͷHW ຊ൪αʔόͷVM

Slide 21

Slide 21 text

OpenTelemetryಋೖঢ়گ

Slide 22

Slide 22 text

ίϯϙʔωϯτ VM k8s Metrics ̋ ̋ Trace ̋ ̋ Log ˚ ˚ OpenTelemetryಋೖঢ়گ ̋ϓϩμΫγϣϯͰಋೖࡁΈ ˚લճ.FFUVQͰ࿩ͨ͠ͱ͖ΑΓਐḿ

Slide 23

Slide 23 text

Metrics

Slide 24

Slide 24 text

Metrics ϗετͷϦιʔε͸hostmetricsͰऔಘ ɹVM͸ͦΕͧΕotel collectorΛΠϯετʔϧ ɹk8s͸DaemonSetͰ/Λmount receivers: hostmetrics: collection_interval: 1m scrapers: cpu: metrics: system.cpu.utilization: enabled: true load: metrics: system.cpu.load_average.1m: enabled: true system.cpu.load_average.5m: enabled: true system.cpu.load_average.15m: ...

Slide 25

Slide 25 text

Metrics %BTICPBSE͸(JU)VCͰϝτϦΫε໊Ͱൃ۷ͨ͠ https://ten-snapon.com/archives/2983

Slide 26

Slide 26 text

Metrics 4%,Λར༻ͯ͠ϝτϦΫεΛܭ૷͍ͯ͠Δ https://ten-snapon.com/archives/2983 if req.Method == http.MethodPost && strings.Contains(req.RequestURI, “cartin")){ if err == nil && res.StatusCode >= 200 && res.StatusCode < 300 { r.successCounter.Add(context.Background(), metric.WithAttributes(semconv.HostName(requestURL.Host))) } else { r.failCounter.Add(context.Background(), 1, metric.WithAttributes(semconv.HostName(requestURL.Host))) } }

Slide 27

Slide 27 text

Trace

Slide 28

Slide 28 text

Trace k8sͷμογϡϘʔυͱͯ͠GrafanaΛಋೖࡁΈ k8sͷϩάݕࡧʹGrafana Loki + S3ΛಋೖࡁΈ (SBGBOB5FNQPΛ࠾༻

Slide 29

Slide 29 text

Trace otelcol k8s DataCenter Tempo pod VM otelcol app VMͷTrace΋Otel CollectorΛ ܦ༝ͯ͠ɺk8sͷTempo΁

Slide 30

Slide 30 text

΄ͱ͹͠ΔϚΠΫϩαʔϏεײ

Slide 31

Slide 31 text

΄ͱ͹͠ΔϚΠΫϩαʔϏεײ

Slide 32

Slide 32 text

Log

Slide 33

Slide 33 text

طଘͷϩάύΠϓϥΠϯ ϩάͷϥΠϑλΠϜʹ߹ΘͤͨετΞʹ֨ೲ

Slide 34

Slide 34 text

Apache Kafka LBGLBDMVTUFS /var/log/audit.log LBGLBCSPLFS BVEJUMPH UPQJD LBGLBCSPLFS TZTMPH UPQJD ϩά͝ͱʹτϐοΫΛ෼͚͍ͯΔ /var/log/syslog

Slide 35

Slide 35 text

Kafka Exporter 5PQJD͕ݱঢ়͸ܾΊଧͪͬΆ͍ https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/kafkaexporter

Slide 36

Slide 36 text

Kafka Exporter ͜͏͍͏ײ͡Ͱॻ͚͍ͨ

Slide 37

Slide 37 text

ͳΜͱॻ͚·͢!!1

Slide 38

Slide 38 text

ࠓޙͷల๬ Self Hosted APM ܭଌ͔ΒվળͷαΠΫϧΛճ͢ OpenTelemetry΁ͷߩݙ

Slide 39

Slide 39 text

Self Hosted APM APMͷՄ༻ੑ͸ߴ͘ͳ͍͍ͯ͘ͷͰɺ ӡ༻Ͱ͖ΔͳΒSelf HostedͰ͍͍ ݱࡏ͸SentryΛධՁத

Slide 40

Slide 40 text

Elastic APM͸΍Ίͨ Ҏલ͸DockerͰىಈͯͦ͜͠ʹϝτϦΫεΛૹ Δ͚͕ͩͩͬͨɺAgentͷΠϯετʔϧ͕ඞཁʹ ͳͬͨΓɺҠߦɺESɺkibanaͷϝϯςίετ͕ ௿͍ͱ͸ݴ͑ͳ͍

Slide 41

Slide 41 text

ܭଌ͔ΒվળͷαΠΫϧΛճ͢ ܭଌͰ͖ΔΑ͏ʹͳ͕ͬͨɺܭଌͰ͖͍ͯΔ͔͸ผ໰୊ ܭଌͰ͖ͨͱͯ͠ɺվળͰ͖Δ͔͸ผ໰୊ πʔϧͷڭҭɺϋϯζΦϯ ϘτϧωοΫͷಛఆɺղফ

Slide 42

Slide 42 text

ܭଌ͔ΒվળͷαΠΫϧΛճ͢

Slide 43

Slide 43 text

永遠のN+1 net.ipv4.tcp_max_syn_backlog Kubernetes Scheduler 突然のYahooトピックス特定のバージョンだけで取られる謎のロック Slow Query

Slide 44

Slide 44 text

ܭଌ͔ΒվળͷαΠΫϧΛճ͢ վળʹ͸͍Ζ͍ΖͳεΩϧɺ஌͕ࣝඞཁ σʔλϕʔε: ΫΤϦύϑΥʔϚϯεɺύϥϝʔλʔνϡʔχϯά OS/Kernel: ωοτϫʔΫɺύϥϝʔλʔɺCPUɺϝϞϦʔ ιϑτ΢ΣΞ։ൃ: ܭࢉྔɺϦόʔεΤϯδχΞϦϯά ৘ใऩूೳྗ: GitHub IssueɺThe Linux Kernel ArchivesɺX

Slide 45

Slide 45 text

ܭଌ͔ΒվળͷαΠΫϧΛճ͢ ͠ΐ͏΋ͳ͍ɺ࢖ΘΕͳ͍ɺҙຯ͕ͳ͍ ͦΜͳ΋ͷͰ΋͍͍ɺͱʹ͔͘ॻ͘͜ͱͰ ॻ͚ΔΑ͏ʹͳΓɺؾ͕͍ͭͨΒເͷதͰ ઃܭ͢ΔΑ͏ʹͳΔ

Slide 46

Slide 46 text

OpenTelemetry΁ͷߩݙ OpenTelemetry(collector)ͷ࣮૷ൣғ͸ͱͯ΋޿͍ receiver,exporter,processor͝ͱʹϝϯςφ͕͍ͯɺ·ΔͰ૊৫ͷΑ͏ ࣮૷΋ࢀߟʹͳΔ͜ͱ͕ଟ͘ɺֶͼͷػձʹͳΔ