ペパボOpenTelemetry革命

OpenTelemetry Meetup #2でお話しました。

Transcript

1. ϖύϘOpenTelemetryֵ໋ ʙډञ԰͚ͩͷҙؾࠐΈ͔͋͠Γ·ͤΜฤʙ
2. ࢁԼ
   ࿨඙
   
   !QZBNB
   (.0ϖύϘ
   ٕज़ج൫νʔϜ
   
   
   γχΞɾϓϦϯγύϧ
   
   
   ɹΩϟϯϓɺཱྀߦɺώϧτϯ८ΓɺιϫχΤ८Γ
   
   
   
   
   
   1SPYZαʔό։ൃɺ0QFO5FMFNFUSZ
3. 45/4
   -JOVY
   /44
   4FSWFS
   TUOTKQ
6. ࠓ೔࿩͢͜ͱ • Πϯϑϥߏ੒֓ཁ • OpenTelemetryಋೖঢ়گ • ϩάΛͲ͏͢Δ͔ΛܾΊ͔Ͷ͍ͯΔ
7. Πϯϑϥߏ੒֓ཁ
8. Πϯϑϥߏ੒֓ཁ 0QFO4UBDL
   #BSFNFUBM
   "84 IFSPLV
   (PPHMF
   $MPVE શαʔϏε߹ܭͰ7.ن໛
   ϗεςΟϯάܥͷαʔϏε΍ɺ֤छج൫Ͱར༻ &,4 3%4 &MBTUJDYYYܥͷϚωʔδυαʔϏεΛத৺ʹར༻ (,& #JH2VFSZΛத৺ʹ"84ͷ%3ɺσʔλΤϯδχΞϦϯάͷج൫ͱͯ͠ར༻
9. 0QFO4UBDL 0QFO4UBDLͰ؅ཧ͢Δ7.ͷ্ʹ
   ࣗࣾ։ൃͷΤϯδϯΛར༻ͯ͠
   ,VCFSOFUFTΛσϓϩΠ ,VCFSOFUFT 0QFO4UBDL #BSFNFUBM #BSFNFUBM #BSFNFUBM 7. 7. 7. ,VCFSOFUFT $POUBJOFS $POUBJOFS $POUBJOFS
10. OpenTelemetryಋೖঢ়گ
11. ίϯϙʔωϯτ VM k8s Metrics ̋ ̋ Trace ̋ ̋ Log ˚ ˚ OpenTelemetryಋೖঢ়گ ̋
    ϓϩμΫγϣϯͰಋೖࡁΈ
    ˚པΉɺ୭͔ԶΛॿ͚ͯ͘Ε
12. Metrics
13. Metrics • hostmetricsͷΈಋೖ • VM͸ͦΕͧΕotel collectorΛ
 Πϯετʔϧ • k8s͸DaemonSetͰ/Λmount receivers: hostmetrics: collection_interval: 1m scrapers: cpu: metrics: system.cpu.utilization: enabled: true load: metrics: system.cpu.load_average.1m: enabled: true system.cpu.load_average.5m: enabled: true system.cpu.load_average.15m: ...
14. Metrics %BTICPBSE͸(JU)VCͰϝτϦΫε໊Ͱൃ۷ͨ͠ https://ten-snapon.com/archives/2983
15. Mackerel͔ΒͷҠߦ • چདྷVM͸Mackerelɺk8s͸PrometheusͰ؂ࢹ͍͕ͯͨ͠ɺotelಋೖΛܖػʹ Prometheus΁ू໿͢Δ(͍ͯ͘͠ɺ͍͖͍ͯͨ͠ɺཔΉʂʂ୭͔ʂʂ) • Mackerelͷcheck plugin؂ࢹɺmetrics plugin؂ࢹ͸
 ੿࡞ͷmackerel-checks-exporterͰPrometheusͷϝτϦΫε΁ͱม׵
16. mackerel-checks-exporter Mackerelͷagent con fi gΛݩʹɺ
 νΣοΫ؂ࢹ΍ɺϝτϦΫεΛऩूͯ͠ɺ
 :9111/metricsͰެ։ receivers: prometheus: con fi g: scrape_con fi gs: - job_name: 'mackerel-checks-exporter' metrics_path: /metrics scrape_interval: 1m static_con fi gs: - targets: - "127.0.0.1:9111"
17. Trace
18. Trace k8sͷμογϡϘʔυͱͯ͠GrafanaΛಋೖࡁΈ k8sͷ୹ظؒͷϩάݕࡧʹGrafana LokiΛಋೖࡁΈ (SBGBOB
    5FNQPΛ࠾༻
19. Trace otelcol k8s DataCenter Tempo pod VM otelcol app ࣗࣾͰ։ൃ͍ͯ͠Δk8s engine͕
 ؂ࢹελοΫΛࣗಈΠϯετʔϧ PrometheusAlertManager
 Grafana
 Grafana Tempo
 Grafana Loki

20. Trace 5SBDFΛಋೖͨ͠ߏ੒ go proxy server go api server rails api MySQL Radis the other rails api
21. Trace (Pͷࣗಈܭ૷ propagationΛ໌ࣔతʹఆٛ͢Δඞཁ͕͋Δ http/sql/redisͳͲ͸ࣗಈܭ૷͕੔උ͞Ε͍ͯΔͷͰɺwrap͢Δ͚ͩͰOK
22. Trace 3VCZ
    PO
    3BJMTͷࣗಈܭ૷ جຊgemΛΠϯετʔϧͯ͠ॳظԽ͢Δ͚ͩ
24. ೋ೔ਲ͍Ζ͘ͳ͜ͱ͕ͳ͍ ېञͩʂېञʂʂʂ
25. Log
26. طଘͷϩάύΠϓϥΠϯ ϩάͷϥΠϑλΠϜʹ߹ΘͤͨετΞʹ֨ೲ
27. Apache Kafka LBGLB
    DMVTUFS /var/log/audit.log LBGLBCSPLFS BVEJUMPH
    UPQJD LBGLBCSPLFS TZTMPH
    UPQJD ϩά͝ͱʹτϐοΫΛ෼͚͍ͯΔ /var/log/syslog
28. Kafka Exporter 5PQJD͕ݱঢ়͸ܾΊଧͪͬΆ͍ https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/kafkaexporter
29. Kafka Exporter ͜͏͍͏ײ͡Ͱॻ͚͍ͨ
30. Log & Trace 5SBDF
    *%Ͱ-PHͱ5SBDFΛ࿈ܞ
31. Log & Trace 5SBDF
    *%Ͱ-PHͱ5SBDFΛ࿈ܞ
32. Log & Trace 5SBDF
    *%Ͱ-PHͱ5SBDFΛ࿈ܞ
33. ΍ͬͯΈͯΘ͔ͬͨ͜ͱ • otel collectorʹू໿͢Δ͚ͩͰɺotelϓϩτίϧͰந৅Խ͞ΕΔͷͰ
 ࢥͬͨΑΓγϯϓϧʹ༷ʑ͕ίϯτϩʔϧͰ͖Δ • ֤ݴޠͷ։ൃऀ΍ɺcontribͷ։ൃऀͷํʑͷߩݙͷ͓ӄͰɺઃఆॻ͚ͩ͘ Ͱେମ͕ಈ͘
34. ࠓޙ΍͍͖͍ͬͯͨ͜ͱ • otel collectorΛ࣠ʹɺςϨϝτϦσʔλͷऩूɺٕज़ΛҰݩԽͯ͠ɺ
 ࣾ಺Ͱϊ΢ϋ΢Λڞ༗ͭͭ͠ɺϨόϨοδΛޮ͔͍ͤͨ • MetricsͷετΞ͸Prometheusʹ·ͱΊ͍ͨ • LogͷपΓ͸exporterΛ࡞Δ͔ɺίϛϡχςΟ͔ΒԠ౴͕͋Ε͹࿩ͯ͠PR ౤͍͖͍͛ͯͨ(ͳΜ͔collectorଆͰେ͖͍มߋ͕ඞཁͳؾ΋͢Δ)
35. ܅΋PUFMʹͳΒͳ͍͔ʁ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU