ペパボOpenTelemetry革命

Slide 1

Slide 1 text

ϖύϘOpenTelemetryֵ໋ ʙډञ԰͚ͩͷҙؾࠐΈ͔͋͠Γ·ͤΜฤʙ

Slide 2

Slide 2 text

ࢁԼ࿨඙!QZBNB (.0ϖύϘٕज़ج൫νʔϜ γχΞɾϓϦϯγύϧ ɹΩϟϯϓɺཱྀߦɺώϧτϯ८ΓɺιϫχΤ८Γ 1SPYZαʔό։ൃɺ0QFO5FMFNFUSZ

Slide 3

Slide 3 text

45/4 -JOVY/444FSWFS TUOTKQ

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

ࠓ೔࿩͢͜ͱ • Πϯϑϥߏ੒֓ཁ • OpenTelemetryಋೖঢ়گ • ϩάΛͲ͏͢Δ͔ΛܾΊ͔Ͷ͍ͯΔ

Slide 7

Slide 7 text

Πϯϑϥߏ੒֓ཁ

Slide 8

Slide 8 text

Πϯϑϥߏ੒֓ཁ 0QFO4UBDL #BSFNFUBM "84 IFSPLV (PPHMF$MPVE શαʔϏε߹ܭͰ7.ن໛ ϗεςΟϯάܥͷαʔϏε΍ɺ֤छج൫Ͱར༻ &,4 3%4 &MBTUJDYYYܥͷϚωʔδυαʔϏεΛத৺ʹར༻ (,& #JH2VFSZΛத৺ʹ"84ͷ%3ɺσʔλΤϯδχΞϦϯάͷج൫ͱͯ͠ར༻

Slide 9

Slide 9 text

0QFO4UBDL 0QFO4UBDLͰ؅ཧ͢Δ7.ͷ্ʹ ࣗࣾ։ൃͷΤϯδϯΛར༻ͯ͠ ,VCFSOFUFTΛσϓϩΠ ,VCFSOFUFT 0QFO4UBDL #BSFNFUBM #BSFNFUBM #BSFNFUBM 7. 7. 7. ,VCFSOFUFT $POUBJOFS $POUBJOFS $POUBJOFS

Slide 10

Slide 10 text

OpenTelemetryಋೖঢ়گ

Slide 11

Slide 11 text

ίϯϙʔωϯτ VM k8s Metrics ̋ ̋ Trace ̋ ̋ Log ˚ ˚ OpenTelemetryಋೖঢ়گ ̋ϓϩμΫγϣϯͰಋೖࡁΈ ˚པΉɺ୭͔ԶΛॿ͚ͯ͘Ε

Slide 12

Slide 12 text

Metrics

Slide 13

Slide 13 text

Metrics • hostmetricsͷΈಋೖ • VM͸ͦΕͧΕotel collectorΛ  Πϯετʔϧ • k8s͸DaemonSetͰ/Λmount receivers: hostmetrics: collection_interval: 1m scrapers: cpu: metrics: system.cpu.utilization: enabled: true load: metrics: system.cpu.load_average.1m: enabled: true system.cpu.load_average.5m: enabled: true system.cpu.load_average.15m: ...

Slide 14

Slide 14 text

Metrics %BTICPBSE͸(JU)VCͰϝτϦΫε໊Ͱൃ۷ͨ͠ https://ten-snapon.com/archives/2983

Slide 15

Slide 15 text

Mackerel͔ΒͷҠߦ • چདྷVM͸Mackerelɺk8s͸PrometheusͰ؂ࢹ͍͕ͯͨ͠ɺotelಋೖΛܖػʹ Prometheus΁ू໿͢Δ(͍ͯ͘͠ɺ͍͖͍ͯͨ͠ɺཔΉʂʂ୭͔ʂʂ) • Mackerelͷcheck plugin؂ࢹɺmetrics plugin؂ࢹ͸  ੿࡞ͷmackerel-checks-exporterͰPrometheusͷϝτϦΫε΁ͱม׵

Slide 16

Slide 16 text

mackerel-checks-exporter Mackerelͷagent con fi gΛݩʹɺ  νΣοΫ؂ࢹ΍ɺϝτϦΫεΛऩूͯ͠ɺ  :9111/metricsͰެ։ receivers: prometheus: con fi g: scrape_con fi gs: - job_name: 'mackerel-checks-exporter' metrics_path: /metrics scrape_interval: 1m static_con fi gs: - targets: - "127.0.0.1:9111"

Slide 17

Slide 17 text

Trace

Slide 18

Slide 18 text

Trace k8sͷμογϡϘʔυͱͯ͠GrafanaΛಋೖࡁΈ k8sͷ୹ظؒͷϩάݕࡧʹGrafana LokiΛಋೖࡁΈ (SBGBOB5FNQPΛ࠾༻

Slide 19

Slide 19 text

Trace otelcol k8s DataCenter Tempo pod VM otelcol app ࣗࣾͰ։ൃ͍ͯ͠Δk8s engine͕  ؂ࢹελοΫΛࣗಈΠϯετʔϧ PrometheusAlertManager  Grafana  Grafana Tempo  Grafana Loki 

Slide 20

Slide 20 text

Trace 5SBDFΛಋೖͨ͠ߏ੒ go proxy server go api server rails api MySQL Radis the other rails api

Slide 21

Slide 21 text

Trace (Pͷࣗಈܭ૷ propagationΛ໌ࣔతʹఆٛ͢Δඞཁ͕͋Δ http/sql/redisͳͲ͸ࣗಈܭ૷͕੔උ͞Ε͍ͯΔͷͰɺwrap͢Δ͚ͩͰOK

Slide 22

Slide 22 text

Trace 3VCZPO3BJMTͷࣗಈܭ૷ جຊgemΛΠϯετʔϧͯ͠ॳظԽ͢Δ͚ͩ

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

ೋ೔ਲ͍Ζ͘ͳ͜ͱ͕ͳ͍ ېञͩʂېञʂʂʂ

Slide 25

Slide 25 text

Log

Slide 26

Slide 26 text

طଘͷϩάύΠϓϥΠϯ ϩάͷϥΠϑλΠϜʹ߹ΘͤͨετΞʹ֨ೲ

Slide 27

Slide 27 text

Apache Kafka LBGLBDMVTUFS /var/log/audit.log LBGLBCSPLFS BVEJUMPH UPQJD LBGLBCSPLFS TZTMPH UPQJD ϩά͝ͱʹτϐοΫΛ෼͚͍ͯΔ /var/log/syslog

Slide 28

Slide 28 text

Kafka Exporter 5PQJD͕ݱঢ়͸ܾΊଧͪͬΆ͍ https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/kafkaexporter

Slide 29

Slide 29 text

Kafka Exporter ͜͏͍͏ײ͡Ͱॻ͚͍ͨ

Slide 30

Slide 30 text

Log & Trace 5SBDF*%Ͱ-PHͱ5SBDFΛ࿈ܞ

Slide 31

Slide 31 text

Log & Trace 5SBDF*%Ͱ-PHͱ5SBDFΛ࿈ܞ

Slide 32

Slide 32 text

Log & Trace 5SBDF*%Ͱ-PHͱ5SBDFΛ࿈ܞ

Slide 33

Slide 33 text

΍ͬͯΈͯΘ͔ͬͨ͜ͱ • otel collectorʹू໿͢Δ͚ͩͰɺotelϓϩτίϧͰந৅Խ͞ΕΔͷͰ  ࢥͬͨΑΓγϯϓϧʹ༷ʑ͕ίϯτϩʔϧͰ͖Δ • ֤ݴޠͷ։ൃऀ΍ɺcontribͷ։ൃऀͷํʑͷߩݙͷ͓ӄͰɺઃఆॻ͚ͩ͘ Ͱେମ͕ಈ͘

Slide 34

Slide 34 text

ࠓޙ΍͍͖͍ͬͯͨ͜ͱ • otel collectorΛ࣠ʹɺςϨϝτϦσʔλͷऩूɺٕज़ΛҰݩԽͯ͠ɺ  ࣾ಺Ͱϊ΢ϋ΢Λڞ༗ͭͭ͠ɺϨόϨοδΛޮ͔͍ͤͨ • MetricsͷετΞ͸Prometheusʹ·ͱΊ͍ͨ • LogͷपΓ͸exporterΛ࡞Δ͔ɺίϛϡχςΟ͔ΒԠ౴͕͋Ε͹࿩ͯ͠PR ౤͍͖͍͛ͯͨ(ͳΜ͔collectorଆͰେ͖͍มߋ͕ඞཁͳؾ΋͢Δ)

Slide 35

Slide 35 text

܅΋PUFMʹͳΒͳ͍͔ʁ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU