An introduction to monitoring Go Application with OpenCensus

Transcript

1. An introduc+on to monitoring Go Applica+on with OpenCensus @munisystem

2. Agenda • APM ͱ͸ͳʹ͔ɺݱঢ়ͷ՝୊ • OpenCensus ͷ֓ཁ • OpenCensus ʹΑΔ

   Go ͷύϑΥʔϚϯεϞχλϦϯά

3. APM ͱ͸ͳʹ͔ɺݱঢ়ͷ՝୊

4. αʔϏε։ൃʹ͓͚ΔϝτϦΫε αʔϏε͕͖ͪΜͱఏڙͰ͖͍ͯΔͷ͔ΛܭΔͨΊͷࢦඪ = ϝτϦΫε ϝτϦΫε͸େ͖͘3ͭʹ෼ྨͰ͖Δ • Business metrics • Applica1on

   metrics • Infrastructure metrics

5. Business metrics Ϗδωεͷ໨ඪΛୡ੒͢ΔͨΊͷݸʑͷϓϩηεʹର͢ΔϝτϦ Ϋε • Conversion Rate • DAU /

   WAU / MAU • PV • Ac7vity Log

6. Applica'on metrics ΞϓϦέʔγϣϯͷύϑΥʔϚϯεʹର͢ΔϝτϦΫε • Throughput / API latency • RPS

   / RPM • Error rate • Language specific metrics

7. Infrastructure metrics ΞϓϦέʔγϣϯΛσϦόϦʔ͢Δ؀ڥʹର͢ΔϝτϦΫε • Memory / CPU / Disk usage

   • Load Average • Network latency

8. Applica'on metrics ͷॏཁੑ Կނ Applica(on metrics Λऔಘ͢Δ͜ͱ͕ॏཁͳͷ͔ʁ • ύϑΥʔϚϯε͕αʔϏεʹ༩͑ΔΠϯύΫτ •

   αʔϏεͷ੒௕ͱػೳ௥Ճʹର͢ΔજࡏతͳϦεΫ • ϚΠΫϩαʔϏεʹΑΔύϑΥʔϚϯε෼ੳͷࠔ೉Խ

9. ύϑΥʔϚϯε͕αʔϏεʹ༩͑ΔΠϯύΫτ ΞϓϦέʔγϣϯͷύϑΥʔϚϯε͕ CVR ΍ϦςϯγϣϯʹӨڹ Λ༩͑Δ͜ͱ͕Ұൠతʹ஌ΒΕ͍ͯΔ 1 Our experiments demonstrate that

   slowing down the search results page by 100 to 400 milliseconds has a measurable impact on the number of searches per user of -0.2% to -0.6% (averaged over four or six weeks depending on the experiment). That's 0.2% to 0.6% fewer searches for changes under half a second! 1 Speed Ma)ers h)ps:/ /ai.googleblog.com/2009/06/speed-ma)ers.html

10. αʔϏεͷ੒௕ͱػೳ௥Ճʹର͢ΔજࡏతͳϦεΫ αʔϏε͕੒௕͢ΔʹͭΕΞϓϦέʔγϣϯͷن໛ͱ૊৫΋େ͖ ͘ͳΓɺ·ͨજࡏతͳϦεΫ΋ߴ͘ͳΔ • σʔλྔͷ૿Ճʹ଱͑ΒΕͳ͍ SQL ͷൃߦ • ผαʔόʹର͢Δ API

    call ͷ૿Ճ αʔϏεͷ৴པੑΛҡ࣋͢ΔͨΊʹ໰୊Λ͍ͪૣ͘ݕ஌͠ରԠ͢ Δ͜ͱ͕ٻΊΒΕΔ

11. ɹϚΠΫϩαʔϏεʹΑΔύϑΥʔϚϯε෼ੳͷࠔ೉Խ ϚΠΫϩαʔϏεͰ͸αʔϏεؒͷ௨৴͕ෳࡶԽɾෆಁ໌Խ͢Δ ͦͷ݁Ռͱͯ͠αʔϏεશମͷৼΔ෣͍΍αʔϏεͷґଘؔ܎ͷ ೺ѲɺγεςϜͷύϑΥʔϚϯεͷ෼ੳɺ·ͨΤϥʔ࣌ʹͲͷ αʔϏε͕໰୊ͩͬͨͷ͔ͷௐ͕ࠪϞϊϦγοΫΞʔΩςΫνϟ ͱൺֱͯ͠ࠔ೉ʹͳΔ શମ૾Λ೺Ѳ͠ϘτϧωοΫΛௐࠪ͢ΔͨΊͷج൫ͱͯ͠෼ࢄτ Ϩʔγϯά͕ඞཁ

12. APM (Applica+on Performance Monitoring) ఏڙ͍ͯ͠ΔαʔϏεͷ৴པੑͱՁ஋Λམͱ͞ͳ͍ͨΊʹɺΞϓ ϦέʔγϣϯͷύϑΥʔϚϯεΛ؂ࢹɺՄࢹԽ͢Δ͜ͱʹΑͬͯ ໰୊Λૣظൃݟ͠ະવʹ๷͙ͨΊͷ࢓૊Έ

13. APM ͷݱঢ় APM ͸େ͖͘2ͭͷ༻్͕͋ΓɺͦΕʹԠͨ͡αʔϏεͷΫϥΠ ΞϯτΛར༻͍ͯ͠Δ • ΞϓϦέʔγϣϯݻ༗ͷϝτϦΫεͷՄࢹԽɾ؂ࢹ • Datadog /

    CloudWatch / Mackerel / Prometheus • ϚΠΫϩαʔϏεͷαʔϏεؒ௨৴ͷՄࢹԽ (෼ࢄτϨʔγϯ ά) • Datadog APM / Stackdriver Trace / Amazon X-Ray / Zipkin

14. APM ͷ՝୊ ϝτϦΫεΛऩू͠όοΫΤϯυʹૹΔػߏ͸େ൒ͷΫϥΠΞϯ τͰ͸ڞ௨Խ͞Ε͍ͯͳ͍ͨΊɺΞϓϦέʔγϣϯͷ࣮૷͕αʔ Ϗεʹґଘͯ͠͠·͏ ͜Ε͸૊৫΍νʔϜɺΞϓϦέʔγϣϯΛߏங͢ΔΠϯϑϥʹ Αͬͯద੾ͳαʔϏεΛબͿΫϥ΢υωΠςΟϒͳ։ൃελΠϧ ʹ͸ෆ޲͖ => ಛఆͷϕϯμʔʹґଘ͠ͳ͍ڞ௨ͷϑϨʔϜϫʔΫ͕ඞཁ

15. OpenCensus ͷ֓ཁ

16. OpenCensus ͱ͸ APM ʹඞཁͳϝτϦΫεΛऩू͠ɺ֎෦αʔϏεʹఏڙ͢ΔͨΊ ͷϑϨʔϜϫʔΫ • ϝτϦΫεΛऩू͢ΔͨΊͷϓϩτίϧٴͼ֎෦αʔϏεʹఏ ڙ͢ΔͨΊͷΠϯλʔϑΣʔεͷఆٛ • ্هΛ࣮૷֤ͨ͠ݴޠͷϥΠϒϥϦ

    • Go / Ruby / Python / Java / C++ / Erlang / PHP

17. Ecosystem Design 2 2 h$ps:/ /raw.githubusercontent.com/census-instrumenta8on/opencensus-specs/master/drawings/ EcosystemLayers.png

18. Ecosystem Design ొ৔ਓ෺ • ϝτϦΫεΛѻ͏ͨΊͷ OpenCensus Library • ্هΛར༻ͯ͠ Web

    Framework ͳͲͰࣗಈతʹϝτϦΫεΛ ऩू͢ΔͨΊͷ Library • औಘͨ͠ϝτϦΫεΛ֎෦αʔϏεʹఏڙ͢ΔͨΊͷ Client Library (Exporter)

19. Exporter OpenCensus Library ଆͰϝτϦΫεΛ֎෦αʔϏεʹఏڙ͢Δͨ ΊͷΠϯλʔϑΣʔε͕ఆٛ͞Ε͍ͯΔ αʔϏεϓϩόΠμ͸ΠϯλʔϑΣʔεΛຬͨ͢Α͏ʹϝτϦΫ εΛม׵͠όοΫΤϯυʹૹΔΑ͏ͳ Client Library Λ࣮૷͢Δ͜

    ͱͰ OpenCensus ͷ࿮૊Έʹ৐Δ͜ͱ͕Ͱ͖Δ type Exporter interface { ExportSpan(s *SpanData) }

20. Components 3 3 h$ps:/ /raw.githubusercontent.com/census-instrumenta8on/opencensus-specs/master/drawings/ LibraryComponents.png

21. Components Request local ʹϝτϦΫεΛऔΓѻ͏ͨΊͷ Context ͷ࣮૷ͱɺ ͦΕΛར༻ͨ͠ϝτϦΫεͷͨΊͷίϯϙʔωϯτ • Trace API

    • ෼ࢄτϨʔγϯάΛ࣮ݱ͢ΔͨΊͷίϯϙʔωϯτ • Tags API (Stats API) • ΞϓϦέʔγϣϯݻ༗ͷϝτϦΫεΛऔΓѻ͏ͨΊͷίϯ ϙʔωϯτ

22. OpenCensus Λ࢖͏͜ͱͷϝϦοτ • ։ൃऀ • ڞ௨ͷϑϨʔϜϫʔΫΛ࢖͏͜ͱͰɺϝτϦΫεΛऩू͢Δ ࣮૷Λมߋ͢Δ͜ͱͳ͘αʔϏεͷ੾Γସ͕͑ՄೳʹͳΔ • αʔϏεϓϩόΠμ •

    Exporter ʹରԠ͢Δ͜ͱͰɺࣗ෼ͷαʔϏεͷ࣮૷Λม͑Δ ͜ͱͳ͘ OpenCensus ͷ࿮૊Έʹ৐Δ͜ͱ͕Ͱ͖Δ

23. OpenCensus ʹΑΔ Go ͷ ύϑΥʔϚϯεϞχλϦϯά

24. Go ͷ OpenCensus Library ͷ࣮૷ঢ়گ Go ͸ http ͱ google.golang.org/grpc

    ͰϝτϦΫεΛऔΓ ѻ͏ͨΊ࣮૷͕ OpenCensus Library ʹؚ·Ε͍ͯΔ • Request local ͳίϯςΩετٴͼ࣍ͷαʔόʹ఻ൖͤ͞ΔͨΊ ͷ࣮૷ • ͲͷΑ͏ͳσʔλߏ଄Ͱ఻ൖͤ͞Δ͔ͷ࢓༷͸ W3C ͷ Working Group ͰܾΊΒΕ͍ͯΔ • RPC ʹඥͮ͘ϝτϦΫεΛཪͰऔಘ͢ΔͨΊͷ࣮૷

25. Go ͷ OpenCensus Library ͷ࣮૷ঢ়گ Exporter ʹҎԼͷ࣮૷͕͋Δ • Prometheus •

    Stackdriver • Amazon X-Ray • DataDog • and more...

26. OpenCensus + Stackdriver ʹΑΔࢀߟ࣮૷ Stackdriver ༻ͷ Exporter ͷ ini1alize exporter,

    err := stackdriver.NewExporter(stackdriver.Options{ ProjectID: projectID, }) if err != nil { log.Fatal(err) }

27. OpenCensus + Stackdriver ʹΑΔࢀߟ࣮૷ http.Handler ʹର͢ΔϝτϦΫε (Stats) Λऔಘɾ؅ཧ͢Δ View ͷઃఆͱɺExporter

    ͷొ࿥ view.RegisterExporter(exporter) if err := view.Register( ochttp.DefaultServerViews..., ); err != nil { log.Fatal(err) }

28. OpenCensus + Stackdriver ʹΑΔࢀߟ࣮૷ ෼ࢄτϨʔγϯάͰऔಘ͢ΔϝτϦΫε (Traces) ͷઃఆͱɺ Exporter ͷొ࿥ trace.RegisterExporter(exporter)

    trace.ApplyConfig(trace.Config{ DefaultSampler: trace.AlwaysSample(), })

29. OpenCensus + Stackdriver ʹΑΔࢀߟ࣮૷ Handler ͷઃఆ http.HandleFunc("/a", func(w http.ResponseWriter, req

    *http.Request) { client := &http.Client{Transport: &ochttp.Transport{ Propagation: &tracecontext.HTTPFormat{}, }} ctx, span := trace.StartSpan(req.Context(), "in a") defer span.End() r, _ := http.NewRequest("GET", "http://localhost:50030/b", nil) resp, err := client.Do(r.WithContext(ctx)) if err != nil { log.Println(err) } _ = resp fmt.Fprintf(w, "I am /a") }) http.HandleFunc("/b", func(w http.ResponseWriter, req *http.Request) { _, span := trace.StartSpan(req.Context(), "in b") defer span.End() fmt.Fprintf(w, "I am /b") }) log.Fatal(http.ListenAndServe(":50030", &ochttp.Handler{ Propagation: &tracecontext.HTTPFormat{}, }))

30. OpenCensus + Stackdriver ʹΑΔࢀߟ࣮૷

31. OpenCensus + Stackdriver ʹΑΔࢀߟ࣮૷

32. Conclusion αʔϏεͷ৴པੑͱύϑΥʔϚϯεΛҡ࣋͢ΔͨΊʹ Applica'on Performance Monitoring (APM) ͸ඞཁෆՄܽͰ͋Δ OpenCensus Λ࢖͏͜ͱͰόοΫΤϯυͷαʔϏεΛҙࣝͤͣʹ APM

    ʹඞཁͳϝτϦΫεͷऩू͕ՄೳʹͳΔ