Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
OpenCensusでcustom context propagationとexporterを...
Search
takashabe
June 27, 2019
Technology
0
1.7k
OpenCensusでcustom context propagationとexporterを書いた話 / OpenCensus with custom context propagation and exporter
OpenCensus/OpenTelemetry meetup vol.2 の発表資料です。
https://opencensus.connpass.com/event/132588/
takashabe
June 27, 2019
Tweet
Share
More Decks by takashabe
See All by takashabe
より良いターミナルでの生活を求めて
takashabe
0
55
pubsub with concurrent
takashabe
1
920
社内ISUCONを開催した話
takashabe
0
1.7k
ISUCON大反省会
takashabe
0
1.9k
gitのブランチ戦略
takashabe
8
5.9k
サルでもわかるgit
takashabe
0
1.5k
playで複数DBする
takashabe
0
1.6k
MySQLで高トラフィックに立ち向かう
takashabe
0
1.8k
GitHubの良さ
takashabe
2
2.2k
Other Decks in Technology
See All in Technology
BtoBプロダクト開発の深層
16bitidol
0
150
AI Agentと MCP Serverで実現する iOSアプリの 自動テスト作成の効率化
spiderplus_cb
0
400
Go Conference 2025: GoのinterfaceとGenericsの内部構造と進化 / Go type system internals
ryokotmng
3
590
From Prompt to Product @ How to Web 2025, Bucharest, Romania
janwerner
0
110
KMP の Swift export
kokihirokawa
0
310
組織観点からIAM Identity CenterとIAMの設計を考える
nrinetcom
PRO
1
150
Geospatialの世界最前線を探る [2025年版]
dayjournal
3
470
Green Tea Garbage Collector の今
zchee
PRO
2
380
非同期処理実行基盤 Delayed脱出 → Solid Queue完全移行への旅路。
srockstyle
3
1.6k
AIAgentの限界を超え、 現場を動かすWorkflowAgentの設計と実践
miyatakoji
0
110
実装で解き明かす並行処理の歴史
zozotech
PRO
1
260
C# 14 / .NET 10 の新機能 (RC 1 時点)
nenonaninu
1
1.4k
Featured
See All Featured
A designer walks into a library…
pauljervisheath
208
24k
Building Applications with DynamoDB
mza
96
6.6k
What's in a price? How to price your products and services
michaelherold
246
12k
Thoughts on Productivity
jonyablonski
70
4.9k
The Language of Interfaces
destraynor
162
25k
Docker and Python
trallard
46
3.6k
Navigating Team Friction
lara
189
15k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
A Modern Web Designer's Workflow
chriscoyier
697
190k
Rails Girls Zürich Keynote
gr2m
95
14k
Raft: Consensus for Rubyists
vanstee
139
7.1k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Transcript
OpenCensus/OpenTelemetry meetup vol.2 @takashabe OpenCensusͰcustom context propagationͱexporterΛॻ͍ ͨ
whoami • Takashi Abe (@takashabe) • גࣜձࣾαΠόʔΤʔδΣϯτ ΞυςΫελδΦ • GoͰࠂ৴ϓϩμΫτΛ࡞͍ͬͯ·͢
• KubernetesͰMicroservicesతͳͷ • ਪ͠ΩʔϘʔυirisͰ͢ • Corne Cherry࡞த…
͡Ίʹ • Go + OpenCensus + Stackdriver Λલఏͱ͍ͯ͠·͢
࣍ • ࢄτϨʔγϯάͷ֓ཁ • context propagation • exporter • ·ͱΊ
ࢄτϨʔγϯά
ࢄτϨʔγϯά https://github.com/census-instrumentation/opencensus-go • 1 traceʹରͯ͠ɺෳͷspan͕ඥͮ͘ • spanಉҰΞϓϦέʔγϣϯͰෳൃߦͯ͠ྑ͍͠ɺҟͳΔΞϓϦέʔγϣϯͰ τϨʔεID͕ಉҰͰ͋Εඥͮ͘
context propagation
context propagation is …
context propagation is … • τϨʔεIDΛൖ͢ΔͨΊͷϓϩτίϧతͳͷ • opencensus-goͰocgrpcochttpͰ࣮͞Ε͍ͯΔ • SpanContextΛհͯ͠σʔλΛൖͤ͞Δ
• TraceID, SpanID, TraceOptions(αϯϓϦϯάର͔Ͳ ͏͔) type SpanContext struct { TraceID TraceID // [16]byte SpanID SpanID // [8]byte TraceOptions TraceOptions // uint32 Tracestate *tracestate.Tracestate // Ұ෦ͷpropagation༻ }
ocgrpc, ochttp ocgrpc ochttp
HTTPFormatͷ࣮ • ocgrpc࣮͕1ͭ • ochttpͰHTTPFormatΛ࣮ͨ͠b3ͱtracecontextͱ͍͏2 ͭͷ࣮͕ଘࡏ͢Δ • b3: ZipkinͰఆٛ͞Εͨpropagation(σϑΥϧτ) •
https://github.com/openzipkin/b3-propagation • tracecontext: w3cͰఆٛ͞Εͨpropagation • https://github.com/w3c/trace-context type HTTPFormat interface { SpanContextFromRequest(req *http.Request) (sc trace.SpanContext, ok bool) SpanContextToRequest(sc trace.SpanContext, req *http.Request) }
B3 Propagation https://github.com/openzipkin/b3-propagation
B3 Propagation https://github.com/openzipkin/b3-propagation 4QBO$POUFYU'SPN3FRVFTU 4QBO$POUFYU5P3FRVFTU
OpenCensusͰͷ࣮(ToRequest) const ( TraceIDHeader = "X-B3-TraceId" SpanIDHeader = "X-B3-SpanId" SampledHeader
= "X-B3-Sampled" ) func (f *HTTPFormat) SpanContextToRequest(sc trace.SpanContext, req *http.Request) { req.Header.Set(TraceIDHeader, hex.EncodeToString(sc.TraceID[:])) req.Header.Set(SpanIDHeader, hex.EncodeToString(sc.SpanID[:])) var sampled string if sc.IsSampled() { sampled = "1" } else { sampled = "0" } req.Header.Set(SampledHeader, sampled) } https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/propagation/b3/b3.go
OpenCensusͰͷ࣮(FromRequest) const ( TraceIDHeader = "X-B3-TraceId" SpanIDHeader = "X-B3-SpanId" SampledHeader
= "X-B3-Sampled" ) func (f *HTTPFormat) SpanContextFromRequest(req *http.Request) (sc trace.SpanContext, ok bool) { tid, ok := ParseTraceID(req.Header.Get(TraceIDHeader)) if !ok { return trace.SpanContext{}, false } sid, ok := ParseSpanID(req.Header.Get(SpanIDHeader)) if !ok { return trace.SpanContext{}, false } sampled, _ := ParseSampled(req.Header.Get(SampledHeader)) return trace.SpanContext{ TraceID: tid, SpanID: sid, TraceOptions: sampled, }, true } https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/propagation/b3/b3.go
ochttpΛར༻͢Δ // import "go.opencensus.io/plugin/ochttp" mux := http.NewServeMux() mux.Handle("/users", ochttp.WithRouteTag(usersHandler, "/users"))
log.Fatal(http.ListenAndServe("localhost:8080", &ochttp.Handler{ Handler: mux, Propagation: &b3.HTTPFormat{}, })) https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/example_test.go • ochttp.Handler͕ϛυϧΣΞͱͯ͠τϨʔγϯάपΓ Λ໘ݟͯ͘ΕΔ • context propagationҙࣝ͠ͳͯ͘ྑ͍
custom context propagation
Cloud Pub/Sub ʹద༻ͨ͠ • GCPͷϚωʔδυPub/Sub • B3 PropagationΛϕʔεʹͯ͠ɺϝοηʔδͷ AttributesʹσʔλΛ٧Ίͯൖͤ͞Δ •
https://github.com/takashabe/oc-propagation-demo • ΄΅ಉ͡ͷΛϓϩμΫγϣϯʹಋೖࡁΈ͕ͩɺڥ ґଘ͕͋ΔͷͰαϯϓϧ࣮Λ༻ҙ • αʔό༻ҙ͢Δͷ໘ͰCloud Run༻ʹHTTPαʔό ͱ࣮͍ͯͯ͠͠ΔͷͰएׯݟͮΒ͍
Pub/Sub൛ͷ࣮(ToRequest) const ( TraceIDField = "X-Pubsub-TraceId" SpanIDField = "X-Pubsub-SpanId" SampledField
= "X-Pubsub-Sampled" ) func WrapMessage(ctx context.Context, m *pubsub.Message) *pubsub.Message { if m.Attributes != nil { if m.Attributes[TraceIDField] != "" || m.Attributes[SpanIDField] != "" { return m } } else { m.Attributes = make(map[string]string, 3) } sc := trace.FromContext(ctx).SpanContext() m.Attributes[TraceIDField] = sc.TraceID.String() m.Attributes[SpanIDField] = sc.SpanID.String() m.Attributes[SampledField] = fmt.Sprintf("%t", sc.IsSampled()) return m }
Pub/Sub൛ͷ࣮(FromRequest) const ( TraceIDField = "X-Pubsub-TraceId" SpanIDField = "X-Pubsub-SpanId" SampledField
= "X-Pubsub-Sampled" ) func SpanContextFromMessage(m *pubsub.Message) trace.SpanContext { if m.Attributes == nil { return trace.SpanContext{} } tid, ok := parseTraceID(m.Attributes[TraceIDField]) if !ok { return trace.SpanContext{} } sid, ok := parseSpanID(m.Attributes[SpanIDField]) if !ok { return trace.SpanContext{} } sampled := parseSampled(m.Attributes[SampledField]) return trace.SpanContext{ TraceID: tid, SpanID: sid, TraceOptions: sampled, } }
ར༻ଆ // import “github.com/takashabe/oc-propagation-demo/internal/propagation" ... ctx, span := trace.StartSpan(context.Background(), "publish")
defer span.End() msg := &pubsub.Message{ Data: []byte("foo"), } topic.Publish(ctx, propagation.WrapMessage(ctx, msg)).Get(ctx) // import “github.com/takashabe/oc-propagation-demo/internal/propagation" ... subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { sc := propagation.SpanContextFromMessage(msg) _, span := trace.StartSpanWithRemoteParent(ctx, "receive", sc) defer span.End() publisher(client) subscriber(server)
Stackdriver Trace্ͷදࣔ publisher subscriber Pub/Sub
exporter
exporter is …
exporter is … • StackdriverDatadogͳͲͷόοΫΤϯυαʔϏεʹ τϨʔεɺ͋Δ͍ϝτϦΫεΛૹΔͨΊͷΈ • σʔλऩू෦ͱexporterʹΑΔόοΫΤϯυαʔϏε ͕͞Ε͍ͯΔͷ͕OpenCensusͷ1ͭͷಛ
exporterͷ࣮(Go) type Exporter interface { ExportSpan(s *SpanData) } • ExporterΠϯλϑΣʔεΛ࣮͢Δ͚ͩͰྑ͍
• ҙͷόοΫΤϯυαʔϏεAPIͳͲΛୟ͍ͯσʔλΛ ૹ͍ͬͯΔ • ࣮ࡍʹύϑΥʔϚϯεͰ͔ͳΓؾΛ͍ͬͯΔ(ͣ)
custom exporter
Tail LatencyΛิ͢ΔͨΊͷexporter • Tail Latency • 99ύʔηϯλΠϧͷΑ͏ͳͷ • ௨ৗɺτϥϑΟοΫͷଟ͍αʔϏεͩͱαϯϓϦϯά ͷඞཁ͕͋Δ͕ɺͦ͏͢ΔͱTail
Latency͕શ͘ه͞ Εͳ͍ • Tail LatencyͰΜͰ͍Δͱ͖मਖ਼ɺܭଌͷαΠΫ ϧΛճ͍͕ͨ͠ɺ؍ଌ͢ΔͨΊʹ࣌ؒҎ্͔͔Δ ͜ͱ͋Δ…
edge exporter • https://github.com/takashabe/edge-exporter • ϓϩμΫγϣϯೖग़དྷͯͳ͍ͷͰࢀߟ࣮ͱͯ͠… • ύϑΥʔϚϯεվળͷ༨͕େ͍ʹͬͯΔ • ߏΛ࿅͍ͬͯΔ࣌ʹϓϩδΣΫτҠಈʹͳͬͨ
• ଞexporterͷલஈͰproxyͱͯ͠ಈ࡞ͤ͞Δ • શ݅αϯϓϦϯάͭͭ͠ɺҰఆ࣌ؒ͝ͱʹ࠷ϨΠςϯ γͷߴ͍τϨʔε͚ͩΛ࠾༻͢Δ • Φʔόʔϔου͋Δ
spanϥΠϑαΠΫϧ • ࠓճͷΑ͏ʹTail LatencyΛั·͑ΔͨΊʹશ݅αϯ ϓϦϯά͢Δඞཁ͕ग़ͯ͘Δ • αϯϓϦϯάରͰͳ͍spanʹର͢Δૢ࡞ૣظϦ λʔϯ͢ΔΑ͏ʹͳ͍ͬͯΔ • αϯϓϦϯάରʹͳ͍ͬͯΔ͚ͩͰएׯͷΦʔόʔ
ϔου͕͋Δ _, span := trace.StartSpan(ctx, name) // αϯϓϦϯάܾఆ. SpanDatasੜ span.AddAttributes(...) // αϯϓϦϯάର֎ͳΒεΩοϓ ... span.End() // ExportSpan͕ݺΕΔ
edge exporter࣮ type EdgeExporter struct { // Stackdriver exporterͳͲͷଞexporterΛอ࣋͢Δ exporters
exportersList // Tail LatencyͷܭଌִؒͱRateLimit interval time.Duration limiter *rate.Limiter // interval͝ͱͷTail LatencyΛอ࣋͢Δ tail *trace.SpanData tailLatency time.Duration tailMu sync.Mutex } • EdgeExporterߏମ
edge exporter࣮ • ExportSpan() func (e *EdgeExporter) ExportSpan(sd *trace.SpanData) {
e.storeTailLatencySpan(sd) if !e.limiter.Allow() { return } e.tailMu.Lock() defer e.tailMu.Unlock() if e.tail == nil { return } for _, exp := range e.exporters.Load() { exp.ExportSpan(e.tail) } e.tail = nil e.tailLatency = 0 }
example func main() { sd, _ := stackdriver.NewExporter(stackdriver.Options{ ProjectID: os.Getenv("PROJECT_ID"),
}) edge := edgeexporter.New(edgeexporter.WithExportInterval(10*time.Second)) edge.RegisterExporter(sd) trace.RegisterExporter(edge) trace.ApplyConfig(trace.Config{ DefaultSampler: trace.AlwaysSample(), }) http.HandleFunc("/", handler) http.ListenAndServe(":8080", nil) } var cnt int64 func handler(w http.ResponseWriter, req *http.Request) { _, span := trace.StartSpan(context.Background(), "handler") c := atomic.LoadInt64(&cnt) if c%5 == 0 { time.Sleep(50 * time.Millisecond) } span.End() atomic.AddInt64(&cnt, 1) } https://github.com/takashabe/edge-exporter/blob/master/example/main.go
example func main() { sd, _ := stackdriver.NewExporter(stackdriver.Options{ ProjectID: os.Getenv("PROJECT_ID"),
}) edge := edgeexporter.New(edgeexporter.WithExportInterval(10*time.Second)) edge.RegisterExporter(sd) trace.RegisterExporter(edge) trace.ApplyConfig(trace.Config{ DefaultSampler: trace.AlwaysSample(), }) http.HandleFunc("/", handler) http.ListenAndServe(":8080", nil) } var cnt int64 func handler(w http.ResponseWriter, req *http.Request) { _, span := trace.StartSpan(context.Background(), "handler") c := atomic.LoadInt64(&cnt) if c%5 == 0 { time.Sleep(50 * time.Millisecond) } span.End() atomic.AddInt64(&cnt, 1) } https://github.com/takashabe/edge-exporter/blob/master/example/main.go 10ඵ͝ͱʹTail Latency SpanΛ Stackdriver exporterʹྲྀ͢
Stackdriver Trace্ͷදࣔ $ echo "GET https://…” | vegeta attack -rate=50
-duration=1m
custom exporterͷεεϝ • ࠓճTail LatencyΛରͱ͕ͨ͠ɺྫ͑Τϥʔ͕ൃ ੜͨ͠ͱ͖͚ͩɺಛఆͷϦΫΤετ͚ͩͳͲԠ༻ޮ ͖ͦ͏ • ܭଌ։࢝࣌(StartSpan)Ͱஅग़དྷͳ͍ͷΛܭଌ͢ ΔͨΊʹࠓճͷΑ͏ʹύϑΥʔϚϯεͱͷτϨʔυ
ΦϑʹͳΓͦ͏
Conclusion
Conclusion • Cloud Pub/Sub༻ͷcontext propagationͱɺTail Latency༻ͷexporterΛ࡞ͬͯΈͨ • ಠ࣮ࣗͷϋʔυϧͦΕ΄Ͳߴ͘ͳ͍ • ؍ଌ͍ͨ͠ͷΛ؍ଌग़དྷΔΑ͏ʹ͠Α͏