Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
OpenCensusでcustom context propagationとexporterを...
Search
takashabe
June 27, 2019
Technology
0
1.5k
OpenCensusでcustom context propagationとexporterを書いた話 / OpenCensus with custom context propagation and exporter
OpenCensus/OpenTelemetry meetup vol.2 の発表資料です。
https://opencensus.connpass.com/event/132588/
takashabe
June 27, 2019
Tweet
Share
More Decks by takashabe
See All by takashabe
より良いターミナルでの生活を求めて
takashabe
0
25
pubsub with concurrent
takashabe
1
840
社内ISUCONを開催した話
takashabe
0
1.5k
ISUCON大反省会
takashabe
0
1.8k
gitのブランチ戦略
takashabe
8
5.7k
サルでもわかるgit
takashabe
0
1.3k
playで複数DBする
takashabe
0
1.6k
MySQLで高トラフィックに立ち向かう
takashabe
0
1.7k
GitHubの良さ
takashabe
2
2.2k
Other Decks in Technology
See All in Technology
隣接領域をBeyondするFinatextのエンジニア組織設計 / beyond-engineering-areas
stajima
1
270
Why App Signing Matters for Your Android Apps - Android Bangkok Conference 2024
akexorcist
0
130
マルチプロダクトな開発組織で 「開発生産性」に向き合うために試みたこと / Improving Multi-Product Dev Productivity
sugamasao
1
300
New Relicを活用したSREの最初のステップ / NRUG OKINAWA VOL.3
isaoshimizu
2
590
エンジニア人生の拡張性を高める 「探索型キャリア設計」の提案
tenshoku_draft
1
120
強いチームと開発生産性
onk
PRO
34
11k
これまでの計測・開発・デプロイ方法全部見せます! / Findy ISUCON 2024-11-14
tohutohu
3
370
ノーコードデータ分析ツールで体験する時系列データ分析超入門
negi111111
0
410
TypeScriptの次なる大進化なるか!? 条件型を返り値とする関数の型推論
uhyo
2
1.6k
Application Development WG Intro at AppDeveloperCon
salaboy
0
190
Incident Response Practices: Waroom's Features and Future Challenges
rrreeeyyy
0
160
ExaDB-D dbaascli で出来ること
oracle4engineer
PRO
0
3.8k
Featured
See All Featured
Side Projects
sachag
452
42k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
16
2.1k
How GitHub (no longer) Works
holman
310
140k
Rails Girls Zürich Keynote
gr2m
94
13k
Writing Fast Ruby
sferik
627
61k
Thoughts on Productivity
jonyablonski
67
4.3k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
26
1.4k
The Cost Of JavaScript in 2023
addyosmani
45
6.7k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
42
9.2k
A designer walks into a library…
pauljervisheath
204
24k
How STYLIGHT went responsive
nonsquared
95
5.2k
Gamification - CAS2011
davidbonilla
80
5k
Transcript
OpenCensus/OpenTelemetry meetup vol.2 @takashabe OpenCensusͰcustom context propagationͱexporterΛॻ͍ ͨ
whoami • Takashi Abe (@takashabe) • גࣜձࣾαΠόʔΤʔδΣϯτ ΞυςΫελδΦ • GoͰࠂ৴ϓϩμΫτΛ࡞͍ͬͯ·͢
• KubernetesͰMicroservicesతͳͷ • ਪ͠ΩʔϘʔυirisͰ͢ • Corne Cherry࡞த…
͡Ίʹ • Go + OpenCensus + Stackdriver Λલఏͱ͍ͯ͠·͢
࣍ • ࢄτϨʔγϯάͷ֓ཁ • context propagation • exporter • ·ͱΊ
ࢄτϨʔγϯά
ࢄτϨʔγϯά https://github.com/census-instrumentation/opencensus-go • 1 traceʹରͯ͠ɺෳͷspan͕ඥͮ͘ • spanಉҰΞϓϦέʔγϣϯͰෳൃߦͯ͠ྑ͍͠ɺҟͳΔΞϓϦέʔγϣϯͰ τϨʔεID͕ಉҰͰ͋Εඥͮ͘
context propagation
context propagation is …
context propagation is … • τϨʔεIDΛൖ͢ΔͨΊͷϓϩτίϧతͳͷ • opencensus-goͰocgrpcochttpͰ࣮͞Ε͍ͯΔ • SpanContextΛհͯ͠σʔλΛൖͤ͞Δ
• TraceID, SpanID, TraceOptions(αϯϓϦϯάର͔Ͳ ͏͔) type SpanContext struct { TraceID TraceID // [16]byte SpanID SpanID // [8]byte TraceOptions TraceOptions // uint32 Tracestate *tracestate.Tracestate // Ұ෦ͷpropagation༻ }
ocgrpc, ochttp ocgrpc ochttp
HTTPFormatͷ࣮ • ocgrpc࣮͕1ͭ • ochttpͰHTTPFormatΛ࣮ͨ͠b3ͱtracecontextͱ͍͏2 ͭͷ࣮͕ଘࡏ͢Δ • b3: ZipkinͰఆٛ͞Εͨpropagation(σϑΥϧτ) •
https://github.com/openzipkin/b3-propagation • tracecontext: w3cͰఆٛ͞Εͨpropagation • https://github.com/w3c/trace-context type HTTPFormat interface { SpanContextFromRequest(req *http.Request) (sc trace.SpanContext, ok bool) SpanContextToRequest(sc trace.SpanContext, req *http.Request) }
B3 Propagation https://github.com/openzipkin/b3-propagation
B3 Propagation https://github.com/openzipkin/b3-propagation 4QBO$POUFYU'SPN3FRVFTU 4QBO$POUFYU5P3FRVFTU
OpenCensusͰͷ࣮(ToRequest) const ( TraceIDHeader = "X-B3-TraceId" SpanIDHeader = "X-B3-SpanId" SampledHeader
= "X-B3-Sampled" ) func (f *HTTPFormat) SpanContextToRequest(sc trace.SpanContext, req *http.Request) { req.Header.Set(TraceIDHeader, hex.EncodeToString(sc.TraceID[:])) req.Header.Set(SpanIDHeader, hex.EncodeToString(sc.SpanID[:])) var sampled string if sc.IsSampled() { sampled = "1" } else { sampled = "0" } req.Header.Set(SampledHeader, sampled) } https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/propagation/b3/b3.go
OpenCensusͰͷ࣮(FromRequest) const ( TraceIDHeader = "X-B3-TraceId" SpanIDHeader = "X-B3-SpanId" SampledHeader
= "X-B3-Sampled" ) func (f *HTTPFormat) SpanContextFromRequest(req *http.Request) (sc trace.SpanContext, ok bool) { tid, ok := ParseTraceID(req.Header.Get(TraceIDHeader)) if !ok { return trace.SpanContext{}, false } sid, ok := ParseSpanID(req.Header.Get(SpanIDHeader)) if !ok { return trace.SpanContext{}, false } sampled, _ := ParseSampled(req.Header.Get(SampledHeader)) return trace.SpanContext{ TraceID: tid, SpanID: sid, TraceOptions: sampled, }, true } https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/propagation/b3/b3.go
ochttpΛར༻͢Δ // import "go.opencensus.io/plugin/ochttp" mux := http.NewServeMux() mux.Handle("/users", ochttp.WithRouteTag(usersHandler, "/users"))
log.Fatal(http.ListenAndServe("localhost:8080", &ochttp.Handler{ Handler: mux, Propagation: &b3.HTTPFormat{}, })) https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/example_test.go • ochttp.Handler͕ϛυϧΣΞͱͯ͠τϨʔγϯάपΓ Λ໘ݟͯ͘ΕΔ • context propagationҙࣝ͠ͳͯ͘ྑ͍
custom context propagation
Cloud Pub/Sub ʹద༻ͨ͠ • GCPͷϚωʔδυPub/Sub • B3 PropagationΛϕʔεʹͯ͠ɺϝοηʔδͷ AttributesʹσʔλΛ٧Ίͯൖͤ͞Δ •
https://github.com/takashabe/oc-propagation-demo • ΄΅ಉ͡ͷΛϓϩμΫγϣϯʹಋೖࡁΈ͕ͩɺڥ ґଘ͕͋ΔͷͰαϯϓϧ࣮Λ༻ҙ • αʔό༻ҙ͢Δͷ໘ͰCloud Run༻ʹHTTPαʔό ͱ࣮͍ͯͯ͠͠ΔͷͰएׯݟͮΒ͍
Pub/Sub൛ͷ࣮(ToRequest) const ( TraceIDField = "X-Pubsub-TraceId" SpanIDField = "X-Pubsub-SpanId" SampledField
= "X-Pubsub-Sampled" ) func WrapMessage(ctx context.Context, m *pubsub.Message) *pubsub.Message { if m.Attributes != nil { if m.Attributes[TraceIDField] != "" || m.Attributes[SpanIDField] != "" { return m } } else { m.Attributes = make(map[string]string, 3) } sc := trace.FromContext(ctx).SpanContext() m.Attributes[TraceIDField] = sc.TraceID.String() m.Attributes[SpanIDField] = sc.SpanID.String() m.Attributes[SampledField] = fmt.Sprintf("%t", sc.IsSampled()) return m }
Pub/Sub൛ͷ࣮(FromRequest) const ( TraceIDField = "X-Pubsub-TraceId" SpanIDField = "X-Pubsub-SpanId" SampledField
= "X-Pubsub-Sampled" ) func SpanContextFromMessage(m *pubsub.Message) trace.SpanContext { if m.Attributes == nil { return trace.SpanContext{} } tid, ok := parseTraceID(m.Attributes[TraceIDField]) if !ok { return trace.SpanContext{} } sid, ok := parseSpanID(m.Attributes[SpanIDField]) if !ok { return trace.SpanContext{} } sampled := parseSampled(m.Attributes[SampledField]) return trace.SpanContext{ TraceID: tid, SpanID: sid, TraceOptions: sampled, } }
ར༻ଆ // import “github.com/takashabe/oc-propagation-demo/internal/propagation" ... ctx, span := trace.StartSpan(context.Background(), "publish")
defer span.End() msg := &pubsub.Message{ Data: []byte("foo"), } topic.Publish(ctx, propagation.WrapMessage(ctx, msg)).Get(ctx) // import “github.com/takashabe/oc-propagation-demo/internal/propagation" ... subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { sc := propagation.SpanContextFromMessage(msg) _, span := trace.StartSpanWithRemoteParent(ctx, "receive", sc) defer span.End() publisher(client) subscriber(server)
Stackdriver Trace্ͷදࣔ publisher subscriber Pub/Sub
exporter
exporter is …
exporter is … • StackdriverDatadogͳͲͷόοΫΤϯυαʔϏεʹ τϨʔεɺ͋Δ͍ϝτϦΫεΛૹΔͨΊͷΈ • σʔλऩू෦ͱexporterʹΑΔόοΫΤϯυαʔϏε ͕͞Ε͍ͯΔͷ͕OpenCensusͷ1ͭͷಛ
exporterͷ࣮(Go) type Exporter interface { ExportSpan(s *SpanData) } • ExporterΠϯλϑΣʔεΛ࣮͢Δ͚ͩͰྑ͍
• ҙͷόοΫΤϯυαʔϏεAPIͳͲΛୟ͍ͯσʔλΛ ૹ͍ͬͯΔ • ࣮ࡍʹύϑΥʔϚϯεͰ͔ͳΓؾΛ͍ͬͯΔ(ͣ)
custom exporter
Tail LatencyΛิ͢ΔͨΊͷexporter • Tail Latency • 99ύʔηϯλΠϧͷΑ͏ͳͷ • ௨ৗɺτϥϑΟοΫͷଟ͍αʔϏεͩͱαϯϓϦϯά ͷඞཁ͕͋Δ͕ɺͦ͏͢ΔͱTail
Latency͕શ͘ه͞ Εͳ͍ • Tail LatencyͰΜͰ͍Δͱ͖मਖ਼ɺܭଌͷαΠΫ ϧΛճ͍͕ͨ͠ɺ؍ଌ͢ΔͨΊʹ࣌ؒҎ্͔͔Δ ͜ͱ͋Δ…
edge exporter • https://github.com/takashabe/edge-exporter • ϓϩμΫγϣϯೖग़དྷͯͳ͍ͷͰࢀߟ࣮ͱͯ͠… • ύϑΥʔϚϯεվળͷ༨͕େ͍ʹͬͯΔ • ߏΛ࿅͍ͬͯΔ࣌ʹϓϩδΣΫτҠಈʹͳͬͨ
• ଞexporterͷલஈͰproxyͱͯ͠ಈ࡞ͤ͞Δ • શ݅αϯϓϦϯάͭͭ͠ɺҰఆ࣌ؒ͝ͱʹ࠷ϨΠςϯ γͷߴ͍τϨʔε͚ͩΛ࠾༻͢Δ • Φʔόʔϔου͋Δ
spanϥΠϑαΠΫϧ • ࠓճͷΑ͏ʹTail LatencyΛั·͑ΔͨΊʹશ݅αϯ ϓϦϯά͢Δඞཁ͕ग़ͯ͘Δ • αϯϓϦϯάରͰͳ͍spanʹର͢Δૢ࡞ૣظϦ λʔϯ͢ΔΑ͏ʹͳ͍ͬͯΔ • αϯϓϦϯάରʹͳ͍ͬͯΔ͚ͩͰएׯͷΦʔόʔ
ϔου͕͋Δ _, span := trace.StartSpan(ctx, name) // αϯϓϦϯάܾఆ. SpanDatasੜ span.AddAttributes(...) // αϯϓϦϯάର֎ͳΒεΩοϓ ... span.End() // ExportSpan͕ݺΕΔ
edge exporter࣮ type EdgeExporter struct { // Stackdriver exporterͳͲͷଞexporterΛอ࣋͢Δ exporters
exportersList // Tail LatencyͷܭଌִؒͱRateLimit interval time.Duration limiter *rate.Limiter // interval͝ͱͷTail LatencyΛอ࣋͢Δ tail *trace.SpanData tailLatency time.Duration tailMu sync.Mutex } • EdgeExporterߏମ
edge exporter࣮ • ExportSpan() func (e *EdgeExporter) ExportSpan(sd *trace.SpanData) {
e.storeTailLatencySpan(sd) if !e.limiter.Allow() { return } e.tailMu.Lock() defer e.tailMu.Unlock() if e.tail == nil { return } for _, exp := range e.exporters.Load() { exp.ExportSpan(e.tail) } e.tail = nil e.tailLatency = 0 }
example func main() { sd, _ := stackdriver.NewExporter(stackdriver.Options{ ProjectID: os.Getenv("PROJECT_ID"),
}) edge := edgeexporter.New(edgeexporter.WithExportInterval(10*time.Second)) edge.RegisterExporter(sd) trace.RegisterExporter(edge) trace.ApplyConfig(trace.Config{ DefaultSampler: trace.AlwaysSample(), }) http.HandleFunc("/", handler) http.ListenAndServe(":8080", nil) } var cnt int64 func handler(w http.ResponseWriter, req *http.Request) { _, span := trace.StartSpan(context.Background(), "handler") c := atomic.LoadInt64(&cnt) if c%5 == 0 { time.Sleep(50 * time.Millisecond) } span.End() atomic.AddInt64(&cnt, 1) } https://github.com/takashabe/edge-exporter/blob/master/example/main.go
example func main() { sd, _ := stackdriver.NewExporter(stackdriver.Options{ ProjectID: os.Getenv("PROJECT_ID"),
}) edge := edgeexporter.New(edgeexporter.WithExportInterval(10*time.Second)) edge.RegisterExporter(sd) trace.RegisterExporter(edge) trace.ApplyConfig(trace.Config{ DefaultSampler: trace.AlwaysSample(), }) http.HandleFunc("/", handler) http.ListenAndServe(":8080", nil) } var cnt int64 func handler(w http.ResponseWriter, req *http.Request) { _, span := trace.StartSpan(context.Background(), "handler") c := atomic.LoadInt64(&cnt) if c%5 == 0 { time.Sleep(50 * time.Millisecond) } span.End() atomic.AddInt64(&cnt, 1) } https://github.com/takashabe/edge-exporter/blob/master/example/main.go 10ඵ͝ͱʹTail Latency SpanΛ Stackdriver exporterʹྲྀ͢
Stackdriver Trace্ͷදࣔ $ echo "GET https://…” | vegeta attack -rate=50
-duration=1m
custom exporterͷεεϝ • ࠓճTail LatencyΛରͱ͕ͨ͠ɺྫ͑Τϥʔ͕ൃ ੜͨ͠ͱ͖͚ͩɺಛఆͷϦΫΤετ͚ͩͳͲԠ༻ޮ ͖ͦ͏ • ܭଌ։࢝࣌(StartSpan)Ͱஅग़དྷͳ͍ͷΛܭଌ͢ ΔͨΊʹࠓճͷΑ͏ʹύϑΥʔϚϯεͱͷτϨʔυ
ΦϑʹͳΓͦ͏
Conclusion
Conclusion • Cloud Pub/Sub༻ͷcontext propagationͱɺTail Latency༻ͷexporterΛ࡞ͬͯΈͨ • ಠ࣮ࣗͷϋʔυϧͦΕ΄Ͳߴ͘ͳ͍ • ؍ଌ͍ͨ͠ͷΛ؍ଌग़དྷΔΑ͏ʹ͠Α͏