How to send distibuted traces to Datadog using build own OpenTelemetry-Lambda distribution

Slide 1

Slide 1 text

2024-06-24 @ OpenTelemetry Meetup 2024-06 OpenTelemetryͱAWS Lambda ͱDatadog id:aereal

Slide 2

Slide 2 text

࿩͢͜ͱ • AWS Lambda͔ΒOpenTelemetryΛ஻ͬͯDatadogʹτϨʔεΛૹΔํ๏ • AWS Lambda Extensionͷ֓ཁ • opentelemetry-lambdaͷ֓ཁ • AWS Distribution for OpenTelemetry Lambdaͱopentelemetry-lambdaͷؔ܎ • DatadogʹτϨʔεΛૹΔͨΊʹඞཁͳΧελϜϏϧυͷ஌ࣝ

Slide 3

Slide 3 text

• id:aereal • GitHub: aereal • @[email protected] • גࣜձࣾClassi ֶशPMF෦ ΞʔΩςΫτ ࣗݾ঺հ

Slide 4

Slide 4 text

঎༻؀ڥͰOpenTelemetry Collector Λಈ͔͍ͯ͠Δਓʑʙ✋

Slide 5

Slide 5 text

ΧελϜϏϧυͨ͠   OpenTelemetry CollectorΛ࢖͍ͬͯΔਓʑʙ✋

Slide 6

Slide 6 text

AWS Lambda࢖͍ͬͯΔਓʑʙ✋

Slide 7

Slide 7 text

FaaS࢖͍ͬͯΔਓʑʙ✋

Slide 8

Slide 8 text

΍Γ͍ͨ͜ͱ • AWS Lambda͔ΒOpenTelemetryΛ஻ͬͯDatadogʹτϨʔεΛૹΓ͍ͨ • Lambdaͷݺͼग़͠΋౰વ؍ଌ͍ͨ͠ • OpenTelemetryͷΤίγεςϜʹऒ͔ΕͯҠߦͨ͠ͷͰ  OTelҎ֎ͷϥΠϒϥϦ͸࢖͍ͨ͘ͳ͍ • refs. https://tech.classi.jp/entry/opentelemetry-migration • DatadogΛ׆༻͍ͯ͠ΔͷͰAWS X-RayͦͷଞʹσʔλΛ෼ࢄͤͨ͘͞ͳ͍

Slide 9

Slide 9 text

DatadogͱClassi

Slide 10

Slide 10 text

OTelͱLambda: opentelemetry-lambda • OTel collectorΛLambda Extensionͱͯ͠ಈ͔࣮͢૷ • Lambda Extension APIΛ஻ͬͯcollectorͷϓϩηεΛಈ͔͢ • Layerͱͯ͠഑෍͞Ε͍ͯΔ • refs. https://docs.aws.amazon.com/lambda/latest/dg/chapter-layers.html

Slide 11

Slide 11 text

AWS Lambda Extension • 2020೥ʹϦϦʔε͞Εͨؔ਺ຊମʹखΛೖΕͣؔ਺Λ֦ு͢ΔΤίγεςϜ • ॴఆͷҐஔʹόΠφϦΛஔ͍͓ͯ͘ͱInit, Restore, Invoke, ShutdownͦΕͧΕ ͷλΠϛϯάͰݺͼग़͞ΕΔ • InitΠϕϯτͰผϓϩηεΛ্ཱͪ͛ͨΓͳͲͰ͖Δ • ֦ுͷొ࿥ʹ͸࣮ߦ࣌ؒͷ੍໿͕͋Γ͜ΕΛ௒͑Δͱొ࿥ʹࣦഊ͠ɺͦͷ֦ு ͸࣮ߦ͞Εͳ͍ • ؔ਺ຊମͷ࣮ߦʹ͸Өڹ͠ͳ͍ɺ͋͘·Ͱແࢹ͞ΕΔ͚ͩ

Slide 12

Slide 12 text

https://docs.aws.amazon.com/images/lambda/latest/dg/images/Overview-Invoke-with-Error.png

Slide 13

Slide 13 text

https://docs.aws.amazon.com/images/lambda/latest/dg/images/telemetry-api-concept-diagram.png

Slide 14

Slide 14 text

OTelͱLambda (2) • AWS Distro for OpenTelemetry Lambda (ADOT Lambda) • AWS͕ϝϯς͍ͯ͠Δopentelemetry-lambdaͷσΟετϦϏϡʔγϣϯ • AWS͕બΜͩίϯϙʔωϯτ͔ΒͳΔcollectorΛಈ͔͢ • ͜Εʹdatadogexporter͸ؚ·Ε͍ͯͳ͍ˠࣗલͰϏϧυ͢Δ΄͔ͳ͍

Slide 15

Slide 15 text

https://github.com/aws-observability/aws-otel-lambda

Slide 16

Slide 16 text

OTel LambdaͷΧελϜϏϧυ • OTel collectorͷΧελϜϏϧυͱߟ͑ํ͸ಉ͡ • ࢖͍͍ͨίϯϙʔωϯτΛimportͯ͠Factoryʹ౉ͯ͋͛͠Δ • ocb͸͋͘·ͰcollectorΛϏϧυ͢ΔπʔϧͳͷͰ࢖͑ͳ͍͜ͱʹ஫ҙ • https://opentelemetry.io/docs/collector/custom-collector/

Slide 17

Slide 17 text

OTel LambdaͷΧελϜϏϧυ 1. opentelemetry-lambdaΛforkͯ͠ 2. collector/lambdacomponents/default.goʹ࢖͍͍ͨίϯϙʔωϯτΛ஫ೖ͠ 3. make͢Δ https://github.com/open-telemetry/opentelemetry-lambda/tree/main/collector

Slide 18

Slide 18 text

https://github.com/open-telemetry/opentelemetry-lambda/blob/c55ad8742d9d4714a950fe4eaf11ed3246673577/collector/ lambdacomponents/default.go

Slide 19

Slide 19 text

࣮ࡍͷΧελϚΠζ ͓඼ॻ͖ • datadogexporterͷ௥Ճ • AWS Parameters Store confmap providerͷ௥Ճ • otel-lambda΁҆શʹAPIΩʔΛ౉͢ํ๏Λఏڙ͢Δ

Slide 20

Slide 20 text

https://github.com/open-telemetry/opentelemetry-lambda/compare/main...aereal:opentelemetry-lambda:support- datadog-exporter#di ff -f619591f6445be6380476b6844bbbe99a603301d4ddd5a9c9bb0ec2e59662894

Slide 21

Slide 21 text

Lambdaͱൿಗ৘ใ • Lambdaؔ਺ʹσϓϩΠෆཁͰ஋Λ౉͢ํ๏͸؀ڥม਺ͱAmazon EFSͷΈ • ؀ڥม਺͸ؔ਺ͷઃఆʹฏจͰهड़͢Δ • Amazon EFS͸ϚωʔδυNFSαʔϏε • → ͲͪΒ΋ൿಗ৘ใͷड͚౉͠ํ๏ͱͯ͠ෆద౰ • Parameters StoreͱSecrets Managerͱ͍͏αʔϏεʹอଘ͠ɺ  otel-lambda͔ΒAPIΛ࢖ͬͯ͜ΕΒ͔Β஋Λऔಘ͢Ε͹Αͦ͞͏

Slide 22

Slide 22 text

confmapͱ͸ • OTel collectorͷઃఆʹ֎෦σʔλιʔε͔Βऔಘͨ͠஋ΛຒΊࠐΉ࢓૊Έ • ${env:DD_API_KEY} ͱॻ͘ͱ DD_API_KEY ͱ͍͏؀ڥม਺ͷ஋Ͱஔ͖׵͑Β ΕΔ • envͷ෦෼͕confmap providerͷ໊લΛࢦ͢ • ଞʹs3ͳͲ͕͋Δ

Slide 23

Slide 23 text

https://github.com/open-telemetry/opentelemetry-lambda/compare/main...aereal:opentelemetry-lambda:support- datadog-exporter#di ff -f301b7c709892efae0bdfdecfca8fea1aa2114d8c621b5e9993cd61390e07c43

Slide 24

Slide 24 text

https://this.aereal.org/entry/the-art-of-otel-col-confmap-provider

Slide 25

Slide 25 text

Ͱ͖͕͋ͬͨϏϧυͷ഑෍ͱར༻ • GitHub Packages Container RegistryΛ࢖ͬͯ഑෍ • Lambdaؔ਺͸ίϯςφΠϝʔδΛσϓϩΠ͢Δ͜ͱ͕Մೳ • ϕʔεΠϝʔδʹࢦఆͯ͋͛͠Δ • ద੾ʹݖݶΛઃఆ͢Ε͹GitHub Actions͔ΒγʔϜϨεʹpullͰ͖ͯศར

Slide 26

Slide 26 text

ࡶײ • ແࣄɺ໨త͸Ռͨ͞Εͨ͠OTelΤίγεςϜͷཧղ͕ਂ·ͬͨ • ΧελϜϏϧυͦͷ΋ͷ͸ඞཁͳෳࡶͩ͞ͱײ͡Δ • αϓϥΠνΣʔϯ߈ܸͷϦεΫ͕ೝ஌͞Ε͍ͯΔࡢࠓɺґଘπϦʔΛ୯७ʹอͪ೺ Ѳ͠ଓ͚Δҙٛ͸େ͖͍ • ܧଓతͳ্ྲྀͷมߋऔΓࠐΈ͕͍ͨ΁Μͳͷ͸ͳΜͱ͔͍ͨ͠ • ͳͥͳΒਓʑ͸େมͳ͜ͱ͸ແࢹͨ͘͠ͳΔ͔Β…… • ͱʹ্͔͘ྲྀΛrebaseͯ͠ςετ͕௨Ε͹Α͍ͷͰɺ͋ͱ͸΍Δ͚ͩ……

Slide 27

Slide 27 text

Fin.