Perlでも分散トレーシングしたい!AWS::XRay による解析とその実装 / YAPC::Tokyo 2019

Perlでも分散トレーシングしたい!AWS::XRay による解析とその実装 / YAPC::Tokyo 2019

YAPC::Tokyo 2019 のトークです

Ca6281fff64797dc419b78f51f25c0a5?s=128

FUJIWARA Shunichiro

January 26, 2019
Tweet

Transcript

  1. 11.

    ෼ࢄτϨʔγϯάͷ࣮૷ͱ࢓༷ Zipkin(OpenZipkin)2: Twitter → OSS 2012 Jeager3: Uber → CNCF

    OSS 2016 OpenTracing4: ෼ࢄτϨʔγϯάͷ࢓༷ CNCF 2016 AWS X-Ray: AWS͕ఏڙ ϚωʔδυαʔϏε 2016 StackDriver Trace: Google͕ఏڙ ZipkinΫϥΠΞϯτޓ׵ 2017 4 https://opentracing.io/ 3 https://www.jaegertracing.io/ 2 https://zipkin.io/
  2. 14.
  3. 16.

    AWS X-Ray ͷߏ੒ཁૉ6 X-Ray API AWS͕ఏڙ͍ͯ͠ΔAPI X-Ray daemon τϨʔεసૹ༻σʔϞϯ X-Ray

    SDK X-Ray daemonʹτϨʔεΛૹ৴͢Δͨ ΊͷϥΠϒϥϦ X-Ray console τϨʔεΛՄࢹԽ͢ΔViewer 6 https://docs.aws.amazon.com/ja_jp/xray/latest/devguide/aws-xray.html
  4. 18.

    ͲͷΑ͏ʹτϨʔεΛૹ৴͢Δ͔ X-Ray daemon7 ͕ UDP port 2000 Λ listen ΞϓϦέʔγϣϯ(SDK)͔ΒτϨʔεσʔλΛUDPͰ౤͛Δ

    {"format": "json", "version": 1} { "trace_id": "1-5759e988-bd862e3fe1be46a994272793", "id": "defdfd9912dc5a56", "start_time": 1461096053.37518, "end_time": 1461096053.4042, "name": "MyApp" } 7 https://github.com/aws/aws-xray-daemon
  5. 19.

    ͲͷΑ͏ʹτϨʔεΛૹ৴͢Δ͔ X-Ray daemon ͕όοϑΝϦϯάͯ͠όονͰ X-Ray API ʹૹ৴ X-Ray API ʹಧ͍ͨ͋ͱ͸׬શʹϚωʔδυ

    σʔλอ࣋ 30೔(ݹ͍ͷ͸ফ͑Δ) τϨʔε਺ͱදࣔ࣌ͷεΩϟϯͨ͠ྔʹΑΔ׬શैྔ՝ۚ
  6. 23.

    X-Ray SDK τϨʔεΛ X-Ray daemon ʹૹΔϥΠϒϥϦ AWS͕ఏڙ͍ͯ͠Δͷ͸ Java, Go, Node,

    Python, Ruby, .NET ͋Εʁࠓ೔͸ YAPC (Yet Another Perl Conference)…
  7. 26.

    AWS::XRay Λॻ͖·ͨ͠ ཁ͢Δʹ JSON Λ UDP Ͱ౤͛Ε͹͍͍͚ͩ ͳͷͰ؆୯ 300ߦ͙Β͍͔͠ͳ͍ capture

    $name, sub {} ͰηάϝϯτΛ࡞ͬͯૹ৴ use AWS::XRay qw/ capture /; capture "myApp", sub { capture "foo", sub { # do something ... capture "bar", sub { # ... }; }; };
  8. 29.

    Devel::KYTProf::Logger::XRay Devel::KYTProf ͱ͍͏ϓϩϑΝΠϥͷϩΨʔͱͯ͠ಈ࡞ͯ͠ τϨʔεσʔλΛૹ৴ Devel::KYTProf ͸ use ͢Δ͚ͩͰDBIͳͲͷݺͼग़͠Λܭଌ ϩάΛग़ྗͯ͘͠ΕΔศརϓϩϑΝΠϥ use

    Devel::KYTProf; # your code 315.837 ms [DBI::st] select * from table where name = ? (1 rows) | main:23 1464.204 ms [LWP::UserAgent] GET http://www.hatena.ne.jp/ | main:25
  9. 31.

    AWS::XRay ։ൃܦҢ Perl Ͱ΋ X-Ray ࢖͍͍ͨ… → JSONΛUDPͰ౤͛Δ͚ͩͩ͠؆୯ͩΑͶʁ → Ͱ΋τϨʔε࢓ࠐΉͷΊΜͲ͍͘͞ͳ…

    → ! Devel::KYTProf ͕औͬͨ݁ՌΛૹΕ͹ʂ → Logger ࠩ͠ସ͑Ͱ͖Δ KYTProf++ ਖ਼௚ KYTProf ͕ͳ͔ͬͨΒ࡞ͬͯͳ͔͔ͬͨ΋͠Εͳ͍
  10. 33.

    ISUCON is ͳʹ ͍͍ײ͡ʹεϐʔυΞοϓίϯςετͷུ ఏڙ͞Εͨαʔό্Ͱಈ͘WebΞϓϦέʔγϣϯΛߴ଎Խ !!!!!!!!!! ༏উ৆ۚ100ສԁ 2011೥͔Βຖ೥։࠵͞Ε͍ͯΔ (ओ࠵ LINEࣾ)

    2018೥ͷ ISUCON 8 Λ DeNA ͞ΜͱΧϠοΫͰग़୊ ࣍ͷ࿮͸ karupanerura ͞ΜʹΑΔ ʮISUCON8༧બ໰୊࡞੒ͷཪଆʯͷτʔΫͰ͢
  11. 37.

    Perl ࣮૷ʹ AWS::XRay Λ૊ΈࠐΉ ఏڙ͞Εͨݴޠ࣮૷͸ Go, Perl, PHP, Python, Ruby

    YAPC ͳͷͰ౰વ Perl ࣮૷Ͱ https://github.com/isucon/isucon8-final
  12. 38.

    X-Ray daemon Λ༻ҙ ॳظ࣮૷͸ docker-compose ͳͷͰద౰ʹ Dockerfile Λ༻ҙ8 FROM amazonlinux

    RUN yum install -y unzip RUN curl -o daemon.zip https://s3.dualstack.us-east-2.amazonaws.com/ aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-linux-3.x.zip RUN unzip daemon.zip && cp xray /usr/bin/xray ENTRYPOINT ["/usr/bin/xray", "-b", "0.0.0.0:2000", "--local-mode"] EXPOSE 2000/udp 8 https://docs.aws.amazon.com/ja_jp/xray/latest/devguide/xray-daemon-local.html
  13. 39.

    docker-compose.yml services: isucoin: environment: # ུ AWS_XRAY_DAEMON_ADDRESS: 'xray:2000' links: -

    mysql - xray xray: build: xray environment: AWS_REGION: "ap-northeast-1" AWS_ACCESS_KEY_ID: "000000000000" AWS_SECRET_ACCESS_KEY: "****"
  14. 40.

    AWS::XRay ૊ΈࠐΈ app.psgi ʹ5ߦ௥Ճ + cpanfile # app.psgi use Devel::KYTProf::Logger::XRay;

    use AWS::XRay; Devel::KYTProf->logger("Devel::KYTProf::Logger::XRay"); builder { enable 'XRay', name => 'isucoin'; # ... # cpanfile requires 'AWS::XRay'; requires 'Plack::Middleware::XRay'; requires 'Devel::KYTProf::Logger::XRay';
  15. 42.
  16. 43.
  17. 44.
  18. 45.
  19. 46.
  20. 47.
  21. 48.
  22. 49.

    ϘτϧωοΫൃݟ SELECT * FROM trade ORDER BY id DESC 23ສߦΛಡΈऔͬͯ1ߦ࢖ࣺ͍ͬͯͯͯΔ

    LIMIT 1 ͢Ε͹ OK ! ʮ͜Ε͸΂ͭʹ෼ࢄτϨʔγϯά͍Βͳ͍ΑͶ?ʯ ! ʮslowlog ݟΕ͹෼͔Δ΍Ζʯ
  23. 51.
  24. 52.
  25. 53.
  26. 55.

    ֎෦API࣮૷ʹ X-Ray Λ૊ΈࠐΜͰΈΔ (ڝٕͰ͸࣮ࡍ͸ϒϥοΫϘοΫε͚ͩͲ) xray-sdk-go Λ૊ΈࠐΈ // sql.Open -> xray.SQL

    db, err := xray.SQL("mysql", dsn) // http.Handler Λ xray.Handler Ͱ wrap return xray.Handler( xray.NewFixedSegmentNamer("Isubank"), myHandler, )
  27. 56.

    Perl ଆͷ HTTPϦΫΤετૹ৴΋ capture ݺͼग़͠ݩΛ఻͑ΔͨΊʹ X-Amzn-Trace-ID ϔομΛ෇༩ capture "externalHTTP", sub

    { my $segment = shift; $res = $self->client->post( $self->endpoint . $p, [ "Content-Type" => "application/json", "Authorization" => "Bearer " . $self->app_id, "X-Amzn-Trace-ID" => $segment->trace_header, ], $body, ); };
  28. 58.
  29. 59.
  30. 63.

    ֎෦ϦΫΤετݺͼग़͠ʹϔομΛ෇༩ use AWS::XRay qw/capture/; capture "externalHTTP", sub { # Ͳ͜Ͱ΋ݺ΂Δ(਌͕͍ͳ͚Ε͹RootʹͳΔ)

    my $segment = shift; $furl->POST( "https://example.com", [ "X-Amzn-Trace-ID" => $segment->trace_header ], $body, ); }; POST / HTTP/1.1 Host: example.com X-Amzn-Trace-ID: Root=1-5c4902d7-e8fe620cab66e01e57df8a12;Parent=851cf9ec07b528c2 ...
  31. 64.

    ϔομΛड৴ͨ͠API(Isubank)͕Ҿ͖ܧ͙ Root=1-5c4902d7-e8fe620cab66e01e57df8a12;Parent=851cf9ec07b528c2 # ࣮ࡍ͸Go͚ͩͲઆ໌ͷͨΊʹPerl AWS::XRay::Segment->new( id => new_id(), trace_id =>

    "1-5c4902d7-e8fe620cab66e01e57df8a12", parent_id => "851cf9ec07b528c2", ); ͦͷτϨʔεΛ X-Ray daemon ΁ૹ৴ → X-Ray consoleͰτϨʔε͕࿈݁ͯ͠දࣔ͞ΕΔ
  32. 66.

    τϨʔε͍ͨ͠ͱ͜ΖͰʮݱࡏͷʯRoot΍Parent͕ඞཁ package App; sub main { capture "App::main", sub {

    # Root my $segment = shift; Model::call($param); # ֎෦ݺͼग़͠Λ൐͏ॲཧ }; }; package Model; sub call() { my $param = shift; # ࣮ࡍʹ֎ΛݺͿͷ͸͕ͩ͜͜ݺͼग़͠ݩͷsegmentΛ஌Βͳ͍ͱRoot͕Θ͔Βͳ͍ $furl->post($url, [ "X-Amzn-Trace-ID" => "????", ], $param); } call($param)Λcall($segment,$param)ʹॻ͖׵͑ͨ͘ͳ͍
  33. 67.

    AWS::XRay Ͱ͸Ҿ਺Λ౉͞ͳ͍͍ͯ͘Α͏ʹͳͬͯΔ package App; sub main { capture "App::main", sub

    { # Root my $segment = shift; Model::call($param); # ֎෦ݺͼग़͠Λ൐͏ॲཧ }; }; package Model; sub call() { my $param = shift; # ਌ͷ͜ͱ͸஌Βͳ͍͚Ͳͱʹ͔͘ capture ͢Ε͹Α͍ capture "Model::call", sub { my $segment = shift; # ਌͔Βݺ͹Ε͍ͯΕ͹ࣗಈతʹࢠʹͳΔ $furl->post($url, [ "X-Amzn-Trace-ID" => $segment->trace_header, ], $param); }; }
  34. 68.

    Ͳ͏΍࣮ͬͯ૷͞Ε͍ͯΔ͔ Perl ͷຐ๏ͷͻͱͭ local perldoc -f local ͋ͳ͕ͨຊ౰ʹ๬ΜͰ͍Δͷ͸ my ͷํͰ͠ΐ͏;

    local ͸΄ͱΜ Ͳͷਓʑ͕ʮϩʔΧϧʯͱߟ͑Δ΋ͷͱ ҧ͏͔ΒͰ͢ɻ "local" ͸ϦετΞοϓ͞Εͨม਺Λɺғ͍ͬͯΔϒϩοΫɺ ϑΝΠϧɺeval ͷதͰɺϩʔΧϧͳ΋ͷʹ͠·͢ɻ !!!!!!!
  35. 69.

    local → μΠφϛοΫείʔϓ (ݺͼग़͠ݩͰείʔϓ͕ܾ·Δ) my → ϨΩγΧϧείʔϓ (ιʔείʔυͷࣈ໘Ͱείʔϓ͕ܾ·Δ) package main;

    our $Foo = "A"; # ύοέʔδม਺ sub foo { print $Foo; } foo(); #= A { local $Foo = "B"; # ͜ͷϒϩοΫ͔Βݺ͹Ε͍ͯΔؒ͸B foo(); #= B } foo(); #= A ϒϩοΫΛൈ͚Δͱࣗಈతʹ໭Δ
  36. 70.

    Կճωετͯ͠ݺͼग़ͯ͠΋ॱ൪ʹ্ॻ͖ͯ͠ɺ΋ͲΕ͹෮ݩ foo(); #= A { local $Foo = "B"; foo();

    #= B { local $Foo = "C"; foo(); #= C { local $Foo = "D"; foo() #= D } foo(); #= C } foo(); #= B } foo(); #= A
  37. 71.

    AWS::XRay Ͱ͸۩ମతʹ͸͜Μͳײ͡ ݱࡏͷ Trace-ID, Segment-ID Λ local Ͱอ࣋ package AWS::XRay;

    our $TRACE_ID; # Root our $SEGMENT_ID; # ݱࡏͷSegmentID sub capture { my ($name, $code) = @_; # $TRACE_ID͕ະఆٛͳΒRootͳͷͰ৽نIDൃߦ local ʹ local $TRACE_ID = $TRACE_ID // new_trace_id(); # ৽͍͠ηάϝϯτΛ࡞Δ # $SEGMENT_ID ͕͋Ε͹ parent_id => $SEGMENT_ID ʹͳ͍ͬͯΔ my $segment = AWS::XRay::Segment->new({ name => $name, type => $SEGMENT_ID ? "subsegment" : undef, parent_id => $SEGMENT_ID, });
  38. 72.

    local $SEGMENT_ID = $segment->{id}; # ৽͍͠ηάϝϯτIDΛ্ॻ͖ͯ͠localʹଋറ # ίʔυΛ࣮ߦ # ͜ͷதͰ͸

    $SEGMENT_ID ͕ઃఆ͞Ε͍ͯΔͷͰ # ͞Βʹ capture ͨ͠Β $segment ͷࢠڙʹͳΔ $ret = $code->($segment); $segment->close(); # ηάϝϯτσʔλૹ৴ return $ret; # $segment ͸ফ͑Δ(my͔ͩΒ) # $SEGMENT_ID ΋ݺͼग़࣌͠ͷ஋ʹ໭Δ # RootͳΒ $TRACE_ID ΋ফ͑Δ(ݩ͕ະఆ͔ٛͩΒ) }
  39. 74.

    ༨ஊ Go Ͱ X-Ray SDK Λ࢖͏৔߹ ϦΫΤετͰൃੜͨ͠ context.Context ΛͲ͜·Ͱ΋ Ҿ਺ʹ౉͍͔ͯ͠ͳ͍ͱτϨʔεͰ͖ͳ͍

    ్தʹ ctx ΛҾ͖ܧ͛ͳ͍ϥΠϒϥϦ(ORMͱ͔)͕ڬ·Δͱ (Isubank ΋࣮ࡍ͸40ՕॴҎ্ ctx Λ౉͢Α͏ʹॻ͖׵͑ͨ)
  40. 76.

    ! ίετͷ࿩ ʮX-Ray ͸Αͦ͞͏͚ͩͲɺ͓ߴ͍ΜͰ͠ΐ͏…?ʯ ௚઀ίετ = X-Ray ར༻ྉۚ ؒ઀ίετ =

    ΞϓϦέʔγϣϯͷύϑΥʔϚϯεྼԽ (݁Ռతʹ࢖༻Ϧιʔε͕૿͑ΔɺϢʔβମݧ͕ଛͳΘΕΔͳͲ)
  41. 77.

    ΞϓϦέʔγϣϯͷύϑΥʔϚϯεྼԽ τϨʔεΛ૊ΈࠐΉ͜ͱʹΑͬͯͲΕ͙Β͍஗͘ͳΔ͔ • 1ॲཧͰ100ݸͷ capture (ίʔυ͸ۭ sub{}) Λੜ੒ • αϯϓϦϯά͠ͳ͍ͰશͯͷτϨʔεΛૹ৴

    Λ 500/sec ఔ౓࣮ߦͰ͖Δ (macbook pro 13 2017) ͭ·Γ1ॲཧ͋ͨΓ2msఔ౓ͷΦʔόʔϔου τϨʔεΛૹ৴͠ͳ͍(captureΛωετ͢Δ͚ͩ)ͳΒ 10,000/sec, 0.1msఔ౓ͷΦʔόʔϔου
  42. 79.

    X-Ray ར༻ྉۚ ຖ݄ͷແྉ࿮ τϨʔεͷه࿥͸ 10 ສճ·Ͱແྉ τϨʔεͷऔಘͱεΩϟϯ͸߹Θͤͯ 100 ສճ·Ͱແྉ ௥ՃͰ

    τϨʔεͷه࿥ 100 ສ݅͋ͨΓ 5 USD τϨʔεͷऔಘͱεΩϟϯΛ߹Θͤͯ 100 ສ݅͋ͨΓ 0.50 USD
  43. 80.

    ࣮ࡍͲΕ͙Βֻ͍͔Δͷʁ ฏۉ 100 req/sec ͷ৔߹ 100 * 86,400 * 30

    / 1,000,000 * 5 USD = 1,296 USD ฏۉ 1,000 req/sec ͳΒ 12,960 USD / month શͯͷϦΫΤετʹ͍ͭͯه࿥͢Δͷ͸ίετ͕ߴ͍
  44. 82.

    αϯϓϦϯάͷ޻෉ ྫɿ1ඵ͝ͱʹ࠷ॳͷϦΫΤετͱ ௥ՃϦΫΤετͷ5%Λه࿥ (ެࣜSDKͷσϑΥϧτઃఆ) 100 req/sec ͷ৔߹ 1 + 100

    * 5% = 6 req/sec ෼͕ه࿥͞ΕΔ ϦΫΤετ͕গͳͯ͘΋ 1 req/sec ͸࣮֬ʹه࿥ ެࣜ SDK ͸ X-Ray console ͔ΒαϯϓϦϯάϧʔϧΛઃఆՄ (AWS::XRay Ͱ͸ະରԠ…)
  45. 85.

    αϯϓϦϯάϧʔϧΛίʔυͰهड़ PSGI $env Λड͚औͬͯਅِΛฦ͢ίʔυΛࢦఆͰ͖Δ enable "XRay" name => "myApp", sampler

    => sub { my $env = shift; state %paths; if ( $paths{$env->{PATH_INFO}++ == 0 ) { # ಛఆͷPATHΛॳΊͯड৴ͨ͠৔߹͸ඞͣه࿥ return 1; } rand() < 0.05; # ͦΕҎ֎͸ 5% };
  46. 86.

    Τϥʔ΍஗͍ϦΫΤετ͸ඞͣه࿥͍ͨ͠ 5% αϯϓϦϯάˠ ΤϥʔΛه࿥Ͱ͖Δ֬཰΋5% શϦΫΤετͰϝϞϦ্ʹτϨʔεΛอଘ͓ͯ͘͠ ϨεϙϯεΛฦ࣌͢఺ͰτϨʔεΛૹ৴͢Δ͔Ͳ͏͔ΛܾΊΔ enable "XRay" name =>

    "myApp", response_filter => sub { my ($env, $res, $elapsed) = @_; # 5xx OR 1ඵҎ্ֻ͔ͬͨ৔߹ʹૹ৴ return $res->[0] >= 500 || $elapsed >= 1; };
  47. 87.