Perlでも分散トレーシングしたい！AWS::XRay による解析とその実装 / YAPC::Tokyo 2019

YAPC::Tokyo 2019 のトークです

Transcript

1. PerlͰ΋෼ࢄτϨʔγϯά͍ͨ͠ʂ
   AWS::XRay ʹΑΔղੳͱͦͷ࣮૷
   ɹ
   2019.01.26 YAPC::Tokyo
   @fujiwara
   

   View Slide

2. @fujiwara
   ٕज़෦ SRE
   github.com/fujiwara
   sfujiwara.hatenablog.com
   ISUCON8 ຊબग़୊νʔϜ
   

   View Slide

3. Game & Community
   

   View Slide

4. Agenda
   ෼ࢄτϨʔγϯάͱ͸
   AWS X-Ray ͷ঺հ
   X-Ray ʹΑΔ ISUCON8 ຊબ໰୊ͷղੳྫ
   AWS::XRay ͷ಺෦࣮૷
   ίετͷ࿩
   

   View Slide

5. ෼ࢄτϨʔγϯάͱ͸
   

   View Slide

6. ෼ࢄτϨʔγϯάͱ͸
   GoogleͷDapper࿦จ1͕΋ͱ (2010೥)
   ෳ਺ͷݴޠͰߏங͞ΕΔେن໛෼ࢄγεςϜͷύϑΥʔϚϯε
   Λղੳ͢ΔͨΊʹ࡞ΒΕͨ
   1 https://ai.google/research/pubs/pub36356
   

   View Slide

7. ෼ࢄτϨʔγϯάͷඞཁੑ
   ෼ࢄγεςϜ(ϚΠΫϩαʔϏε)Ͱ͸ͻ
   ͱͭͷॲཧ͕ෳ਺ͷίϯϙʔωϯτʹ·
   ͕ͨΔ
   ύϑΥʔϚϯεͷ໰୊͕෼͔ΓͮΒ͍
   ো֐Λݟ͚ͭΔͷ΋೉͍͠
   ਤ͸ Dapper࿦จ 1 ΑΓ
   1 https://ai.google/research/pubs/pub36356
   

   View Slide

8. ෼ࢄτϨʔγϯάͰՄࢹԽ
   ෳ਺ͷίϯϙʔωϯτʹ·͕ͨͬͯॲཧ
   ͞Εͨ݁ՌΛՄࢹԽͯ͠೺ѲΛ༰қʹ
   ֤ίϯϙʔωϯτ͕ݺͼग़͞Εͨॱংɺ
   λΠϛϯάɺཁ͕ͨ࣌ؒ͠ݟ͑Δ
   

   View Slide

9. ֓೦
   Trace͸ෳ਺ͷSpanΛؚΉ༗޲ඇ८ճάϥϑ5
   5 https://github.com/opentracing/specification/blob/master/specification.md
   

   View Slide

10. ֓೦
    Span ͸ҎԼͷཁૉΛ΋ͭ
    ໊લ
    ։࢝࣌ࠁ
    ऴྃ࣌ࠁ
    Context (ID, Trace-ID, Parent)
    Tag {Key: Value},...
    Log {Key: Value},...
    

    View Slide

11. ෼ࢄτϨʔγϯάͷ࣮૷ͱ࢓༷
    Zipkin(OpenZipkin)2: Twitter → OSS 2012
    Jeager3: Uber → CNCF OSS 2016
    OpenTracing4: ෼ࢄτϨʔγϯάͷ࢓༷ CNCF 2016
    AWS X-Ray: AWS͕ఏڙ ϚωʔδυαʔϏε 2016
    StackDriver Trace: Google͕ఏڙ ZipkinΫϥΠΞϯτޓ׵ 2017
    4 https://opentracing.io/
    3 https://www.jaegertracing.io/
    2 https://zipkin.io/
    

    View Slide

12. Ͳ͏΍ͬͯτϨʔε͢Δ͔
    ΞϓϦέʔγϣϯͰ
    ΞϓϦέʔγϣϯ(SDK)ͰτϨʔεσʔ
    λΛ࡞੒ͯ͠సૹ༻ΤʔδΣϯτʹૹ৴
    ͔ͦ͜Βαʔό(ίϨΫλ)ʹૹΒΕΔ
    !
    ΞϓϦέʔγϣϯ಺෦ͷڍಈ΋τ
    ϨʔεͰ͖Δ
    "
    ݴޠʹґଘ(SDK)
    

    View Slide

13. Ͳ͏΍ͬͯτϨʔε͢Δ͔
    αʔϏεϝογϡͰ
    αʔϏεϝογϡͷσʔλϓϨʔϯ(ྫ
    Envoy) Ͱߦ͏
    !
    ݴޠʹґଘ͠ͳ͍
    "
    ΞϓϦέʔγϣϯ಺෦ͷڍಈ͸τ
    ϨʔεͰ͖ͳ͍
    

    View Slide

14. AWS X-Ray
    

    View Slide

15. AWS X-Ray
    AWS ʹΑΔ෼ࢄτϨʔγϯάͷϚωʔδυαʔϏε
    

    View Slide

16. AWS X-Ray ͷߏ੒ཁૉ6
    X-Ray API
    AWS͕ఏڙ͍ͯ͠ΔAPI
    X-Ray daemon
    τϨʔεసૹ༻σʔϞϯ
    X-Ray SDK
    X-Ray daemonʹτϨʔεΛૹ৴͢Δͨ
    ΊͷϥΠϒϥϦ
    X-Ray console
    τϨʔεΛՄࢹԽ͢ΔViewer
    6 https://docs.aws.amazon.com/ja_jp/xray/latest/devguide/aws-xray.html
    

    View Slide

17. ϚωʔδυαʔϏεͳͷͰτϨʔεΛૹΔ͚ͩͰ͙͢࢖͑Δ
    σʔλͷอଘ΍αʔόͷՄ༻ੑΛؾʹ͢Δඞཁ͕શ͘ͳ͍
    

    View Slide

18. ͲͷΑ͏ʹτϨʔεΛૹ৴͢Δ͔
    X-Ray daemon7 ͕ UDP port 2000 Λ listen
    ΞϓϦέʔγϣϯ(SDK)͔ΒτϨʔεσʔλΛUDPͰ౤͛Δ
    {"format": "json", "version": 1}
    {
    "trace_id": "1-5759e988-bd862e3fe1be46a994272793",
    "id": "defdfd9912dc5a56",
    "start_time": 1461096053.37518,
    "end_time": 1461096053.4042,
    "name": "MyApp"
    }
    7 https://github.com/aws/aws-xray-daemon
    

    View Slide

19. ͲͷΑ͏ʹτϨʔεΛૹ৴͢Δ͔
    X-Ray daemon ͕όοϑΝϦϯάͯ͠όονͰ X-Ray API ʹૹ৴
    X-Ray API ʹಧ͍ͨ͋ͱ͸׬શʹϚωʔδυ
    σʔλอ࣋ 30೔(ݹ͍ͷ͸ফ͑Δ)
    τϨʔε਺ͱදࣔ࣌ͷεΩϟϯͨ͠ྔʹΑΔ׬શैྔ՝ۚ
    

    View Slide

20. X-Ray ͷΞʔΩςΫνϟͷ͍͍ͱ͜Ζ
    τϨʔεσʔλ͸େྔʹൃੜ͢Δ
    1ϦΫΤετॲཧதʹଟ͍ͱ਺ेҎ্
    ຖճΞϓϦέʔγϣϯ͔Βಉظ௨৴(TCP)
    → ϨΠςϯγ૿Ճʹ௚݁͢Δ ಛʹো֐࣌
    X-Ray daemon ʹ͸UDPͰ౤͛ͬͺͳ͠
    → ΞϓϦέʔγϣϯ΁ͷӨڹ͕ܰඍ
    X-Ray daemon ͕མ͍ͪͯͯ΋ΞϓϦʹӨڹ͠ͳ͍
    

    View Slide

21. X-Ray ༻ޠͰ͸
    εύϯ → ηάϝϯτ
    ࠷ॳʹઆ໌ͨ͠Ұൠͷ༻ޠ(εύϯ)ͱ͸ҧ͏͚Ͳಉ͡΋ͷ
    Ҏ߱͸ X-Ray ͷ࿩ͳͷͰηάϝϯτͱݺͼ·͢
    

    View Slide

22. AWS X-Ray Λ Perl Ͱ࢖͓͏
    

    View Slide

23. X-Ray SDK
    τϨʔεΛ X-Ray daemon ʹૹΔϥΠϒϥϦ
    AWS͕ఏڙ͍ͯ͠Δͷ͸
    Java, Go, Node, Python, Ruby, .NET
    ͋Εʁࠓ೔͸ YAPC (Yet Another Perl Conference)…
    

    View Slide

24. !
    Perl SDK ͸ެࣜʹ͸ͳ͍
    CPAN ʹ΋ݟ౰ͨΒͳ͍
    X-Ray API Λୟ͘ϥΠϒϥϦ͸͋Δ (Paws::XRay)
    

    View Slide

25. ͍͍ͩͨશ෦ Perl
    ࠷ۙͷ৽͍͠ϓϩδΣΫτ͸ Go ͚ͩͲ௕೥ͷப͸ Perl
    

    View Slide

26. AWS::XRay Λॻ͖·ͨ͠
    ཁ͢Δʹ JSON Λ UDP Ͱ౤͛Ε͹͍͍͚ͩ ͳͷͰ؆୯
    300ߦ͙Β͍͔͠ͳ͍
    capture $name, sub {} ͰηάϝϯτΛ࡞ͬͯૹ৴
    use AWS::XRay qw/ capture /;
    capture "myApp", sub {
    capture "foo", sub {
    # do something ...
    capture "bar", sub {
    # ...
    };
    };
    };
    

    View Slide

27. Plack::Middleware::XRay
    ૊ΈࠐΉͱτϨʔεΛ։࢝ͯ͘͠ΕΔ Plack Middleware
    100ߦ͙Β͍
    .psgi Ͱ enable ͢Δ͚ͩ (Φϓγϣϯ͸ޙड़)
    use Plack::Builder;
    builder {
    enable "XRay",
    name => "myApp";
    $app;
    };
    

    View Slide

28. AWS::XRay ։ൃܦҢ
    Perl Ͱ΋ X-Ray ࢖͍͍ͨ…
    → JSONΛUDPͰ౤͛Δ͚ͩͩ͠؆୯ͩΑͶʁ
    → Ͱ΋τϨʔε࢓ࠐΉͷΊΜͲ͍͘͞ͳ…
    →
    !
    Devel::KYTProf ͕औͬͨ݁ՌΛૹΕ͹ʂ
    

    View Slide

29. Devel::KYTProf::Logger::XRay
    Devel::KYTProf ͱ͍͏ϓϩϑΝΠϥͷϩΨʔͱͯ͠ಈ࡞ͯ͠
    τϨʔεσʔλΛૹ৴
    Devel::KYTProf ͸ use ͢Δ͚ͩͰDBIͳͲͷݺͼग़͠Λܭଌ
    ϩάΛग़ྗͯ͘͠ΕΔศརϓϩϑΝΠϥ
    use Devel::KYTProf;
    # your code
    315.837 ms [DBI::st] select * from table where name = ? (1 rows) | main:23
    1464.204 ms [LWP::UserAgent] GET http://www.hatena.ne.jp/ | main:25
    

    View Slide

30. Devel::KYTProf::Logger::XRay
    use Devel::KYTProf::Logger::XRay;
    Devel::KYTProf->logger("Devel::KYTProf::Logger::XRay");
    ͜ΕͰDevel::KYTProfͰܭଌͰ͖Δ΋ͷ͸શͯτϨʔεର৅ʹ
    DBI, LWP::UserAgent, Furl::HTTP, Cache::Memcached::Fast,
    MogileFS::Client
    ࣗ෼Ͱର৅ϞδϡʔϧΛ௥Ճ͢Δ͜ͱ΋Մೳ (Redis::Fastͱ͔)
    

    View Slide

31. AWS::XRay ։ൃܦҢ
    Perl Ͱ΋ X-Ray ࢖͍͍ͨ…
    → JSONΛUDPͰ౤͛Δ͚ͩͩ͠؆୯ͩΑͶʁ
    → Ͱ΋τϨʔε࢓ࠐΉͷΊΜͲ͍͘͞ͳ…
    →
    !
    Devel::KYTProf ͕औͬͨ݁ՌΛૹΕ͹ʂ
    → Logger ࠩ͠ସ͑Ͱ͖Δ KYTProf++
    ਖ਼௚ KYTProf ͕ͳ͔ͬͨΒ࡞ͬͯͳ͔͔ͬͨ΋͠Εͳ͍
    

    View Slide

32. ISUCON 8 ຊબ໰୊ͷղੳ
    

    View Slide

33. ISUCON is ͳʹ
    ͍͍ײ͡ʹεϐʔυΞοϓίϯςετͷུ
    ఏڙ͞Εͨαʔό্Ͱಈ͘WebΞϓϦέʔγϣϯΛߴ଎Խ
    !!!!!!!!!!
    ༏উ৆ۚ100ສԁ
    2011೥͔Βຖ೥։࠵͞Ε͍ͯΔ (ओ࠵ LINEࣾ)
    2018೥ͷ ISUCON 8 Λ DeNA ͞ΜͱΧϠοΫͰग़୊
    ࣍ͷ࿮͸ karupanerura ͞ΜʹΑΔ
    ʮISUCON8༧બ໰୊࡞੒ͷཪଆʯͷτʔΫͰ͢
    

    View Slide

34. ISUCON 8 ຊબ໰୊
    Ծ૝௨՟औҾαʔϏε ISUCOIN
    ۜߦAPI, ϩάه࿥API͸ӡӦ͕༻ҙͨ͠
    ϒϥοΫϘοΫεͰఏڙ͞Εɺhttps Ͱ
    ϦΫΤετΛૹΔඞཁ͕͋Δ
    ֎෦APIݺͼग़͠Λ͍͔ʹ্ख͘ॲཧ͢
    Δ͔͕ϙΠϯτ
    

    View Slide

35. ISUCON 8 ຊબ໰୊
    

    View Slide

36. ISUCON 8 ຊબ໰୊Λ X-Ray Ͱղੳͯ͠ΈΔ
    

    View Slide

37. Perl ࣮૷ʹ AWS::XRay Λ૊ΈࠐΉ
    ఏڙ͞Εͨݴޠ࣮૷͸ Go, Perl, PHP, Python, Ruby
    YAPC ͳͷͰ౰વ Perl ࣮૷Ͱ
    https://github.com/isucon/isucon8-final
    

    View Slide

38. X-Ray daemon Λ༻ҙ
    ॳظ࣮૷͸ docker-compose ͳͷͰద౰ʹ Dockerfile Λ༻ҙ8
    FROM amazonlinux
    RUN yum install -y unzip
    RUN curl -o daemon.zip https://s3.dualstack.us-east-2.amazonaws.com/
    aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-linux-3.x.zip
    RUN unzip daemon.zip && cp xray /usr/bin/xray
    ENTRYPOINT ["/usr/bin/xray", "-b", "0.0.0.0:2000", "--local-mode"]
    EXPOSE 2000/udp
    8 https://docs.aws.amazon.com/ja_jp/xray/latest/devguide/xray-daemon-local.html
    

    View Slide

39. docker-compose.yml
    services:
    isucoin:
    environment:
    # ུ
    AWS_XRAY_DAEMON_ADDRESS: 'xray:2000'
    links:
    - mysql
    - xray
    xray:
    build: xray
    environment:
    AWS_REGION: "ap-northeast-1"
    AWS_ACCESS_KEY_ID: "000000000000"
    AWS_SECRET_ACCESS_KEY: "****"
    

    View Slide

40. AWS::XRay ૊ΈࠐΈ
    app.psgi ʹ5ߦ௥Ճ + cpanfile
    # app.psgi
    use Devel::KYTProf::Logger::XRay;
    use AWS::XRay;
    Devel::KYTProf->logger("Devel::KYTProf::Logger::XRay");
    builder {
    enable 'XRay',
    name => 'isucoin';
    # ...
    # cpanfile
    requires 'AWS::XRay';
    requires 'Plack::Middleware::XRay';
    requires 'Devel::KYTProf::Logger::XRay';
    

    View Slide

41. ϕϯνΛ࣮ߦ͢Δͱ…τϨʔε͕ʂ
    

    View Slide

42. View Slide

43. View Slide

44. View Slide

45. View Slide

46. View Slide

47. View Slide

48. View Slide

49. ϘτϧωοΫൃݟ
    SELECT * FROM trade ORDER BY id DESC
    23ສߦΛಡΈऔͬͯ1ߦ࢖ࣺ͍ͬͯͯͯΔ
    LIMIT 1 ͢Ε͹ OK
    !
    ʮ͜Ε͸΂ͭʹ෼ࢄτϨʔγϯά͍Βͳ͍ΑͶ?ʯ
    !
    ʮslowlog ݟΕ͹෼͔Δ΍Ζʯ
    

    View Slide

50. ֎෦APIݺͼग़͠Λղੳ
    

    View Slide

51. View Slide

52. View Slide

53. View Slide

54. ֎෦ͷHTTP APIݺͼग़͠ͷதͰ஗͍΋ͷΛಛఆ
    (ग़୊ҙਤͰ) ֎෦API͝ͱʹϨΠςϯγʹࠩΛ෇͚͍ͯͨ
    !
    ʮ͜Ε΋ผʹʰ෼ࢄʱͰ͸ͳ͍ΑͶʯ
    !
    ʮDevel::KYTProf ͰϩάΛग़ͤ͹෼͔Δ΍Ζʯ
    

    View Slide

55. ֎෦API࣮૷ʹ X-Ray Λ૊ΈࠐΜͰΈΔ
    (ڝٕͰ͸࣮ࡍ͸ϒϥοΫϘοΫε͚ͩͲ)
    xray-sdk-go Λ૊ΈࠐΈ
    // sql.Open -> xray.SQL
    db, err := xray.SQL("mysql", dsn)
    // http.Handler Λ xray.Handler Ͱ wrap
    return xray.Handler(
    xray.NewFixedSegmentNamer("Isubank"),
    myHandler,
    )
    

    View Slide

56. Perl ଆͷ HTTPϦΫΤετૹ৴΋ capture
    ݺͼग़͠ݩΛ఻͑ΔͨΊʹ X-Amzn-Trace-ID ϔομΛ෇༩
    capture "externalHTTP", sub {
    my $segment = shift;
    $res = $self->client->post(
    $self->endpoint . $p,
    [
    "Content-Type" => "application/json",
    "Authorization" => "Bearer " . $self->app_id,
    "X-Amzn-Trace-ID" => $segment->trace_header,
    ],
    $body,
    );
    };
    

    View Slide

57. Isucoin(ΞϓϦ)→Isubank(API)
    

    View Slide

58. View Slide

59. View Slide

60. ֎෦APIݺͼग़͠ͷதͰߦΘΕͨSQLΫΤϦ΋ಉ͡ը໘Ͱ
    

    View Slide

61. ʰ෼ࢄʱτϨʔγϯάʂ
    

    View Slide

62. ϓϩηεΛ·͕ͨͬͨ௥੻ͷ࢓૊Έ
    Isucoin(Perl) ϦΫΤετ։࢝࣌ʹ X-Amzn-Trace-ID Λൃߦ
    1-5c4902d7-e8fe620cab66e01e57df8a12
    1-[timestamp]-[unique_id]
    sprintf(
    "1-%x-%s",
    CORE::time(),
    unpack("H*", Crypt::URandom::urandom(12)),
    );
    

    View Slide

63. ֎෦ϦΫΤετݺͼग़͠ʹϔομΛ෇༩
    use AWS::XRay qw/capture/;
    capture "externalHTTP", sub { # Ͳ͜Ͱ΋ݺ΂Δ(਌͕͍ͳ͚Ε͹RootʹͳΔ)
    my $segment = shift;
    $furl->POST(
    "https://example.com",
    [ "X-Amzn-Trace-ID" => $segment->trace_header ],
    $body,
    );
    };
    POST / HTTP/1.1
    Host: example.com
    X-Amzn-Trace-ID: Root=1-5c4902d7-e8fe620cab66e01e57df8a12;Parent=851cf9ec07b528c2
    ...
    

    View Slide

64. ϔομΛड৴ͨ͠API(Isubank)͕Ҿ͖ܧ͙
    Root=1-5c4902d7-e8fe620cab66e01e57df8a12;Parent=851cf9ec07b528c2
    # ࣮ࡍ͸Go͚ͩͲઆ໌ͷͨΊʹPerl
    AWS::XRay::Segment->new(
    id => new_id(),
    trace_id => "1-5c4902d7-e8fe620cab66e01e57df8a12",
    parent_id => "851cf9ec07b528c2",
    );
    ͦͷτϨʔεΛ X-Ray daemon ΁ૹ৴
    → X-Ray consoleͰτϨʔε͕࿈݁ͯ͠දࣔ͞ΕΔ
    

    View Slide

65. AWS::XRay ͷ಺෦࣮૷
    

    View Slide

66. τϨʔε͍ͨ͠ͱ͜ΖͰʮݱࡏͷʯRoot΍Parent͕ඞཁ
    package App;
    sub main {
    capture "App::main", sub { # Root
    my $segment = shift;
    Model::call($param); # ֎෦ݺͼग़͠Λ൐͏ॲཧ
    };
    };
    package Model;
    sub call() {
    my $param = shift;
    # ࣮ࡍʹ֎ΛݺͿͷ͸͕ͩ͜͜ݺͼग़͠ݩͷsegmentΛ஌Βͳ͍ͱRoot͕Θ͔Βͳ͍
    $furl->post($url, [
    "X-Amzn-Trace-ID" => "????",
    ], $param);
    }
    call($param)Λcall($segment,$param)ʹॻ͖׵͑ͨ͘ͳ͍
    

    View Slide

67. AWS::XRay Ͱ͸Ҿ਺Λ౉͞ͳ͍͍ͯ͘Α͏ʹͳͬͯΔ
    package App;
    sub main {
    capture "App::main", sub { # Root
    my $segment = shift;
    Model::call($param); # ֎෦ݺͼग़͠Λ൐͏ॲཧ
    };
    };
    package Model;
    sub call() {
    my $param = shift;
    # ਌ͷ͜ͱ͸஌Βͳ͍͚Ͳͱʹ͔͘ capture ͢Ε͹Α͍
    capture "Model::call", sub {
    my $segment = shift; # ਌͔Βݺ͹Ε͍ͯΕ͹ࣗಈతʹࢠʹͳΔ
    $furl->post($url, [
    "X-Amzn-Trace-ID" => $segment->trace_header,
    ], $param);
    };
    }
    

    View Slide

68. Ͳ͏΍࣮ͬͯ૷͞Ε͍ͯΔ͔
    Perl ͷຐ๏ͷͻͱͭ local
    perldoc -f local
    ͋ͳ͕ͨຊ౰ʹ๬ΜͰ͍Δͷ͸ my ͷํͰ͠ΐ͏; local ͸΄ͱΜ
    Ͳͷਓʑ͕ʮϩʔΧϧʯͱߟ͑Δ΋ͷͱ ҧ͏͔ΒͰ͢ɻ
    "local" ͸ϦετΞοϓ͞Εͨม਺Λɺғ͍ͬͯΔϒϩοΫɺ
    ϑΝΠϧɺeval ͷதͰɺϩʔΧϧͳ΋ͷʹ͠·͢ɻ
    !!!!!!!
    

    View Slide

69. local → μΠφϛοΫείʔϓ (ݺͼग़͠ݩͰείʔϓ͕ܾ·Δ)
    my → ϨΩγΧϧείʔϓ (ιʔείʔυͷࣈ໘Ͱείʔϓ͕ܾ·Δ)
    package main;
    our $Foo = "A"; # ύοέʔδม਺
    sub foo {
    print $Foo;
    }
    foo(); #= A
    {
    local $Foo = "B"; # ͜ͷϒϩοΫ͔Βݺ͹Ε͍ͯΔؒ͸B
    foo(); #= B
    }
    foo(); #= A ϒϩοΫΛൈ͚Δͱࣗಈతʹ໭Δ
    

    View Slide

70. Կճωετͯ͠ݺͼग़ͯ͠΋ॱ൪ʹ্ॻ͖ͯ͠ɺ΋ͲΕ͹෮ݩ
    foo(); #= A
    {
    local $Foo = "B";
    foo(); #= B
    {
    local $Foo = "C";
    foo(); #= C
    {
    local $Foo = "D";
    foo() #= D
    }
    foo(); #= C
    }
    foo(); #= B
    }
    foo(); #= A
    

    View Slide

71. AWS::XRay Ͱ͸۩ମతʹ͸͜Μͳײ͡
    ݱࡏͷ Trace-ID, Segment-ID Λ local Ͱอ࣋
    package AWS::XRay;
    our $TRACE_ID; # Root
    our $SEGMENT_ID; # ݱࡏͷSegmentID
    sub capture {
    my ($name, $code) = @_;
    # $TRACE_ID͕ະఆٛͳΒRootͳͷͰ৽نIDൃߦ local ʹ
    local $TRACE_ID = $TRACE_ID // new_trace_id();
    # ৽͍͠ηάϝϯτΛ࡞Δ
    # $SEGMENT_ID ͕͋Ε͹ parent_id => $SEGMENT_ID ʹͳ͍ͬͯΔ
    my $segment = AWS::XRay::Segment->new({
    name => $name,
    type => $SEGMENT_ID ? "subsegment" : undef,
    parent_id => $SEGMENT_ID,
    });
    

    View Slide

72. local $SEGMENT_ID = $segment->{id}; # ৽͍͠ηάϝϯτIDΛ্ॻ͖ͯ͠localʹଋറ
    # ίʔυΛ࣮ߦ
    # ͜ͷதͰ͸ $SEGMENT_ID ͕ઃఆ͞Ε͍ͯΔͷͰ
    # ͞Βʹ capture ͨ͠Β $segment ͷࢠڙʹͳΔ
    $ret = $code->($segment);
    $segment->close(); # ηάϝϯτσʔλૹ৴
    return $ret;
    # $segment ͸ফ͑Δ(my͔ͩΒ)
    # $SEGMENT_ID ΋ݺͼग़࣌͠ͷ஋ʹ໭Δ
    # RootͳΒ $TRACE_ID ΋ফ͑Δ(ݩ͕ະఆ͔ٛͩΒ)
    }
    

    View Slide

73. Perl ʹ͓͚Δ local
    ͦͷϒϩοΫ(౳)͔Βݺͼग़͞Ε͍ͯΔ͚ؒͩม਺ͷ஋Λมߋ
    ྫ֎Ͱ΋ԿͰ΋ൈ͚ͨΒݩʹ໭Δ
    Կஈ֊Ͱ΋ωετͰ͖Δ
    ݴޠͰαϙʔτ͞Ε͍ͯΔͷͰʮ࣮֬ʹʯ໭Δ
    Perl ࠷ߴศརʂʂ
    

    View Slide

74. ༨ஊ
    Go Ͱ X-Ray SDK Λ࢖͏৔߹
    ϦΫΤετͰൃੜͨ͠ context.Context ΛͲ͜·Ͱ΋
    Ҿ਺ʹ౉͍͔ͯ͠ͳ͍ͱτϨʔεͰ͖ͳ͍
    ్தʹ ctx ΛҾ͖ܧ͛ͳ͍ϥΠϒϥϦ(ORMͱ͔)͕ڬ·Δͱ
    (Isubank ΋࣮ࡍ͸40ՕॴҎ্ ctx Λ౉͢Α͏ʹॻ͖׵͑ͨ)
    

    View Slide

75. !
    ίετͷ࿩
    

    View Slide

76. !
    ίετͷ࿩
    ʮX-Ray ͸Αͦ͞͏͚ͩͲɺ͓ߴ͍ΜͰ͠ΐ͏…?ʯ
    ௚઀ίετ = X-Ray ར༻ྉۚ
    ؒ઀ίετ = ΞϓϦέʔγϣϯͷύϑΥʔϚϯεྼԽ
    (݁Ռతʹ࢖༻Ϧιʔε͕૿͑ΔɺϢʔβମݧ͕ଛͳΘΕΔͳͲ)
    

    View Slide

77. ΞϓϦέʔγϣϯͷύϑΥʔϚϯεྼԽ
    τϨʔεΛ૊ΈࠐΉ͜ͱʹΑͬͯͲΕ͙Β͍஗͘ͳΔ͔
    • 1ॲཧͰ100ݸͷ capture (ίʔυ͸ۭ sub{}) Λੜ੒
    • αϯϓϦϯά͠ͳ͍ͰશͯͷτϨʔεΛૹ৴
    Λ 500/sec ఔ౓࣮ߦͰ͖Δ (macbook pro 13 2017)
    ͭ·Γ1ॲཧ͋ͨΓ2msఔ౓ͷΦʔόʔϔου
    τϨʔεΛૹ৴͠ͳ͍(captureΛωετ͢Δ͚ͩ)ͳΒ
    10,000/sec, 0.1msఔ౓ͷΦʔόʔϔου
    

    View Slide

78. ΞϓϦέʔγϣϯͷύϑΥʔϚϯεྼԽ
    ͍͍ͨͯͷWebΞϓϦέʔγϣϯʹରͯ͠͸໰୊ͳ͍଎౓
    ࣮ࡍʹ͸αϯϓϦϯάΛߦ͏ͷͰߋʹӨڹ͸ܰඍ
    αϯϓϦϯά1%Ͱຊ൪౤ೖͯ͠ύϑΥʔϚϯε΁ͷӨڹͳ͠
    ࣮ࡍͷܭଌίʔυ͸ xt/01_overhead.t10 ࢀর
    10 https://github.com/fujiwara/AWS-XRay/blob/master/xt/01_overhead.t
    

    View Slide

79. X-Ray ར༻ྉۚ
    ຖ݄ͷແྉ࿮
    τϨʔεͷه࿥͸ 10 ສճ·Ͱແྉ
    τϨʔεͷऔಘͱεΩϟϯ͸߹Θͤͯ 100 ສճ·Ͱແྉ
    ௥ՃͰ
    τϨʔεͷه࿥ 100 ສ݅͋ͨΓ 5 USD
    τϨʔεͷऔಘͱεΩϟϯΛ߹Θͤͯ 100 ສ݅͋ͨΓ 0.50 USD
    

    View Slide

80. ࣮ࡍͲΕ͙Βֻ͍͔Δͷʁ
    ฏۉ 100 req/sec ͷ৔߹
    100 * 86,400 * 30 / 1,000,000 * 5 USD = 1,296 USD
    ฏۉ 1,000 req/sec ͳΒ 12,960 USD / month
    શͯͷϦΫΤετʹ͍ͭͯه࿥͢Δͷ͸ίετ͕ߴ͍
    

    View Slide

81. αϯϓϦϯά͢Δ
    ద౰ʹαϯϓϦϯά͢Ε͹܏޲͸௫ΊΔ
    ͋·Γൃੜ͠ͳ͍Τϥʔɺslowdown ࣌ͷௐࠪʹ͸࢖͍ͮΒ͍
    

    View Slide

82. αϯϓϦϯάͷ޻෉
    ྫɿ1ඵ͝ͱʹ࠷ॳͷϦΫΤετͱ ௥ՃϦΫΤετͷ5%Λه࿥
    (ެࣜSDKͷσϑΥϧτઃఆ)
    100 req/sec ͷ৔߹ 1 + 100 * 5% = 6 req/sec ෼͕ه࿥͞ΕΔ
    ϦΫΤετ͕গͳͯ͘΋ 1 req/sec ͸࣮֬ʹه࿥
    ެࣜ SDK ͸ X-Ray console ͔ΒαϯϓϦϯάϧʔϧΛઃఆՄ
    (AWS::XRay Ͱ͸ະରԠ…)
    

    View Slide

83. X-Ray console ͰͷαϯϓϦϯάઃఆ
    

    View Slide

84. Plack::Middleware::XRay ͰͷαϯϓϦϯάઃఆ
    ݻఆϨʔτ
    enable "XRay"
    name => "myApp",
    sampling_rate => 0.05; # 5%
    

    View Slide

85. αϯϓϦϯάϧʔϧΛίʔυͰهड़
    PSGI $env Λड͚औͬͯਅِΛฦ͢ίʔυΛࢦఆͰ͖Δ
    enable "XRay"
    name => "myApp",
    sampler => sub {
    my $env = shift;
    state %paths;
    if ( $paths{$env->{PATH_INFO}++ == 0 ) {
    # ಛఆͷPATHΛॳΊͯड৴ͨ͠৔߹͸ඞͣه࿥
    return 1;
    }
    rand() < 0.05; # ͦΕҎ֎͸ 5%
    };
    

    View Slide

86. Τϥʔ΍஗͍ϦΫΤετ͸ඞͣه࿥͍ͨ͠
    5% αϯϓϦϯάˠ ΤϥʔΛه࿥Ͱ͖Δ֬཰΋5%
    શϦΫΤετͰϝϞϦ্ʹτϨʔεΛอଘ͓ͯ͘͠
    ϨεϙϯεΛฦ࣌͢఺ͰτϨʔεΛૹ৴͢Δ͔Ͳ͏͔ΛܾΊΔ
    enable "XRay"
    name => "myApp",
    response_filter => sub {
    my ($env, $res, $elapsed) = @_;
    # 5xx OR 1ඵҎ্ֻ͔ͬͨ৔߹ʹૹ৴
    return $res->[0] >= 500 || $elapsed >= 1;
    };
    

    View Slide

87. ·ͱΊ
    

    View Slide

88. ·ͱΊ
    ෼ࢄτϨʔγϯά͸ύϑΥʔϚϯεɺো֐ղੳʹେม༗༻
    ର ISUCON ༻ܾઓฌثͱͯ͠΋༗༻
    AWS X-Ray ͸ϚωʔδυͰಋೖ͕༰қ
    αϯϓϦϯάઃఆʹ͸ཁ஫ҙ
    !
    Perl SDK ͸ͳ͔ͬͨͷͰ AWS::XRay Λ࡞ͬͨ
    पลϞδϡʔϧؚΊͯ΋਺ඦߦͰ͍͍͔Μ͡ʹ
    

    View Slide