クックパッドのログをいい感じにしているアーキテクチャ / Logging architecture at Cookpad

Cookpad Tech Kitchen #9 https://cookpad.connpass.com/event/60831/

Transcript

1. ΫοΫύουͷϩάΛ
   ͍͍ײ͡ʹ͍ͯ͠Δ
   ΞʔΩςΫνϟ
   2017/07/26 Cookpad Tech Kitchen #9
   Hokuto Hoshi
   hokuto@cookpad.com
   

   View Slide

2. ੕ ๺ే (΄͠ ΄͘ͱ) / @kani_b
   • ΫοΫύουגࣜձࣾ

   ΠϯϑϥετϥΫνϟʔ෦௕
   • SRE
   • AWS ͱηΩϡϦςΟ͕ϝΠϯྖҬ
   • ࣾ಺֎ͷ৘ใηΩϡϦςΟશൠ΋୲౰
   

   View Slide

3. ࠓ೔ͷςʔϚ: ϩά
   

   View Slide

4. https://speakerdeck.com/kanny/miao-jian-shu-mo-falseroguwoiigan-zinisuruakitekutiya
   

   View Slide

5. ͳͥɺਓ͸ϩάΛूΊΔͷ͔
   

   View Slide

6. “ى͖ͨ͜ͱ” ͸ϩά͚͕ͩࣔ͢
   • αʔϏεվળͷͨΊʹ͸ඞཁෆՄܽ
   • Ծઆͷݕূ
   • ύϑΥʔϚϯεվળ
   • γεςϜτϥϒϧͷղফ
   • etc…
   

   View Slide

7. ऩू͍ͯ͠Δϩά
   • PV ϩά
   • ϖʔδӾཡ͝ͱʹه࿥͞ΕΔ
   • Ӿཡ਺ɺݕࡧಈ޲ͷ෼ੳͳͲ
   • ޿ࠂܥϩά
   • ΠϯϓϨογϣϯɺΫϦοΫɺ

   ίϯόʔδϣϯ
   • ഑৴੍ޚ΍ޮՌଌఆʹ
   • ΞΫςΟϏςΟϩά
   • Ϣʔβߦಈ (ը໘ભҠͳͲ)
   • ෼ੳɺࢪࡦݕূɺ

   ϞόΠϧΞϓϦͷಈ࡞ϩά
   • ؂ࠪϩά
   • Ϣʔβσʔλͷ࡞੒ɾߋ৽ɾ

   ࡟আɺϩάΠϯϩάͳͲ
   • αϙʔτɺηΩϡϦςΟ؂ࠪ
   

   View Slide

8. ऩू͍ͯ͠Δϩά
   • KPI ༻ϩά
   • “ग़དྷࣄ” Λه࿥͢Δϩά
   • UU, DU ͳͲ΋ࣗಈܭࢉɺ

   KPI ؅ཧ༻
   • γεςϜϩά
   • OSɺϛυϧ΢ΣΞͷϩά
   • γεςϜ؅ཧ༻
   • etc…
   

   View Slide

9. ݱࡏͷن໛
   • Fluentd ʹྲྀΕ͍ͯΔϩά (֓ࢉ)
   • σʔλ૯ྔ: 400~600GB / ೔
   • Ϩίʔυ਺: 8ԯϨίʔυҎ্ / ೔
   • ඵؒ8,000 ~ 25,000 Ϩίʔυ͘Β͍
   ※ࡢ೥ͷൃද͔࣌ΒϨίʔυͷ࡟ݮΛ͍ͯ͠·͢
   

   View Slide

10. ཁ͸͍Ζ͍Ζͳϩά͕

    ͍ͬͺ͍͋Δ
    

    View Slide

11. ϩάʹٻΊΔཁ݅
    • ࣮֬ʹ഑ૹ͞ΕΔ͜ͱ
    • Ͱ͖Δ͚ͩ஗Ԇ͕গͳ͍͜ͱ
    • ྔΛݮΒ͢ (αϯϓϦϯά) ඞཁ͕ͳ͍͜ͱ
    • ෼ੳ͠΍͍͢ܗͰอ࣋Ͱ͖Δ͜ͱ
    

    View Slide

12. ϩάͷҰੜ
    • ͋ͭΊͯ
    • ॲཧͯ͠
    • ͔ͭ͏
    ͜ͷ΁Μͷ࿩Λ͠·͢
    

    View Slide

13. ΫοΫύουͷ

    ϩάऩू
    (αʔϏεͰར༻͍ͯ͠Δ΋ͷ)
    

    View Slide

14. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS
    

    View Slide

15. -PH

    "HHSFHBUPS
    ,JOFTJT
    4USFBNT
    "QQ4FSWFST
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "QQ4FSWFST
    "QQ4FSWFST
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    

    View Slide

16. ΞϓϦέʔγϣϯαʔό͔Β
    • ΈΜͳେ޷͖ fluentd
    • ਖ਼֬ʹ͸ td-agent
    • Ͱ͖ΔݶΓసૹઌ͸গͳ͘
    • ू໿ϊʔυ or Kinesis Streams
    

    View Slide

17. td-agent ू໿ϊʔυ
    • ॾࣄ৘ʹΑΓ·ͩ 0.12
    • શϩά͕ूத͢ΔͨΊߴෛՙ
    • multiprocess plugin Ͱ͏·͍͜ͱ΍͍ͬͯΔ
    

    View Slide

18. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS
    

    View Slide

19. Redshift
    • ࣾ಺ͷσʔλ෼ੳͷத৺ (DWH)
    • ϩά͚ͩͰͳࣾ͘಺ͷશ DB ͷσʔλΛ

    औΓࠐΜͰ͓Γ౷߹෼ੳ͕Մೳ
    • શࣾһ͕ར༻͢Δ (Tableau ΍ Redash)
    • S3 ͷϩάσʔλΛόονδϣϒͰऔΓࠐΈ
    • ϩʔυִؒ͸௕Ί
    

    View Slide

20. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS
    

    View Slide

21. Lambda + DynamoDB
    • ετϦʔϜूܭʹར༻ (ओʹ޿ࠂ)
    • Redshift ΁ͷΠϯϙʔτִؒΛ௒͑Δ༻్
    • Kinesis Streams Ͱड͚ Lambda Ͱू໿ɺ
    DynamoDB ʹॻ͘
    • ྲྀྔ͕େ͖͍ͨΊू໿ϊʔυΛ࢖Θͳ͍
    • ϩάͦͷ΋ͷ͸ S3 ʹ΋ॏෳૹ৴
    • ௕ظతͳ෼ੳ͸ Redshift Ͱߦ͏
    

    View Slide

22. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS
    

    View Slide

23. Elasticsearch
    • ूܭ΍μογϡϘʔυʹར༻
    • αʔϏεʹ͸ར༻͞Εͳ͍
    • ߴ͍ྲྀྔͷϩάʹ͸࢖Θͳ͍ or αϯϓϦϯά
    • Amazon Elasticsearch Service Λར༻
    

    View Slide

24. ͦͷଞ
    • getsentry/sentry
    • Τϥʔϩάऩूπʔϧ
    • Rails ΞϓϦέʔγϣϯͷΤϥʔΛૹ৴
    • Slack
    • Ϣʔβ͔Βͷ͝ҙݟͳͲΛૹ৴
    

    View Slide

25. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS
    

    View Slide

26. Athena
    • ݕূத
    • Redshift ͷϩʔυִؒΛ଴ͯͳ͍͕

    SQL Ͱ෼ੳ͍ͨ͠έʔε ʹ࢖͏૝ఆ
    • Redshift Spectrum ͱ߹Θͤͯߟ͑த
    

    View Slide

27. -PH

    "HHSFHBUPS
    ,JOFTJT
    4USFBNT
    "QQ4FSWFST
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "QQ4FSWFST
    "QQ4FSWFST
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    

    View Slide

28. શମ૾
    • σʔλετΞ΍πʔϧʹΑΓಘҙෆಘҙ͕͋Δ
    • ΞϓϦ͔Β৮Δͱྑ͍΋ͷɺ

    ਓ͕ؒ෼ੳ͠΍͍͢΋ͷ (SQL Ͱ৮ΕΔͳͲ)
    • සൟͳॻ͖ࠐΈ͕ଟ͍΋ͷɺͦ͏Ͱͳ͍΋ͷ
    • αʔϏεϨϕϧ΍ڐ༰Ͱ͖Δ஗Ԇɺ༻్ͰબͿ
    

    View Slide

29. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS
    

    View Slide

30. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS
    Ͱ͖Δ͚ͩ୹͍஗ԆͰ
    ΞϓϦ͔Βͷར༻ϝΠϯ
    ͋Δఔ౓ͷ஗Ԇ͸ڐ༰Ͱ͖Δ
    ΞϓϦ͔Β΋ਓؒ΋ར༻
    αʔϏεӡ༻্
    ΫϦςΟΧϧͰͳ͍ϩάʹར༻
    

    View Slide

31. ΫοΫύουͷ

    ϩάऩू
    (ͦͷଞ)
    

    View Slide

32. ΞϓϦέʔγϣϯͷಈ࡞ϩά
    • td-agent Ͱऩू
    • CloudWatch Logs ʹૹ৴
    • fluent-plugin-cloudwatch-logs
    • Docker Խ͞Ε͍ͯΔΞϓϦ͸ඪ४૷උ
    • Τϥʔϩά͸ Sentry ʹૹ৴
    

    View Slide

33. γεςϜϩά
    • ݱࡏ͸2ͭΛฒߦՔಈ
    • rsyslog
    • ;ͭ͏ͷϩάαʔό
    • td-agent Ͱऩूͯ͠ Kinesis Streams -> Lambda Ͱ
    Graylog ΁
    • 2.3 ͔Β Elasticsearch Service ͕࢖͑ΔΑ͏ʹ
    

    View Slide

34. ·ͱΊ
    • ΫοΫύουͷϩάऩूج൫ʹ͍ͭͯ

    ͓࿩͠·ͨ͠
    • த৺ʹͳ͍ͬͯΔͷ͸ S3 ͱ Redshift
    • αʔϏεϨϕϧɺ஗Ԇɺ༻్ʹΑͬͯ

    ద੾ͳ΋ͷΛબ୒͍ͯ͘͠
    

    View Slide