クックパッドのログをいい感じにしているアーキテクチャ / Logging architecture at Cookpad

Cookpad Tech Kitchen #9 https://cookpad.connpass.com/event/60831/

Transcript

1. ΫοΫύουͷϩάΛ ͍͍ײ͡ʹ͍ͯ͠Δ ΞʔΩςΫνϟ 2017/07/26 Cookpad Tech Kitchen #9 Hokuto Hoshi hokuto@cookpad.com
2. ੕ ๺ే (΄͠ ΄͘ͱ) / @kani_b • ΫοΫύουגࣜձࣾ
 ΠϯϑϥετϥΫνϟʔ෦௕ • SRE • AWS ͱηΩϡϦςΟ͕ϝΠϯྖҬ • ࣾ಺֎ͷ৘ใηΩϡϦςΟશൠ΋୲౰
3. ࠓ೔ͷςʔϚ: ϩά
4. https://speakerdeck.com/kanny/miao-jian-shu-mo-falseroguwoiigan-zinisuruakitekutiya
5. ͳͥɺਓ͸ϩάΛूΊΔͷ͔
6. “ى͖ͨ͜ͱ” ͸ϩά͚͕ͩࣔ͢ • αʔϏεվળͷͨΊʹ͸ඞཁෆՄܽ • Ծઆͷݕূ • ύϑΥʔϚϯεվળ • γεςϜτϥϒϧͷղফ • etc…
7. ऩू͍ͯ͠Δϩά • PV ϩά • ϖʔδӾཡ͝ͱʹه࿥͞ΕΔ • Ӿཡ਺ɺݕࡧಈ޲ͷ෼ੳͳͲ • ޿ࠂܥϩά • ΠϯϓϨογϣϯɺΫϦοΫɺ
 ίϯόʔδϣϯ • ഑৴੍ޚ΍ޮՌଌఆʹ • ΞΫςΟϏςΟϩά • Ϣʔβߦಈ (ը໘ભҠͳͲ) • ෼ੳɺࢪࡦݕূɺ
 ϞόΠϧΞϓϦͷಈ࡞ϩά • ؂ࠪϩά • Ϣʔβσʔλͷ࡞੒ɾߋ৽ɾ
 ࡟আɺϩάΠϯϩάͳͲ • αϙʔτɺηΩϡϦςΟ؂ࠪ
8. ऩू͍ͯ͠Δϩά • KPI ༻ϩά • “ग़དྷࣄ” Λه࿥͢Δϩά • UU, DU ͳͲ΋ࣗಈܭࢉɺ
 KPI ؅ཧ༻ • γεςϜϩά • OSɺϛυϧ΢ΣΞͷϩά • γεςϜ؅ཧ༻ • etc…
9. ݱࡏͷن໛ • Fluentd ʹྲྀΕ͍ͯΔϩά (֓ࢉ) • σʔλ૯ྔ: 400~600GB / ೔ • Ϩίʔυ਺: 8ԯϨίʔυҎ্ / ೔ • ඵؒ8,000 ~ 25,000 Ϩίʔυ͘Β͍ ※ࡢ೥ͷൃද͔࣌ΒϨίʔυͷ࡟ݮΛ͍ͯ͠·͢
10. ཁ͸͍Ζ͍Ζͳϩά͕
 ͍ͬͺ͍͋Δ
11. ϩάʹٻΊΔཁ݅ • ࣮֬ʹ഑ૹ͞ΕΔ͜ͱ • Ͱ͖Δ͚ͩ஗Ԇ͕গͳ͍͜ͱ • ྔΛݮΒ͢ (αϯϓϦϯά) ඞཁ͕ͳ͍͜ͱ • ෼ੳ͠΍͍͢ܗͰอ࣋Ͱ͖Δ͜ͱ
12. ϩάͷҰੜ • ͋ͭΊͯ • ॲཧͯ͠ • ͔ͭ͏ ͜ͷ΁Μͷ࿩Λ͠·͢
13. ΫοΫύουͷ
 ϩάऩू (αʔϏεͰར༻͍ͯ͠Δ΋ͷ)
14. ,JOFTJT
    4USFBNT 4 -BNCEB
    GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU
    $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ
    4FSWFST "QQ
    4FSWFST "QQ
    4FSWFST -PH
 "HHSFHBUPS
15. -PH
 "HHSFHBUPS ,JOFTJT
    4USFBNT "QQ
    4FSWFST 4 -BNCEB
    GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU
    $MVTUFS "QQ
    4FSWFST "QQ
    4FSWFST "UIFOB &MBTUJDTFBSDI ߏ੒
16. ΞϓϦέʔγϣϯαʔό͔Β • ΈΜͳେ޷͖ fluentd • ਖ਼֬ʹ͸ td-agent • Ͱ͖ΔݶΓసૹઌ͸গͳ͘ • ू໿ϊʔυ or Kinesis Streams
17. td-agent ू໿ϊʔυ • ॾࣄ৘ʹΑΓ·ͩ 0.12 • શϩά͕ूத͢ΔͨΊߴෛՙ • multiprocess plugin Ͱ͏·͍͜ͱ΍͍ͬͯΔ
18. ,JOFTJT
    4USFBNT 4 -BNCEB
    GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU
    $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ
    4FSWFST "QQ
    4FSWFST "QQ
    4FSWFST -PH
 "HHSFHBUPS
19. Redshift • ࣾ಺ͷσʔλ෼ੳͷத৺ (DWH) • ϩά͚ͩͰͳࣾ͘಺ͷશ DB ͷσʔλΛ
 औΓࠐΜͰ͓Γ౷߹෼ੳ͕Մೳ • શࣾһ͕ར༻͢Δ (Tableau ΍ Redash) • S3 ͷϩάσʔλΛόονδϣϒͰऔΓࠐΈ • ϩʔυִؒ͸௕Ί
20. ,JOFTJT
    4USFBNT 4 -BNCEB
    GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU
    $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ
    4FSWFST "QQ
    4FSWFST "QQ
    4FSWFST -PH
 "HHSFHBUPS
21. Lambda + DynamoDB • ετϦʔϜूܭʹར༻ (ओʹ޿ࠂ) • Redshift ΁ͷΠϯϙʔτִؒΛ௒͑Δ༻్ • Kinesis Streams Ͱड͚ Lambda Ͱू໿ɺ DynamoDB ʹॻ͘ • ྲྀྔ͕େ͖͍ͨΊू໿ϊʔυΛ࢖Θͳ͍ • ϩάͦͷ΋ͷ͸ S3 ʹ΋ॏෳૹ৴ • ௕ظతͳ෼ੳ͸ Redshift Ͱߦ͏
22. ,JOFTJT
    4USFBNT 4 -BNCEB
    GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU
    $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ
    4FSWFST "QQ
    4FSWFST "QQ
    4FSWFST -PH
 "HHSFHBUPS
23. Elasticsearch • ूܭ΍μογϡϘʔυʹར༻ • αʔϏεʹ͸ར༻͞Εͳ͍ • ߴ͍ྲྀྔͷϩάʹ͸࢖Θͳ͍ or αϯϓϦϯά • Amazon Elasticsearch Service Λར༻
24. ͦͷଞ • getsentry/sentry • Τϥʔϩάऩूπʔϧ • Rails ΞϓϦέʔγϣϯͷΤϥʔΛૹ৴ • Slack • Ϣʔβ͔Βͷ͝ҙݟͳͲΛૹ৴
25. ,JOFTJT
    4USFBNT 4 -BNCEB
    GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU
    $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ
    4FSWFST "QQ
    4FSWFST "QQ
    4FSWFST -PH
 "HHSFHBUPS
26. Athena • ݕূத • Redshift ͷϩʔυִؒΛ଴ͯͳ͍͕
 SQL Ͱ෼ੳ͍ͨ͠έʔε ʹ࢖͏૝ఆ • Redshift Spectrum ͱ߹Θͤͯߟ͑த
27. -PH
 "HHSFHBUPS ,JOFTJT
    4USFBNT "QQ
    4FSWFST 4 -BNCEB
    GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU
    $MVTUFS "QQ
    4FSWFST "QQ
    4FSWFST "UIFOB &MBTUJDTFBSDI ߏ੒
28. શମ૾ • σʔλετΞ΍πʔϧʹΑΓಘҙෆಘҙ͕͋Δ • ΞϓϦ͔Β৮Δͱྑ͍΋ͷɺ
 ਓ͕ؒ෼ੳ͠΍͍͢΋ͷ (SQL Ͱ৮ΕΔͳͲ) • සൟͳॻ͖ࠐΈ͕ଟ͍΋ͷɺͦ͏Ͱͳ͍΋ͷ • αʔϏεϨϕϧ΍ڐ༰Ͱ͖Δ஗Ԇɺ༻్ͰબͿ
29. ,JOFTJT
    4USFBNT 4 -BNCEB
    GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU
    $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ
    4FSWFST "QQ
    4FSWFST "QQ
    4FSWFST -PH
 "HHSFHBUPS
30. ,JOFTJT
    4USFBNT 4 -BNCEB
    GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU
    $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ
    4FSWFST "QQ
    4FSWFST "QQ
    4FSWFST -PH
 "HHSFHBUPS Ͱ͖Δ͚ͩ୹͍஗ԆͰ
    ΞϓϦ͔Βͷར༻ϝΠϯ ͋Δఔ౓ͷ஗Ԇ͸ڐ༰Ͱ͖Δ
    ΞϓϦ͔Β΋ਓؒ΋ར༻ αʔϏεӡ༻্
    ΫϦςΟΧϧͰͳ͍ϩάʹར༻
31. ΫοΫύουͷ
 ϩάऩू (ͦͷଞ)
32. ΞϓϦέʔγϣϯͷಈ࡞ϩά • td-agent Ͱऩू • CloudWatch Logs ʹૹ৴ • fluent-plugin-cloudwatch-logs • Docker Խ͞Ε͍ͯΔΞϓϦ͸ඪ४૷උ • Τϥʔϩά͸ Sentry ʹૹ৴
33. γεςϜϩά • ݱࡏ͸2ͭΛฒߦՔಈ • rsyslog • ;ͭ͏ͷϩάαʔό • td-agent Ͱऩूͯ͠ Kinesis Streams -> Lambda Ͱ Graylog ΁ • 2.3 ͔Β Elasticsearch Service ͕࢖͑ΔΑ͏ʹ
34. ·ͱΊ • ΫοΫύουͷϩάऩूج൫ʹ͍ͭͯ
 ͓࿩͠·ͨ͠ • த৺ʹͳ͍ͬͯΔͷ͸ S3 ͱ Redshift • αʔϏεϨϕϧɺ஗Ԇɺ༻్ʹΑͬͯ
 ద੾ͳ΋ͷΛબ୒͍ͯ͘͠