クックパッドのログをいい感じにしているアーキテクチャ / Logging architecture at Cookpad

14a602891dce5c68facca9de28340522?s=47 Hokuto Hoshi
July 26, 2017
Technology
22
13k

クックパッドのログをいい感じにしているアーキテクチャ / Logging architecture at Cookpad

Cookpad Tech Kitchen #9 https://cookpad.connpass.com/event/60831/

14a602891dce5c68facca9de28340522?s=128

Hokuto Hoshi

July 26, 2017
Tweet

Want more?

14a602891dce5c68facca9de28340522?s=48 kanny
7
1.9k
14a602891dce5c68facca9de28340522?s=48 kanny
2
3.2k
14a602891dce5c68facca9de28340522?s=48 kanny
1
3.5k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
10
530
A9a491b0fcbe0fbce3d64063a37add99?s=48 rmw
2
270
766b9746b59e6f09e280cd33cf4ed419?s=48 skipperchong
0
170
61857dafbd287b3027c4dcea9008ad3c?s=48 carmenhchung
6
340
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
6
220
C0390e8b11079f8761c0cd3274ed61cb?s=48 productmarketing
3
300
C35e01544a0dd94a2cf1619ee8a42ebb?s=48 clairelew
4
570
6bb82b13cda86d3b821fa44100335ddf?s=48 thoeni
1
170
B3ace07412a5178ea778e7eec161fc0a?s=48 jcasabona
4
250

Transcript

  1. 1.

    ΫοΫύουͷϩάΛ ͍͍ײ͡ʹ͍ͯ͠Δ ΞʔΩςΫνϟ 2017/07/26 Cookpad Tech Kitchen #9 Hokuto Hoshi

    hokuto@cookpad.com
  2. 2.

    ੕ ๺ే (΄͠ ΄͘ͱ) / @kani_b • ΫοΫύουגࣜձࣾ
 ΠϯϑϥετϥΫνϟʔ෦௕ •

    SRE • AWS ͱηΩϡϦςΟ͕ϝΠϯྖҬ • ࣾ಺֎ͷ৘ใηΩϡϦςΟશൠ΋୲౰
  3. 3.

    ࠓ೔ͷςʔϚ: ϩά

  4. 4.

    https://speakerdeck.com/kanny/miao-jian-shu-mo-falseroguwoiigan-zinisuruakitekutiya

  5. 5.

    ͳͥɺਓ͸ϩάΛूΊΔͷ͔

  6. 6.

    “ى͖ͨ͜ͱ” ͸ϩά͚͕ͩࣔ͢ • αʔϏεվળͷͨΊʹ͸ඞཁෆՄܽ • Ծઆͷݕূ • ύϑΥʔϚϯεվળ • γεςϜτϥϒϧͷղফ

    • etc…
  7. 7.

    ऩू͍ͯ͠Δϩά • PV ϩά • ϖʔδӾཡ͝ͱʹه࿥͞ΕΔ • Ӿཡ਺ɺݕࡧಈ޲ͷ෼ੳͳͲ • ޿ࠂܥϩά

    • ΠϯϓϨογϣϯɺΫϦοΫɺ
 ίϯόʔδϣϯ • ഑৴੍ޚ΍ޮՌଌఆʹ • ΞΫςΟϏςΟϩά • Ϣʔβߦಈ (ը໘ભҠͳͲ) • ෼ੳɺࢪࡦݕূɺ
 ϞόΠϧΞϓϦͷಈ࡞ϩά • ؂ࠪϩά • Ϣʔβσʔλͷ࡞੒ɾߋ৽ɾ
 ࡟আɺϩάΠϯϩάͳͲ • αϙʔτɺηΩϡϦςΟ؂ࠪ
  8. 8.

    ऩू͍ͯ͠Δϩά • KPI ༻ϩά • “ग़དྷࣄ” Λه࿥͢Δϩά • UU, DU

    ͳͲ΋ࣗಈܭࢉɺ
 KPI ؅ཧ༻ • γεςϜϩά • OSɺϛυϧ΢ΣΞͷϩά • γεςϜ؅ཧ༻ • etc…
  9. 9.

    ݱࡏͷن໛ • Fluentd ʹྲྀΕ͍ͯΔϩά (֓ࢉ) • σʔλ૯ྔ: 400~600GB / ೔

    • Ϩίʔυ਺: 8ԯϨίʔυҎ্ / ೔ • ඵؒ8,000 ~ 25,000 Ϩίʔυ͘Β͍ ※ࡢ೥ͷൃද͔࣌ΒϨίʔυͷ࡟ݮΛ͍ͯ͠·͢
  10. 10.

    ཁ͸͍Ζ͍Ζͳϩά͕
 ͍ͬͺ͍͋Δ

  11. 11.

    ϩάʹٻΊΔཁ݅ • ࣮֬ʹ഑ૹ͞ΕΔ͜ͱ • Ͱ͖Δ͚ͩ஗Ԇ͕গͳ͍͜ͱ • ྔΛݮΒ͢ (αϯϓϦϯά) ඞཁ͕ͳ͍͜ͱ •

    ෼ੳ͠΍͍͢ܗͰอ࣋Ͱ͖Δ͜ͱ
  12. 12.

    ϩάͷҰੜ • ͋ͭΊͯ • ॲཧͯ͠ • ͔ͭ͏ ͜ͷ΁Μͷ࿩Λ͠·͢

  13. 13.

    ΫοΫύουͷ
 ϩάऩू (αʔϏεͰར༻͍ͯ͠Δ΋ͷ)

  14. 14.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS
  15. 15.

    -PH
 "HHSFHBUPS ,JOFTJT 4USFBNT "QQ4FSWFST 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI


    4FSWFS 3FETIJGU $MVTUFS "QQ4FSWFST "QQ4FSWFST "UIFOB &MBTUJDTFBSDI ߏ੒
  16. 16.

    ΞϓϦέʔγϣϯαʔό͔Β • ΈΜͳେ޷͖ fluentd • ਖ਼֬ʹ͸ td-agent • Ͱ͖ΔݶΓసૹઌ͸গͳ͘ •

    ू໿ϊʔυ or Kinesis Streams
  17. 17.

    td-agent ू໿ϊʔυ • ॾࣄ৘ʹΑΓ·ͩ 0.12 • શϩά͕ूத͢ΔͨΊߴෛՙ • multiprocess plugin

    Ͱ͏·͍͜ͱ΍͍ͬͯΔ
  18. 18.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS
  19. 19.

    Redshift • ࣾ಺ͷσʔλ෼ੳͷத৺ (DWH) • ϩά͚ͩͰͳࣾ͘಺ͷશ DB ͷσʔλΛ
 औΓࠐΜͰ͓Γ౷߹෼ੳ͕Մೳ •

    શࣾһ͕ར༻͢Δ (Tableau ΍ Redash) • S3 ͷϩάσʔλΛόονδϣϒͰऔΓࠐΈ • ϩʔυִؒ͸௕Ί
  20. 20.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS
  21. 21.

    Lambda + DynamoDB • ετϦʔϜूܭʹར༻ (ओʹ޿ࠂ) • Redshift ΁ͷΠϯϙʔτִؒΛ௒͑Δ༻్ •

    Kinesis Streams Ͱड͚ Lambda Ͱू໿ɺ DynamoDB ʹॻ͘ • ྲྀྔ͕େ͖͍ͨΊू໿ϊʔυΛ࢖Θͳ͍ • ϩάͦͷ΋ͷ͸ S3 ʹ΋ॏෳૹ৴ • ௕ظతͳ෼ੳ͸ Redshift Ͱߦ͏
  22. 22.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS
  23. 23.

    Elasticsearch • ूܭ΍μογϡϘʔυʹར༻ • αʔϏεʹ͸ར༻͞Εͳ͍ • ߴ͍ྲྀྔͷϩάʹ͸࢖Θͳ͍ or αϯϓϦϯά •

    Amazon Elasticsearch Service Λར༻
  24. 24.

    ͦͷଞ • getsentry/sentry • Τϥʔϩάऩूπʔϧ • Rails ΞϓϦέʔγϣϯͷΤϥʔΛૹ৴ • Slack

    • Ϣʔβ͔Βͷ͝ҙݟͳͲΛૹ৴
  25. 25.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS
  26. 26.

    Athena • ݕূத • Redshift ͷϩʔυִؒΛ଴ͯͳ͍͕
 SQL Ͱ෼ੳ͍ͨ͠έʔε ʹ࢖͏૝ఆ •

    Redshift Spectrum ͱ߹Θͤͯߟ͑த
  27. 27.

    -PH
 "HHSFHBUPS ,JOFTJT 4USFBNT "QQ4FSWFST 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI


    4FSWFS 3FETIJGU $MVTUFS "QQ4FSWFST "QQ4FSWFST "UIFOB &MBTUJDTFBSDI ߏ੒
  28. 28.

    શମ૾ • σʔλετΞ΍πʔϧʹΑΓಘҙෆಘҙ͕͋Δ • ΞϓϦ͔Β৮Δͱྑ͍΋ͷɺ
 ਓ͕ؒ෼ੳ͠΍͍͢΋ͷ (SQL Ͱ৮ΕΔͳͲ) • සൟͳॻ͖ࠐΈ͕ଟ͍΋ͷɺͦ͏Ͱͳ͍΋ͷ

    • αʔϏεϨϕϧ΍ڐ༰Ͱ͖Δ஗Ԇɺ༻్ͰબͿ
  29. 29.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS
  30. 30.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS Ͱ͖Δ͚ͩ୹͍஗ԆͰ ΞϓϦ͔Βͷར༻ϝΠϯ ͋Δఔ౓ͷ஗Ԇ͸ڐ༰Ͱ͖Δ ΞϓϦ͔Β΋ਓؒ΋ར༻ αʔϏεӡ༻্ ΫϦςΟΧϧͰͳ͍ϩάʹར༻
  31. 31.

    ΫοΫύουͷ
 ϩάऩू (ͦͷଞ)

  32. 32.

    ΞϓϦέʔγϣϯͷಈ࡞ϩά • td-agent Ͱऩू • CloudWatch Logs ʹૹ৴ • fluent-plugin-cloudwatch-logs

    • Docker Խ͞Ε͍ͯΔΞϓϦ͸ඪ४૷උ • Τϥʔϩά͸ Sentry ʹૹ৴
  33. 33.

    γεςϜϩά • ݱࡏ͸2ͭΛฒߦՔಈ • rsyslog • ;ͭ͏ͷϩάαʔό • td-agent Ͱऩूͯ͠

    Kinesis Streams -> Lambda Ͱ Graylog ΁ • 2.3 ͔Β Elasticsearch Service ͕࢖͑ΔΑ͏ʹ
  34. 34.

    ·ͱΊ • ΫοΫύουͷϩάऩूج൫ʹ͍ͭͯ
 ͓࿩͠·ͨ͠ • த৺ʹͳ͍ͬͯΔͷ͸ S3 ͱ Redshift •

    αʔϏεϨϕϧɺ஗Ԇɺ༻్ʹΑͬͯ
 ద੾ͳ΋ͷΛબ୒͍ͯ͘͠