クックパッドのログをいい感じにしているアーキテクチャ / Logging architecture at Cookpad

クックパッドのログをいい感じにしているアーキテクチャ / Logging architecture at Cookpad

14a602891dce5c68facca9de28340522?s=128

Hokuto Hoshi

July 26, 2017
Tweet

Transcript

  1. 2.

    ੕ ๺ే (΄͠ ΄͘ͱ) / @kani_b • ΫοΫύουגࣜձࣾ
 ΠϯϑϥετϥΫνϟʔ෦௕ •

    SRE • AWS ͱηΩϡϦςΟ͕ϝΠϯྖҬ • ࣾ಺֎ͷ৘ใηΩϡϦςΟશൠ΋୲౰
  2. 7.

    ऩू͍ͯ͠Δϩά • PV ϩά • ϖʔδӾཡ͝ͱʹه࿥͞ΕΔ • Ӿཡ਺ɺݕࡧಈ޲ͷ෼ੳͳͲ • ޿ࠂܥϩά

    • ΠϯϓϨογϣϯɺΫϦοΫɺ
 ίϯόʔδϣϯ • ഑৴੍ޚ΍ޮՌଌఆʹ • ΞΫςΟϏςΟϩά • Ϣʔβߦಈ (ը໘ભҠͳͲ) • ෼ੳɺࢪࡦݕূɺ
 ϞόΠϧΞϓϦͷಈ࡞ϩά • ؂ࠪϩά • Ϣʔβσʔλͷ࡞੒ɾߋ৽ɾ
 ࡟আɺϩάΠϯϩάͳͲ • αϙʔτɺηΩϡϦςΟ؂ࠪ
  3. 8.

    ऩू͍ͯ͠Δϩά • KPI ༻ϩά • “ग़དྷࣄ” Λه࿥͢Δϩά • UU, DU

    ͳͲ΋ࣗಈܭࢉɺ
 KPI ؅ཧ༻ • γεςϜϩά • OSɺϛυϧ΢ΣΞͷϩά • γεςϜ؅ཧ༻ • etc…
  4. 9.

    ݱࡏͷن໛ • Fluentd ʹྲྀΕ͍ͯΔϩά (֓ࢉ) • σʔλ૯ྔ: 400~600GB / ೔

    • Ϩίʔυ਺: 8ԯϨίʔυҎ্ / ೔ • ඵؒ8,000 ~ 25,000 Ϩίʔυ͘Β͍ ※ࡢ೥ͷൃද͔࣌ΒϨίʔυͷ࡟ݮΛ͍ͯ͠·͢
  5. 14.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS
  6. 15.

    -PH
 "HHSFHBUPS ,JOFTJT 4USFBNT "QQ4FSWFST 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI


    4FSWFS 3FETIJGU $MVTUFS "QQ4FSWFST "QQ4FSWFST "UIFOB &MBTUJDTFBSDI ߏ੒
  7. 18.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS
  8. 19.

    Redshift • ࣾ಺ͷσʔλ෼ੳͷத৺ (DWH) • ϩά͚ͩͰͳࣾ͘಺ͷશ DB ͷσʔλΛ
 औΓࠐΜͰ͓Γ౷߹෼ੳ͕Մೳ •

    શࣾһ͕ར༻͢Δ (Tableau ΍ Redash) • S3 ͷϩάσʔλΛόονδϣϒͰऔΓࠐΈ • ϩʔυִؒ͸௕Ί
  9. 20.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS
  10. 21.

    Lambda + DynamoDB • ετϦʔϜूܭʹར༻ (ओʹ޿ࠂ) • Redshift ΁ͷΠϯϙʔτִؒΛ௒͑Δ༻్ •

    Kinesis Streams Ͱड͚ Lambda Ͱू໿ɺ DynamoDB ʹॻ͘ • ྲྀྔ͕େ͖͍ͨΊू໿ϊʔυΛ࢖Θͳ͍ • ϩάͦͷ΋ͷ͸ S3 ʹ΋ॏෳૹ৴ • ௕ظతͳ෼ੳ͸ Redshift Ͱߦ͏
  11. 22.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS
  12. 25.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS
  13. 27.

    -PH
 "HHSFHBUPS ,JOFTJT 4USFBNT "QQ4FSWFST 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI


    4FSWFS 3FETIJGU $MVTUFS "QQ4FSWFST "QQ4FSWFST "UIFOB &MBTUJDTFBSDI ߏ੒
  14. 29.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS
  15. 30.

    ,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI
 4FSWFS 3FETIJGU $MVTUFS

    "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH
 "HHSFHBUPS Ͱ͖Δ͚ͩ୹͍஗ԆͰ ΞϓϦ͔Βͷར༻ϝΠϯ ͋Δఔ౓ͷ஗Ԇ͸ڐ༰Ͱ͖Δ ΞϓϦ͔Β΋ਓؒ΋ར༻ αʔϏεӡ༻্ ΫϦςΟΧϧͰͳ͍ϩάʹར༻
  16. 32.

    ΞϓϦέʔγϣϯͷಈ࡞ϩά • td-agent Ͱऩू • CloudWatch Logs ʹૹ৴ • fluent-plugin-cloudwatch-logs

    • Docker Խ͞Ε͍ͯΔΞϓϦ͸ඪ४૷උ • Τϥʔϩά͸ Sentry ʹૹ৴
  17. 33.

    γεςϜϩά • ݱࡏ͸2ͭΛฒߦՔಈ • rsyslog • ;ͭ͏ͷϩάαʔό • td-agent Ͱऩूͯ͠

    Kinesis Streams -> Lambda Ͱ Graylog ΁ • 2.3 ͔Β Elasticsearch Service ͕࢖͑ΔΑ͏ʹ
  18. 34.