Upgrade to Pro — share decks privately, control downloads, hide ads and more …

クックパッドのログをいい感じにしているアーキテクチャ / Logging architecture at Cookpad

クックパッドのログをいい感じにしているアーキテクチャ / Logging architecture at Cookpad

Hokuto Hoshi

July 26, 2017
Tweet

More Decks by Hokuto Hoshi

Other Decks in Technology

Transcript

  1. ΫοΫύουͷϩάΛ
    ͍͍ײ͡ʹ͍ͯ͠Δ
    ΞʔΩςΫνϟ
    2017/07/26 Cookpad Tech Kitchen #9
    Hokuto Hoshi
    [email protected]

    View Slide

  2. ੕ ๺ే (΄͠ ΄͘ͱ) / @kani_b
    • ΫοΫύουגࣜձࣾ

    ΠϯϑϥετϥΫνϟʔ෦௕
    • SRE
    • AWS ͱηΩϡϦςΟ͕ϝΠϯྖҬ
    • ࣾ಺֎ͷ৘ใηΩϡϦςΟશൠ΋୲౰

    View Slide

  3. ࠓ೔ͷςʔϚ: ϩά

    View Slide

  4. https://speakerdeck.com/kanny/miao-jian-shu-mo-falseroguwoiigan-zinisuruakitekutiya

    View Slide

  5. ͳͥɺਓ͸ϩάΛूΊΔͷ͔

    View Slide

  6. “ى͖ͨ͜ͱ” ͸ϩά͚͕ͩࣔ͢
    • αʔϏεվળͷͨΊʹ͸ඞཁෆՄܽ
    • Ծઆͷݕূ
    • ύϑΥʔϚϯεվળ
    • γεςϜτϥϒϧͷղফ
    • etc…

    View Slide

  7. ऩू͍ͯ͠Δϩά
    • PV ϩά
    • ϖʔδӾཡ͝ͱʹه࿥͞ΕΔ
    • Ӿཡ਺ɺݕࡧಈ޲ͷ෼ੳͳͲ
    • ޿ࠂܥϩά
    • ΠϯϓϨογϣϯɺΫϦοΫɺ

    ίϯόʔδϣϯ
    • ഑৴੍ޚ΍ޮՌଌఆʹ
    • ΞΫςΟϏςΟϩά
    • Ϣʔβߦಈ (ը໘ભҠͳͲ)
    • ෼ੳɺࢪࡦݕূɺ

    ϞόΠϧΞϓϦͷಈ࡞ϩά
    • ؂ࠪϩά
    • Ϣʔβσʔλͷ࡞੒ɾߋ৽ɾ

    ࡟আɺϩάΠϯϩάͳͲ
    • αϙʔτɺηΩϡϦςΟ؂ࠪ

    View Slide

  8. ऩू͍ͯ͠Δϩά
    • KPI ༻ϩά
    • “ग़དྷࣄ” Λه࿥͢Δϩά
    • UU, DU ͳͲ΋ࣗಈܭࢉɺ

    KPI ؅ཧ༻
    • γεςϜϩά
    • OSɺϛυϧ΢ΣΞͷϩά
    • γεςϜ؅ཧ༻
    • etc…

    View Slide

  9. ݱࡏͷن໛
    • Fluentd ʹྲྀΕ͍ͯΔϩά (֓ࢉ)
    • σʔλ૯ྔ: 400~600GB / ೔
    • Ϩίʔυ਺: 8ԯϨίʔυҎ্ / ೔
    • ඵؒ8,000 ~ 25,000 Ϩίʔυ͘Β͍
    ※ࡢ೥ͷൃද͔࣌ΒϨίʔυͷ࡟ݮΛ͍ͯ͠·͢

    View Slide

  10. ཁ͸͍Ζ͍Ζͳϩά͕

    ͍ͬͺ͍͋Δ

    View Slide

  11. ϩάʹٻΊΔཁ݅
    • ࣮֬ʹ഑ૹ͞ΕΔ͜ͱ
    • Ͱ͖Δ͚ͩ஗Ԇ͕গͳ͍͜ͱ
    • ྔΛݮΒ͢ (αϯϓϦϯά) ඞཁ͕ͳ͍͜ͱ
    • ෼ੳ͠΍͍͢ܗͰอ࣋Ͱ͖Δ͜ͱ

    View Slide

  12. ϩάͷҰੜ
    • ͋ͭΊͯ
    • ॲཧͯ͠
    • ͔ͭ͏
    ͜ͷ΁Μͷ࿩Λ͠·͢

    View Slide

  13. ΫοΫύουͷ

    ϩάऩू
    (αʔϏεͰར༻͍ͯ͠Δ΋ͷ)

    View Slide

  14. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS

    View Slide

  15. -PH

    "HHSFHBUPS
    ,JOFTJT
    4USFBNT
    "QQ4FSWFST
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "QQ4FSWFST
    "QQ4FSWFST
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒

    View Slide

  16. ΞϓϦέʔγϣϯαʔό͔Β
    • ΈΜͳେ޷͖ fluentd
    • ਖ਼֬ʹ͸ td-agent
    • Ͱ͖ΔݶΓసૹઌ͸গͳ͘
    • ू໿ϊʔυ or Kinesis Streams

    View Slide

  17. td-agent ू໿ϊʔυ
    • ॾࣄ৘ʹΑΓ·ͩ 0.12
    • શϩά͕ूத͢ΔͨΊߴෛՙ
    • multiprocess plugin Ͱ͏·͍͜ͱ΍͍ͬͯΔ

    View Slide

  18. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS

    View Slide

  19. Redshift
    • ࣾ಺ͷσʔλ෼ੳͷத৺ (DWH)
    • ϩά͚ͩͰͳࣾ͘಺ͷશ DB ͷσʔλΛ

    औΓࠐΜͰ͓Γ౷߹෼ੳ͕Մೳ
    • શࣾһ͕ར༻͢Δ (Tableau ΍ Redash)
    • S3 ͷϩάσʔλΛόονδϣϒͰऔΓࠐΈ
    • ϩʔυִؒ͸௕Ί

    View Slide

  20. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS

    View Slide

  21. Lambda + DynamoDB
    • ετϦʔϜूܭʹར༻ (ओʹ޿ࠂ)
    • Redshift ΁ͷΠϯϙʔτִؒΛ௒͑Δ༻్
    • Kinesis Streams Ͱड͚ Lambda Ͱू໿ɺ
    DynamoDB ʹॻ͘
    • ྲྀྔ͕େ͖͍ͨΊू໿ϊʔυΛ࢖Θͳ͍
    • ϩάͦͷ΋ͷ͸ S3 ʹ΋ॏෳૹ৴
    • ௕ظతͳ෼ੳ͸ Redshift Ͱߦ͏

    View Slide

  22. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS

    View Slide

  23. Elasticsearch
    • ूܭ΍μογϡϘʔυʹར༻
    • αʔϏεʹ͸ར༻͞Εͳ͍
    • ߴ͍ྲྀྔͷϩάʹ͸࢖Θͳ͍ or αϯϓϦϯά
    • Amazon Elasticsearch Service Λར༻

    View Slide

  24. ͦͷଞ
    • getsentry/sentry
    • Τϥʔϩάऩूπʔϧ
    • Rails ΞϓϦέʔγϣϯͷΤϥʔΛૹ৴
    • Slack
    • Ϣʔβ͔Βͷ͝ҙݟͳͲΛૹ৴

    View Slide

  25. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS

    View Slide

  26. Athena
    • ݕূத
    • Redshift ͷϩʔυִؒΛ଴ͯͳ͍͕

    SQL Ͱ෼ੳ͍ͨ͠έʔε ʹ࢖͏૝ఆ
    • Redshift Spectrum ͱ߹Θͤͯߟ͑த

    View Slide

  27. -PH

    "HHSFHBUPS
    ,JOFTJT
    4USFBNT
    "QQ4FSWFST
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "QQ4FSWFST
    "QQ4FSWFST
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒

    View Slide

  28. શମ૾
    • σʔλετΞ΍πʔϧʹΑΓಘҙෆಘҙ͕͋Δ
    • ΞϓϦ͔Β৮Δͱྑ͍΋ͷɺ

    ਓ͕ؒ෼ੳ͠΍͍͢΋ͷ (SQL Ͱ৮ΕΔͳͲ)
    • සൟͳॻ͖ࠐΈ͕ଟ͍΋ͷɺͦ͏Ͱͳ͍΋ͷ
    • αʔϏεϨϕϧ΍ڐ༰Ͱ͖Δ஗Ԇɺ༻్ͰબͿ

    View Slide

  29. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS

    View Slide

  30. ,JOFTJT
    4USFBNT
    4
    -BNCEB
    GVODUJPO
    %ZOBNP%#
    #BUDI

    4FSWFS
    3FETIJGU
    $MVTUFS
    "UIFOB
    &MBTUJDTFBSDI
    ߏ੒
    "QQ4FSWFST
    "QQ4FSWFST
    "QQ4FSWFST
    -PH

    "HHSFHBUPS
    Ͱ͖Δ͚ͩ୹͍஗ԆͰ
    ΞϓϦ͔Βͷར༻ϝΠϯ
    ͋Δఔ౓ͷ஗Ԇ͸ڐ༰Ͱ͖Δ
    ΞϓϦ͔Β΋ਓؒ΋ར༻
    αʔϏεӡ༻্
    ΫϦςΟΧϧͰͳ͍ϩάʹར༻

    View Slide

  31. ΫοΫύουͷ

    ϩάऩू
    (ͦͷଞ)

    View Slide

  32. ΞϓϦέʔγϣϯͷಈ࡞ϩά
    • td-agent Ͱऩू
    • CloudWatch Logs ʹૹ৴
    • fluent-plugin-cloudwatch-logs
    • Docker Խ͞Ε͍ͯΔΞϓϦ͸ඪ४૷උ
    • Τϥʔϩά͸ Sentry ʹૹ৴

    View Slide

  33. γεςϜϩά
    • ݱࡏ͸2ͭΛฒߦՔಈ
    • rsyslog
    • ;ͭ͏ͷϩάαʔό
    • td-agent Ͱऩूͯ͠ Kinesis Streams -> Lambda Ͱ
    Graylog ΁
    • 2.3 ͔Β Elasticsearch Service ͕࢖͑ΔΑ͏ʹ

    View Slide

  34. ·ͱΊ
    • ΫοΫύουͷϩάऩूج൫ʹ͍ͭͯ

    ͓࿩͠·ͨ͠
    • த৺ʹͳ͍ͬͯΔͷ͸ S3 ͱ Redshift
    • αʔϏεϨϕϧɺ஗Ԇɺ༻్ʹΑͬͯ

    ద੾ͳ΋ͷΛબ୒͍ͯ͘͠

    View Slide