クックパッドのログをいい感じにしているアーキテクチャ / Logging architecture at Cookpad

Slide 1

Slide 1 text

ΫοΫύουͷϩάΛ ͍͍ײ͡ʹ͍ͯ͠Δ ΞʔΩςΫνϟ 2017/07/26 Cookpad Tech Kitchen #9 Hokuto Hoshi [email protected]

Slide 2

Slide 2 text

੕ ๺ే (΄͠ ΄͘ͱ) / @kani_b • ΫοΫύουגࣜձࣾ  ΠϯϑϥετϥΫνϟʔ෦௕ • SRE • AWS ͱηΩϡϦςΟ͕ϝΠϯྖҬ • ࣾ಺֎ͷ৘ใηΩϡϦςΟશൠ΋୲౰

Slide 3

Slide 3 text

ࠓ೔ͷςʔϚ: ϩά

Slide 4

Slide 4 text

https://speakerdeck.com/kanny/miao-jian-shu-mo-falseroguwoiigan-zinisuruakitekutiya

Slide 5

Slide 5 text

ͳͥɺਓ͸ϩάΛूΊΔͷ͔

Slide 6

Slide 6 text

“ى͖ͨ͜ͱ” ͸ϩά͚͕ͩࣔ͢ • αʔϏεվળͷͨΊʹ͸ඞཁෆՄܽ • Ծઆͷݕূ • ύϑΥʔϚϯεվળ • γεςϜτϥϒϧͷղফ • etc…

Slide 7

Slide 7 text

ऩू͍ͯ͠Δϩά • PV ϩά • ϖʔδӾཡ͝ͱʹه࿥͞ΕΔ • Ӿཡ਺ɺݕࡧಈ޲ͷ෼ੳͳͲ • ޿ࠂܥϩά • ΠϯϓϨογϣϯɺΫϦοΫɺ  ίϯόʔδϣϯ • ഑৴੍ޚ΍ޮՌଌఆʹ • ΞΫςΟϏςΟϩά • Ϣʔβߦಈ (ը໘ભҠͳͲ) • ෼ੳɺࢪࡦݕূɺ  ϞόΠϧΞϓϦͷಈ࡞ϩά • ؂ࠪϩά • Ϣʔβσʔλͷ࡞੒ɾߋ৽ɾ  ࡟আɺϩάΠϯϩάͳͲ • αϙʔτɺηΩϡϦςΟ؂ࠪ

Slide 8

Slide 8 text

ऩू͍ͯ͠Δϩά • KPI ༻ϩά • “ग़དྷࣄ” Λه࿥͢Δϩά • UU, DU ͳͲ΋ࣗಈܭࢉɺ  KPI ؅ཧ༻ • γεςϜϩά • OSɺϛυϧ΢ΣΞͷϩά • γεςϜ؅ཧ༻ • etc…

Slide 9

Slide 9 text

ݱࡏͷن໛ • Fluentd ʹྲྀΕ͍ͯΔϩά (֓ࢉ) • σʔλ૯ྔ: 400~600GB / ೔ • Ϩίʔυ਺: 8ԯϨίʔυҎ্ / ೔ • ඵؒ8,000 ~ 25,000 Ϩίʔυ͘Β͍ ※ࡢ೥ͷൃද͔࣌ΒϨίʔυͷ࡟ݮΛ͍ͯ͠·͢

Slide 10

Slide 10 text

ཁ͸͍Ζ͍Ζͳϩά͕  ͍ͬͺ͍͋Δ

Slide 11

Slide 11 text

ϩάʹٻΊΔཁ݅ • ࣮֬ʹ഑ૹ͞ΕΔ͜ͱ • Ͱ͖Δ͚ͩ஗Ԇ͕গͳ͍͜ͱ • ྔΛݮΒ͢ (αϯϓϦϯά) ඞཁ͕ͳ͍͜ͱ • ෼ੳ͠΍͍͢ܗͰอ࣋Ͱ͖Δ͜ͱ

Slide 12

Slide 12 text

ϩάͷҰੜ • ͋ͭΊͯ • ॲཧͯ͠ • ͔ͭ͏ ͜ͷ΁Μͷ࿩Λ͠·͢

Slide 13

Slide 13 text

ΫοΫύουͷ  ϩάऩू (αʔϏεͰར༻͍ͯ͠Δ΋ͷ)

Slide 14

Slide 14 text

,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI  4FSWFS 3FETIJGU $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH  "HHSFHBUPS

Slide 15

Slide 15 text

-PH  "HHSFHBUPS ,JOFTJT 4USFBNT "QQ4FSWFST 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI  4FSWFS 3FETIJGU $MVTUFS "QQ4FSWFST "QQ4FSWFST "UIFOB &MBTUJDTFBSDI ߏ੒

Slide 16

Slide 16 text

ΞϓϦέʔγϣϯαʔό͔Β • ΈΜͳେ޷͖ ﬂuentd • ਖ਼֬ʹ͸ td-agent • Ͱ͖ΔݶΓసૹઌ͸গͳ͘ • ू໿ϊʔυ or Kinesis Streams

Slide 17

Slide 17 text

td-agent ू໿ϊʔυ • ॾࣄ৘ʹΑΓ·ͩ 0.12 • શϩά͕ूத͢ΔͨΊߴෛՙ • multiprocess plugin Ͱ͏·͍͜ͱ΍͍ͬͯΔ

Slide 18

Slide 18 text

,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI  4FSWFS 3FETIJGU $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH  "HHSFHBUPS

Slide 19

Slide 19 text

Redshift • ࣾ಺ͷσʔλ෼ੳͷத৺ (DWH) • ϩά͚ͩͰͳࣾ͘಺ͷશ DB ͷσʔλΛ  औΓࠐΜͰ͓Γ౷߹෼ੳ͕Մೳ • શࣾһ͕ར༻͢Δ (Tableau ΍ Redash) • S3 ͷϩάσʔλΛόονδϣϒͰऔΓࠐΈ • ϩʔυִؒ͸௕Ί

Slide 20

Slide 20 text

,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI  4FSWFS 3FETIJGU $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH  "HHSFHBUPS

Slide 21

Slide 21 text

Lambda + DynamoDB • ετϦʔϜूܭʹར༻ (ओʹ޿ࠂ) • Redshift ΁ͷΠϯϙʔτִؒΛ௒͑Δ༻్ • Kinesis Streams Ͱड͚ Lambda Ͱू໿ɺ DynamoDB ʹॻ͘ • ྲྀྔ͕େ͖͍ͨΊू໿ϊʔυΛ࢖Θͳ͍ • ϩάͦͷ΋ͷ͸ S3 ʹ΋ॏෳૹ৴ • ௕ظతͳ෼ੳ͸ Redshift Ͱߦ͏

Slide 22

Slide 22 text

,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI  4FSWFS 3FETIJGU $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH  "HHSFHBUPS

Slide 23

Slide 23 text

Elasticsearch • ूܭ΍μογϡϘʔυʹར༻ • αʔϏεʹ͸ར༻͞Εͳ͍ • ߴ͍ྲྀྔͷϩάʹ͸࢖Θͳ͍ or αϯϓϦϯά • Amazon Elasticsearch Service Λར༻

Slide 24

Slide 24 text

ͦͷଞ • getsentry/sentry • Τϥʔϩάऩूπʔϧ • Rails ΞϓϦέʔγϣϯͷΤϥʔΛૹ৴ • Slack • Ϣʔβ͔Βͷ͝ҙݟͳͲΛૹ৴

Slide 25

Slide 25 text

,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI  4FSWFS 3FETIJGU $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH  "HHSFHBUPS

Slide 26

Slide 26 text

Athena • ݕূத • Redshift ͷϩʔυִؒΛ଴ͯͳ͍͕  SQL Ͱ෼ੳ͍ͨ͠έʔε ʹ࢖͏૝ఆ • Redshift Spectrum ͱ߹Θͤͯߟ͑த

Slide 27

Slide 27 text

-PH  "HHSFHBUPS ,JOFTJT 4USFBNT "QQ4FSWFST 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI  4FSWFS 3FETIJGU $MVTUFS "QQ4FSWFST "QQ4FSWFST "UIFOB &MBTUJDTFBSDI ߏ੒

Slide 28

Slide 28 text

શମ૾ • σʔλετΞ΍πʔϧʹΑΓಘҙෆಘҙ͕͋Δ • ΞϓϦ͔Β৮Δͱྑ͍΋ͷɺ  ਓ͕ؒ෼ੳ͠΍͍͢΋ͷ (SQL Ͱ৮ΕΔͳͲ) • සൟͳॻ͖ࠐΈ͕ଟ͍΋ͷɺͦ͏Ͱͳ͍΋ͷ • αʔϏεϨϕϧ΍ڐ༰Ͱ͖Δ஗Ԇɺ༻్ͰબͿ

Slide 29

Slide 29 text

,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI  4FSWFS 3FETIJGU $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH  "HHSFHBUPS

Slide 30

Slide 30 text

,JOFTJT 4USFBNT 4 -BNCEB GVODUJPO %ZOBNP%# #BUDI  4FSWFS 3FETIJGU $MVTUFS "UIFOB &MBTUJDTFBSDI ߏ੒ "QQ4FSWFST "QQ4FSWFST "QQ4FSWFST -PH  "HHSFHBUPS Ͱ͖Δ͚ͩ୹͍஗ԆͰ ΞϓϦ͔Βͷར༻ϝΠϯ ͋Δఔ౓ͷ஗Ԇ͸ڐ༰Ͱ͖Δ ΞϓϦ͔Β΋ਓؒ΋ར༻ αʔϏεӡ༻্ ΫϦςΟΧϧͰͳ͍ϩάʹར༻

Slide 31

Slide 31 text

ΫοΫύουͷ  ϩάऩू (ͦͷଞ)

Slide 32

Slide 32 text

ΞϓϦέʔγϣϯͷಈ࡞ϩά • td-agent Ͱऩू • CloudWatch Logs ʹૹ৴ • ﬂuent-plugin-cloudwatch-logs • Docker Խ͞Ε͍ͯΔΞϓϦ͸ඪ४૷උ • Τϥʔϩά͸ Sentry ʹૹ৴

Slide 33

Slide 33 text

γεςϜϩά • ݱࡏ͸2ͭΛฒߦՔಈ • rsyslog • ;ͭ͏ͷϩάαʔό • td-agent Ͱऩूͯ͠ Kinesis Streams -> Lambda Ͱ Graylog ΁ • 2.3 ͔Β Elasticsearch Service ͕࢖͑ΔΑ͏ʹ

Slide 34

Slide 34 text

·ͱΊ • ΫοΫύουͷϩάऩूج൫ʹ͍ͭͯ  ͓࿩͠·ͨ͠ • த৺ʹͳ͍ͬͯΔͷ͸ S3 ͱ Redshift • αʔϏεϨϕϧɺ஗Ԇɺ༻్ʹΑͬͯ  ద੾ͳ΋ͷΛબ୒͍ͯ͘͠