Gunosy のログ収集基盤

Slide 1

Slide 1 text

Gunosy ͷϩάऩूج൫ Architecture Night #1 ໜ໦େເ@mgi166

Slide 2

Slide 2 text

ࣗݾ঺հ • גࣜձࣾ Gunosy ։ൃɾӡ༻ਪਐ෦ • ໜ໦େເ • ݩ Rails ΤϯδχΞ • ࠓ͸ Gunosy ͷ AWS पΓɺΠϯϑϥ पΓΛ୲౰

Slide 3

Slide 3 text

ຊ೔ͷ಺༰ • Gunosy ͷϩάपΓΛத৺ͱͨ͠ɺΠϯϑϥ ΞʔΩςΫνϟͷࣄྫΛ঺հ͠·͢

Slide 4

Slide 4 text

ϩάΛूΊΔ

Slide 5

Slide 5 text

ͳΜͰϩάΛूΊΔͷʁ

Slide 6

Slide 6 text

ྫ͑͹͜Μͳཧ༝ • ࣮ࢪͨ͠ࢪࡦͷ൓ԠΛݟΔͨΊ • Ϣʔβʔ͸Ͳ͏൓Ԡͨ͠ͷ͔ʁΛ஌Δ • ϓϩμΫτΛΑΓྑ͘ϒϥογϡΞοϓ͢ΔͨΊ • ϩάΛ input ʹɺػցֶशͰ࡞ͬͨϞσϧΛڧ͍ͯ͘͘͠ • Τϥʔ͕͋ͬͨͱ͖ͷݪҼௐࠪ • ϩά͕ͳ͍ͱݪҼΛ௥͑ͳ͍ • ϩάʹྲྀΕΔϝοηʔδ͔ΒɺΞϥʔτΛఆٛͯ͠ɺҟৗʹؾ͖͍ͮͨ • ා͍ਓʹ٧ΊΒΕͨͱ͖ʹɺূڌͱͳΔه࿥Λग़ͨ͢Ί • ϩάΛஷΊΔ͜ͱ͕ɺސ٬ʹఏڙ͢ΔՁ஋Ͱ͋Δ • e.x.) Papertrail, mackerel.io, etc…

Slide 7

Slide 7 text

Slide 8

Slide 8 text

ຊ೔ͷ಺༰ • Gunosy Ͱ͸ϩάΛͲ͏ूΊͯɺͲ͏׆༻ͯ͠ ͍Δ͔ • ෼ੳ༻్Ͱ׆༻͢Δ • ϓϩμΫτվળ༻్Ͱ׆༻͢Δ

Slide 9

Slide 9 text

෼ੳ༻్

Slide 10

Slide 10 text

·ͣ Gunosy ͷུ֓ਤ(ࡶver) ػցֶश σʔλஔ͖৔ API αʔόʔ

Slide 11

Slide 11 text

෼ੳʹؔ܎͍ͯ͠Δ෦෼͸͜͜ ػցֶश σʔλஔ͖৔ API αʔόʔ

Slide 12

Slide 12 text

ϩά͸Ͳ͏ྲྀΕ͍͔ͯ͘ • ﬂuentd Ͱ S3 or BQ ΁ • ETL or όονͰ DB ʹอଘ • Re:dash ͰՄࢹԽ API server

Slide 13

Slide 13 text

Re:dash ͷ DataSource ͸ όονͰੜ੒ raw log Redshift Re:dash formated log parquet BigQuery batch ECS RDS

Slide 14

Slide 14 text

Re:dash ͸ேձͰຖ೔ݟΔ

Slide 15

Slide 15 text

͜͜·Ͱͷ·ͱΊ • ීஈͷϩάͷҰ࣍ஔ͖৔͸ S3 ͱ BQ • ඞཁʹԠͯ͡ DB ʹϩʔυͯ͠ɺRe:dash ͰՄ ࢹԽ • ՄࢹԽͨ͠ Re:dash ͷओͳ KPI ͸ேձͰڞ༗ • (ߏ੒ࣗମ͸ׂͱී௨)

Slide 16

Slide 16 text

ϓϩμΫτվળ༻్

Slide 17

Slide 17 text

Ͳ͏͍͏ϩά͕Ͳ͏࢖ΘΕΔʁ • Ͳ͏͍͏ϩά͕ʁ • Ϣʔβʔͷߦಈϩά • هࣄͷΫϦοΫϩά • ޿ࠂͷΠϯϓϨογϣϯ • ͦͷଞ৭ʑ… • Ͳ͏࢖ΘΕΔʁ • Ϣʔβʔʹ࠷దͳ৘ใΛಧ͚ΔͨΊʹੜ͔͞ΕΔ • ϢʔβʔͻͱΓͻͱΓʹରͯ͠࠷దͳهࣄΛग़͢͜ͱʹ࢖ΘΕΔ • Ϣʔβʔʹରͯ͠ڵຯͷ͋Δ Push ௨஌Λଧͭ

Slide 18

Slide 18 text

શମ૾ click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2 user vector ͍͍ײ͡ʹ͢ΔϚϯ # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -ϫʔΫϑϩʔฤ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/gunosy-personalize-digdag-workﬂow)

Slide 19

Slide 19 text

هࣄ৘ใͷείΞϦϯά click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2 user vector # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -ϫʔΫϑϩʔฤ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/gunosy-personalize-digdag-workﬂow)

Slide 20

Slide 20 text

Ϣʔβʔ৘ใͱهࣄσʔλ user data • S3 ʹཷ·͍ͬͯΔϩά͚ͩͰͳͯ͘ɺRDS ͔Β΋ఆظతʹɺϢʔβʔͷଐੑ৘ใΛ S3 ʹஔ͍ͯ ͍Δ • Ϣʔβʔ৘ใͳͲͷٵ্͍͛͸ɺόον + ఆظ࣮ߦͰ؅ཧ • digdag on ECS Ͱಈ͔͍ͯ͠Δ • ٵ্͍͛ͨσʔλͱϩάσʔλͱͷ join + ൿີͳʹ͔͸ɺAirﬂow on ECS Ͱ؅ཧ log ECS RDS ECS EMR ‟vector ‟vector

Slide 21

Slide 21 text

Ϣʔβʔଐੑͷߋ৽ click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2 user vector GET ͍͍ײ͡ʹ͢ΔϚϯ

Slide 22

Slide 22 text

ϦΞϧλΠϜͰϢʔβʔϕΫτϧ Λߋ৽ # Ϣʔβʔߦಈͷ਺ཧϞσϧͱ ߴ଎ਪનγεςϜ (https://speakerdeck.com/mathetake/yusaxing-dong-falseshu-li-moteruto-gao-su-tui-jian-sisutemu) Click log stream Push/Trim Trigger Put Put Batch Get Batch Get Click logger Article Vectorizer Crawler Article Vector User Vector • click log ͸ client ͔Β kinesis stream ʹ௚઀ૹΓɺ lambda Λ௨ͯ͠ dynamo ΁

Slide 23

Slide 23 text

શମ૾(࠶ܝ) click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2 user vector # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -1ΫϦοΫͰ࢝·ΔύʔιφϥΠζ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/realtime-vectorization-with-dynamodb)

Slide 24

Slide 24 text

͜͜·Ͱͷ·ͱΊ • ϢʔβʔͷߦಈϩάΛ΋ͱʹɺϓϩμΫτͷվળΛߦͬ ͍ͯΔ • ಛʹϢʔβʔͷΫϦοΫϩά͔ΒɺϢʔβʔϕΫτϧߋ ৽Λ΄΅ϦΞϧλΠϜߦ͍ͬͯΔ • Ұ෦͸ϦΞϧλΠϜͳධՁΛߦ͍ͬͯͳͯ͘ɺόονͰ ରԠ • ͜ΕΒͷόον͸ɺdigdag on ECS Ͱߦ͍ͬͯΔ

Slide 25

Slide 25 text

·ͱΊ

Slide 26

Slide 26 text

ӡ༻͍ͯͯ͠ࢥ͏͜ͱ • ྑ͘΋ѱ͘΋ AWS Ͳͬ΀Γਁ͔ͬͯΔ • ϑϧϚωʔδυͰɺॳظಋೖɺӡ༻ίετ͸௿͍΋ͷͷɺ AWS ͷ࢓༷ʹҾͬுΒΕ ͍ͯΔ෦෼͸৭ʑ͋Δͱࢥ͏ • ϩά΍෼ੳ༻్ͷ DBɺS3 bucket ͸֤ॴʹࢄΒ͹ͬͯΔͷͰɺΧΦεײ • ͜Ε࢖ͬͯΔ͚ͬʁɺ࣮͸΄ͱΜͲ࢖ͬͯͳ͍ɺΈ͍ͨͷ͸͍͔ͭ͋͘Δ͸ͣ • ෼ੳ༻్Ͱ࢖͏ϩάͷεΩʔϚ؅ཧʹ͍ͭͯ͸೰·͍͠ • json Ͱอଘ͠ɺඞཁʹԠͯ͡ json Λ෼ղͨ͠Ұ࣌ςʔϒϧΛ࡞ͬͯରॲ • σʔλ͸ࣗ༝ʹ৮ΕΔҰํɺύʔςΟγϣϯࢦఆͤͣɺϑϧεΩϟϯΫΤϦ࣮ߦ͠ଠ࿠ ͕ఆظతʹग़ݱ͢Δ • ϚωʔతͳҙຯͰ͏͔ͬΓ௧खΛ௥͏ͷ͸ආ͚͍͕ͨɺ͔ͩΒͱ͍ͬͯΞΫηεݖ Λୣ͏ͷ͸ҧ͏ؾ͕͢Δ

Slide 27

Slide 27 text

·ͱΊ • Gunosy Ͱѻ͍ͬͯΔϩάΛɺʮ෼ੳ༻్ʯʮϓϩμΫτվળ༻్ʯ෼͚ ͯͦΕͧΕ͝঺հ͠·ͨ͠ • ෼ੳ༻్Ͱ͸ɺηΦϦʔ௨Γͷߏ੒ • S3 -> தؒσʔλετϨʔδ -> Re:dash • ओཁͳ KPI ͸ɺRe:dash Λ࢖ͬͯάϥϑԽ͠ɺேձͰຖ೔ڞ༗͍ͯ͠Δ • ϓϩμΫτվળ༻్Ͱ͸ɺόον + ϦΞϧλΠϜͰϩάΛ࢖͍ͬͯ·͢ • AWS ʹେ͖͘ॿ͚ΒΕ͍ͯ·͢

Slide 28

Slide 28 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠