Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Gunosy のログ収集基盤
Search
mgi166
June 23, 2018
1
660
Gunosy のログ収集基盤
mgi166
June 23, 2018
Tweet
Share
More Decks by mgi166
See All by mgi166
realtime_rails
mgi166
0
39
Music as Code
mgi166
0
570
Featured
See All Featured
How to train your dragon (web standard)
notwaldorf
71
5.1k
10 Git Anti Patterns You Should be Aware of
lemiorhan
644
57k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
19
1.9k
Raft: Consensus for Rubyists
vanstee
130
6.2k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
1
1.2k
The Invisible Customer
myddelton
114
12k
KATA
mclloyd
14
11k
Fontdeck: Realign not Redesign
paulrobertlloyd
75
4.8k
Six Lessons from altMBA
skipperchong
19
2.9k
Infographics Made Easy
chrislema
237
18k
BBQ
matthewcrist
78
8.7k
Code Review Best Practice
trishagee
54
15k
Transcript
Gunosy ͷϩάऩूج൫ Architecture Night #1 ໜେເ@mgi166
ࣗݾհ • גࣜձࣾ Gunosy ։ൃɾӡ༻ਪਐ෦ • ໜେເ • ݩ Rails
ΤϯδχΞ • ࠓ Gunosy ͷ AWS पΓɺΠϯϑϥ पΓΛ୲
ຊͷ༰ • Gunosy ͷϩάपΓΛத৺ͱͨ͠ɺΠϯϑϥ ΞʔΩςΫνϟͷࣄྫΛհ͠·͢
ϩάΛूΊΔ
ͳΜͰϩάΛूΊΔͷʁ
ྫ͑͜Μͳཧ༝ • ࣮ࢪͨ͠ࢪࡦͷԠΛݟΔͨΊ • ϢʔβʔͲ͏Ԡͨ͠ͷ͔ʁΛΔ • ϓϩμΫτΛΑΓྑ͘ϒϥογϡΞοϓ͢ΔͨΊ • ϩάΛ input
ʹɺػցֶशͰ࡞ͬͨϞσϧΛڧ͍ͯ͘͘͠ • Τϥʔ͕͋ͬͨͱ͖ͷݪҼௐࠪ • ϩά͕ͳ͍ͱݪҼΛ͑ͳ͍ • ϩάʹྲྀΕΔϝοηʔδ͔ΒɺΞϥʔτΛఆٛͯ͠ɺҟৗʹؾ͖͍ͮͨ • ා͍ਓʹ٧ΊΒΕͨͱ͖ʹɺূڌͱͳΔهΛग़ͨ͢Ί • ϩάΛஷΊΔ͜ͱ͕ɺސ٬ʹఏڙ͢ΔՁͰ͋Δ • e.x.) Papertrail, mackerel.io, etc…
ྫ͑͜Μͳཧ༝ • ࣮ࢪͨ͠ࢪࡦͷԠΛݟΔͨΊ • ϢʔβʔͲ͏Ԡͨ͠ͷ͔ʁΛΔ • ϓϩμΫτΛΑΓྑ͘ϒϥογϡΞοϓ͢ΔͨΊ • ϩάΛ input
ʹɺػցֶशͰ࡞ͬͨϞσϧΛڧ͍ͯ͘͘͠ • Τϥʔ͕͋ͬͨͱ͖ͷݪҼௐࠪ • ϩά͕ͳ͍ͱݪҼΛ͑ͳ͍ • ϩάʹྲྀΕΔϝοηʔδ͔ΒɺΞϥʔτΛఆٛͯ͠ɺҟৗʹؾ͖͍ͮͨ • ා͍ਓʹ٧ΊΒΕͨͱ͖ʹɺূڌͱͳΔهΛग़ͨ͢Ί • ϩάΛஷΊΔ͜ͱ͕ɺސ٬ʹఏڙ͢ΔՁͰ͋Δ • e.x.) Papertrail, mackerel.io, etc…
ຊͷ༰ • Gunosy ͰϩάΛͲ͏ूΊͯɺͲ͏׆༻ͯ͠ ͍Δ͔ • ੳ༻్Ͱ׆༻͢Δ • ϓϩμΫτվળ༻్Ͱ׆༻͢Δ
ੳ༻్
·ͣ Gunosy ͷུ֓ਤ(ࡶver) ػցֶश σʔλஔ͖ API αʔόʔ
ੳʹ͍ؔͯ͠Δ෦͜͜ ػցֶश σʔλஔ͖ API αʔόʔ
ϩάͲ͏ྲྀΕ͍͔ͯ͘ • fluentd Ͱ S3 or BQ • ETL
or όονͰ DB ʹอଘ • Re:dash ͰՄࢹԽ API server
Re:dash ͷ DataSource όονͰੜ raw log Redshift Re:dash formated
log parquet BigQuery batch ECS RDS
Re:dash ேձͰຖݟΔ
͜͜·Ͱͷ·ͱΊ • ීஈͷϩάͷҰ࣍ஔ͖ S3 ͱ BQ • ඞཁʹԠͯ͡ DB ʹϩʔυͯ͠ɺRe:dash
ͰՄ ࢹԽ • ՄࢹԽͨ͠ Re:dash ͷओͳ KPI ேձͰڞ༗ • (ߏࣗମׂͱී௨)
ϓϩμΫτվળ༻్
Ͳ͏͍͏ϩά͕Ͳ͏ΘΕΔʁ • Ͳ͏͍͏ϩά͕ʁ • Ϣʔβʔͷߦಈϩά • هࣄͷΫϦοΫϩά • ࠂͷΠϯϓϨογϣϯ •
ͦͷଞ৭ʑ… • Ͳ͏ΘΕΔʁ • Ϣʔβʔʹ࠷దͳใΛಧ͚ΔͨΊʹੜ͔͞ΕΔ • ϢʔβʔͻͱΓͻͱΓʹରͯ͠࠷దͳهࣄΛग़͢͜ͱʹΘΕΔ • Ϣʔβʔʹରͯ͠ڵຯͷ͋Δ Push ௨Λଧͭ
શମ૾ click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2 user
vector ͍͍ײ͡ʹ͢ΔϚϯ # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -ϫʔΫϑϩʔฤ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/gunosy-personalize-digdag-workflow)
هࣄใͷείΞϦϯά click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2
user vector # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -ϫʔΫϑϩʔฤ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/gunosy-personalize-digdag-workflow)
Ϣʔβʔใͱهࣄσʔλ user data • S3 ʹཷ·͍ͬͯΔϩά͚ͩͰͳͯ͘ɺRDS ͔ΒఆظతʹɺϢʔβʔͷଐੑใΛ S3 ʹஔ͍ͯ ͍Δ
• ϢʔβʔใͳͲͷٵ্͍͛ɺόον + ఆظ࣮ߦͰཧ • digdag on ECS Ͱಈ͔͍ͯ͠Δ • ٵ্͍͛ͨσʔλͱϩάσʔλͱͷ join + ൿີͳʹ͔ɺAirflow on ECS Ͱཧ log ECS RDS ECS EMR ‟vector ‟vector
Ϣʔβʔଐੑͷߋ৽ click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2 user
vector GET ͍͍ײ͡ʹ͢ΔϚϯ
ϦΞϧλΠϜͰϢʔβʔϕΫτϧ Λߋ৽ # ϢʔβʔߦಈͷཧϞσϧͱ ߴਪનγεςϜ (https://speakerdeck.com/mathetake/yusaxing-dong-falseshu-li-moteruto-gao-su-tui-jian-sisutemu) Click log stream Push/Trim
Trigger Put Put Batch Get Batch Get Click logger Article Vectorizer Crawler Article Vector User Vector • click log client ͔Β kinesis stream ʹૹΓɺ lambda Λ௨ͯ͠ dynamo
શମ૾(࠶ܝ) click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2
user vector # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -1ΫϦοΫͰ࢝·ΔύʔιφϥΠζ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/realtime-vectorization-with-dynamodb)
͜͜·Ͱͷ·ͱΊ • ϢʔβʔͷߦಈϩάΛͱʹɺϓϩμΫτͷվળΛߦͬ ͍ͯΔ • ಛʹϢʔβʔͷΫϦοΫϩά͔ΒɺϢʔβʔϕΫτϧߋ ৽Λ΄΅ϦΞϧλΠϜߦ͍ͬͯΔ • Ұ෦ϦΞϧλΠϜͳධՁΛߦ͍ͬͯͳͯ͘ɺόονͰ ରԠ
• ͜ΕΒͷόονɺdigdag on ECS Ͱߦ͍ͬͯΔ
·ͱΊ
ӡ༻͍ͯͯ͠ࢥ͏͜ͱ • ྑ͘ѱ͘ AWS ͲͬΓਁ͔ͬͯΔ • ϑϧϚωʔδυͰɺॳظಋೖɺӡ༻ίετ͍ͷͷɺ AWS ͷ༷ʹҾͬுΒΕ ͍ͯΔ෦৭ʑ͋Δͱࢥ͏
• ϩάੳ༻్ͷ DBɺS3 bucket ֤ॴʹࢄΒͬͯΔͷͰɺΧΦεײ • ͜ΕͬͯΔ͚ͬʁɺ࣮΄ͱΜͲͬͯͳ͍ɺΈ͍ͨͷ͍͔ͭ͋͘Δͣ • ੳ༻్Ͱ͏ϩάͷεΩʔϚཧʹ͍ͭͯ·͍͠ • json Ͱอଘ͠ɺඞཁʹԠͯ͡ json Λղͨ͠Ұ࣌ςʔϒϧΛ࡞ͬͯରॲ • σʔλࣗ༝ʹ৮ΕΔҰํɺύʔςΟγϣϯࢦఆͤͣɺϑϧεΩϟϯΫΤϦ࣮ߦ͠ଠ ͕ఆظతʹग़ݱ͢Δ • ϚωʔతͳҙຯͰ͏͔ͬΓ௧खΛ͏ͷආ͚͍͕ͨɺ͔ͩΒͱ͍ͬͯΞΫηεݖ Λୣ͏ͷҧ͏ؾ͕͢Δ
·ͱΊ • Gunosy Ͱѻ͍ͬͯΔϩάΛɺʮੳ༻్ʯʮϓϩμΫτվળ༻్ʯ͚ ͯͦΕͧΕ͝հ͠·ͨ͠ • ੳ༻్ͰɺηΦϦʔ௨Γͷߏ • S3 ->
தؒσʔλετϨʔδ -> Re:dash • ओཁͳ KPI ɺRe:dash ΛͬͯάϥϑԽ͠ɺேձͰຖڞ༗͍ͯ͠Δ • ϓϩμΫτվળ༻్Ͱɺόον + ϦΞϧλΠϜͰϩάΛ͍ͬͯ·͢ • AWS ʹେ͖͘ॿ͚ΒΕ͍ͯ·͢
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠