Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Gunosy のログ収集基盤
Search
mgi166
June 23, 2018
2
690
Gunosy のログ収集基盤
mgi166
June 23, 2018
Tweet
Share
More Decks by mgi166
See All by mgi166
realtime_rails
mgi166
0
44
Music as Code
mgi166
0
600
Featured
See All Featured
Rebuilding a faster, lazier Slack
samanthasiow
79
8.7k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
26
2.1k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
250
21k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
169
50k
Designing the Hi-DPI Web
ddemaree
280
34k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
28
8.2k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
7
560
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
33k
What's in a price? How to price your products and services
michaelherold
243
12k
Music & Morning Musume
bryan
46
6.2k
Why Our Code Smells
bkeepers
PRO
334
57k
10 Git Anti Patterns You Should be Aware of
lemiorhan
654
59k
Transcript
Gunosy ͷϩάऩूج൫ Architecture Night #1 ໜେເ@mgi166
ࣗݾհ • גࣜձࣾ Gunosy ։ൃɾӡ༻ਪਐ෦ • ໜେເ • ݩ Rails
ΤϯδχΞ • ࠓ Gunosy ͷ AWS पΓɺΠϯϑϥ पΓΛ୲
ຊͷ༰ • Gunosy ͷϩάपΓΛத৺ͱͨ͠ɺΠϯϑϥ ΞʔΩςΫνϟͷࣄྫΛհ͠·͢
ϩάΛूΊΔ
ͳΜͰϩάΛूΊΔͷʁ
ྫ͑͜Μͳཧ༝ • ࣮ࢪͨ͠ࢪࡦͷԠΛݟΔͨΊ • ϢʔβʔͲ͏Ԡͨ͠ͷ͔ʁΛΔ • ϓϩμΫτΛΑΓྑ͘ϒϥογϡΞοϓ͢ΔͨΊ • ϩάΛ input
ʹɺػցֶशͰ࡞ͬͨϞσϧΛڧ͍ͯ͘͘͠ • Τϥʔ͕͋ͬͨͱ͖ͷݪҼௐࠪ • ϩά͕ͳ͍ͱݪҼΛ͑ͳ͍ • ϩάʹྲྀΕΔϝοηʔδ͔ΒɺΞϥʔτΛఆٛͯ͠ɺҟৗʹؾ͖͍ͮͨ • ා͍ਓʹ٧ΊΒΕͨͱ͖ʹɺূڌͱͳΔهΛग़ͨ͢Ί • ϩάΛஷΊΔ͜ͱ͕ɺސ٬ʹఏڙ͢ΔՁͰ͋Δ • e.x.) Papertrail, mackerel.io, etc…
ྫ͑͜Μͳཧ༝ • ࣮ࢪͨ͠ࢪࡦͷԠΛݟΔͨΊ • ϢʔβʔͲ͏Ԡͨ͠ͷ͔ʁΛΔ • ϓϩμΫτΛΑΓྑ͘ϒϥογϡΞοϓ͢ΔͨΊ • ϩάΛ input
ʹɺػցֶशͰ࡞ͬͨϞσϧΛڧ͍ͯ͘͘͠ • Τϥʔ͕͋ͬͨͱ͖ͷݪҼௐࠪ • ϩά͕ͳ͍ͱݪҼΛ͑ͳ͍ • ϩάʹྲྀΕΔϝοηʔδ͔ΒɺΞϥʔτΛఆٛͯ͠ɺҟৗʹؾ͖͍ͮͨ • ා͍ਓʹ٧ΊΒΕͨͱ͖ʹɺূڌͱͳΔهΛग़ͨ͢Ί • ϩάΛஷΊΔ͜ͱ͕ɺސ٬ʹఏڙ͢ΔՁͰ͋Δ • e.x.) Papertrail, mackerel.io, etc…
ຊͷ༰ • Gunosy ͰϩάΛͲ͏ूΊͯɺͲ͏׆༻ͯ͠ ͍Δ͔ • ੳ༻్Ͱ׆༻͢Δ • ϓϩμΫτվળ༻్Ͱ׆༻͢Δ
ੳ༻్
·ͣ Gunosy ͷུ֓ਤ(ࡶver) ػցֶश σʔλஔ͖ API αʔόʔ
ੳʹ͍ؔͯ͠Δ෦͜͜ ػցֶश σʔλஔ͖ API αʔόʔ
ϩάͲ͏ྲྀΕ͍͔ͯ͘ • fluentd Ͱ S3 or BQ • ETL
or όονͰ DB ʹอଘ • Re:dash ͰՄࢹԽ API server
Re:dash ͷ DataSource όονͰੜ raw log Redshift Re:dash formated
log parquet BigQuery batch ECS RDS
Re:dash ேձͰຖݟΔ
͜͜·Ͱͷ·ͱΊ • ීஈͷϩάͷҰ࣍ஔ͖ S3 ͱ BQ • ඞཁʹԠͯ͡ DB ʹϩʔυͯ͠ɺRe:dash
ͰՄ ࢹԽ • ՄࢹԽͨ͠ Re:dash ͷओͳ KPI ேձͰڞ༗ • (ߏࣗମׂͱී௨)
ϓϩμΫτվળ༻్
Ͳ͏͍͏ϩά͕Ͳ͏ΘΕΔʁ • Ͳ͏͍͏ϩά͕ʁ • Ϣʔβʔͷߦಈϩά • هࣄͷΫϦοΫϩά • ࠂͷΠϯϓϨογϣϯ •
ͦͷଞ৭ʑ… • Ͳ͏ΘΕΔʁ • Ϣʔβʔʹ࠷దͳใΛಧ͚ΔͨΊʹੜ͔͞ΕΔ • ϢʔβʔͻͱΓͻͱΓʹରͯ͠࠷దͳهࣄΛग़͢͜ͱʹΘΕΔ • Ϣʔβʔʹରͯ͠ڵຯͷ͋Δ Push ௨Λଧͭ
શମ૾ click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2 user
vector ͍͍ײ͡ʹ͢ΔϚϯ # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -ϫʔΫϑϩʔฤ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/gunosy-personalize-digdag-workflow)
هࣄใͷείΞϦϯά click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2
user vector # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -ϫʔΫϑϩʔฤ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/gunosy-personalize-digdag-workflow)
Ϣʔβʔใͱهࣄσʔλ user data • S3 ʹཷ·͍ͬͯΔϩά͚ͩͰͳͯ͘ɺRDS ͔ΒఆظతʹɺϢʔβʔͷଐੑใΛ S3 ʹஔ͍ͯ ͍Δ
• ϢʔβʔใͳͲͷٵ্͍͛ɺόον + ఆظ࣮ߦͰཧ • digdag on ECS Ͱಈ͔͍ͯ͠Δ • ٵ্͍͛ͨσʔλͱϩάσʔλͱͷ join + ൿີͳʹ͔ɺAirflow on ECS Ͱཧ log ECS RDS ECS EMR ‟vector ‟vector
Ϣʔβʔଐੑͷߋ৽ click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2 user
vector GET ͍͍ײ͡ʹ͢ΔϚϯ
ϦΞϧλΠϜͰϢʔβʔϕΫτϧ Λߋ৽ # ϢʔβʔߦಈͷཧϞσϧͱ ߴਪનγεςϜ (https://speakerdeck.com/mathetake/yusaxing-dong-falseshu-li-moteruto-gao-su-tui-jian-sisutemu) Click log stream Push/Trim
Trigger Put Put Batch Get Batch Get Click logger Article Vectorizer Crawler Article Vector User Vector • click log client ͔Β kinesis stream ʹૹΓɺ lambda Λ௨ͯ͠ dynamo
શମ૾(࠶ܝ) click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2
user vector # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -1ΫϦοΫͰ࢝·ΔύʔιφϥΠζ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/realtime-vectorization-with-dynamodb)
͜͜·Ͱͷ·ͱΊ • ϢʔβʔͷߦಈϩάΛͱʹɺϓϩμΫτͷվળΛߦͬ ͍ͯΔ • ಛʹϢʔβʔͷΫϦοΫϩά͔ΒɺϢʔβʔϕΫτϧߋ ৽Λ΄΅ϦΞϧλΠϜߦ͍ͬͯΔ • Ұ෦ϦΞϧλΠϜͳධՁΛߦ͍ͬͯͳͯ͘ɺόονͰ ରԠ
• ͜ΕΒͷόονɺdigdag on ECS Ͱߦ͍ͬͯΔ
·ͱΊ
ӡ༻͍ͯͯ͠ࢥ͏͜ͱ • ྑ͘ѱ͘ AWS ͲͬΓਁ͔ͬͯΔ • ϑϧϚωʔδυͰɺॳظಋೖɺӡ༻ίετ͍ͷͷɺ AWS ͷ༷ʹҾͬுΒΕ ͍ͯΔ෦৭ʑ͋Δͱࢥ͏
• ϩάੳ༻్ͷ DBɺS3 bucket ֤ॴʹࢄΒͬͯΔͷͰɺΧΦεײ • ͜ΕͬͯΔ͚ͬʁɺ࣮΄ͱΜͲͬͯͳ͍ɺΈ͍ͨͷ͍͔ͭ͋͘Δͣ • ੳ༻్Ͱ͏ϩάͷεΩʔϚཧʹ͍ͭͯ·͍͠ • json Ͱอଘ͠ɺඞཁʹԠͯ͡ json Λղͨ͠Ұ࣌ςʔϒϧΛ࡞ͬͯରॲ • σʔλࣗ༝ʹ৮ΕΔҰํɺύʔςΟγϣϯࢦఆͤͣɺϑϧεΩϟϯΫΤϦ࣮ߦ͠ଠ ͕ఆظతʹग़ݱ͢Δ • ϚωʔతͳҙຯͰ͏͔ͬΓ௧खΛ͏ͷආ͚͍͕ͨɺ͔ͩΒͱ͍ͬͯΞΫηεݖ Λୣ͏ͷҧ͏ؾ͕͢Δ
·ͱΊ • Gunosy Ͱѻ͍ͬͯΔϩάΛɺʮੳ༻్ʯʮϓϩμΫτվળ༻్ʯ͚ ͯͦΕͧΕ͝հ͠·ͨ͠ • ੳ༻్ͰɺηΦϦʔ௨Γͷߏ • S3 ->
தؒσʔλετϨʔδ -> Re:dash • ओཁͳ KPI ɺRe:dash ΛͬͯάϥϑԽ͠ɺேձͰຖڞ༗͍ͯ͠Δ • ϓϩμΫτվળ༻్Ͱɺόον + ϦΞϧλΠϜͰϩάΛ͍ͬͯ·͢ • AWS ʹେ͖͘ॿ͚ΒΕ͍ͯ·͢
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠