Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Gunosy のログ収集基盤
Search
mgi166
June 23, 2018
2
690
Gunosy のログ収集基盤
mgi166
June 23, 2018
Tweet
Share
More Decks by mgi166
See All by mgi166
realtime_rails
mgi166
0
44
Music as Code
mgi166
0
600
Featured
See All Featured
Docker and Python
trallard
40
3.1k
Building Flexible Design Systems
yeseniaperezcruz
327
38k
The Pragmatic Product Professional
lauravandoore
31
6.3k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
159
15k
Gamification - CAS2011
davidbonilla
80
5k
YesSQL, Process and Tooling at Scale
rocio
169
14k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.5k
The Power of CSS Pseudo Elements
geoffreycrofte
73
5.3k
Why You Should Never Use an ORM
jnunemaker
PRO
54
9.1k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
33
1.9k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
0
120
Being A Developer After 40
akosma
87
590k
Transcript
Gunosy ͷϩάऩूج൫ Architecture Night #1 ໜେເ@mgi166
ࣗݾհ • גࣜձࣾ Gunosy ։ൃɾӡ༻ਪਐ෦ • ໜେເ • ݩ Rails
ΤϯδχΞ • ࠓ Gunosy ͷ AWS पΓɺΠϯϑϥ पΓΛ୲
ຊͷ༰ • Gunosy ͷϩάपΓΛத৺ͱͨ͠ɺΠϯϑϥ ΞʔΩςΫνϟͷࣄྫΛհ͠·͢
ϩάΛूΊΔ
ͳΜͰϩάΛूΊΔͷʁ
ྫ͑͜Μͳཧ༝ • ࣮ࢪͨ͠ࢪࡦͷԠΛݟΔͨΊ • ϢʔβʔͲ͏Ԡͨ͠ͷ͔ʁΛΔ • ϓϩμΫτΛΑΓྑ͘ϒϥογϡΞοϓ͢ΔͨΊ • ϩάΛ input
ʹɺػցֶशͰ࡞ͬͨϞσϧΛڧ͍ͯ͘͘͠ • Τϥʔ͕͋ͬͨͱ͖ͷݪҼௐࠪ • ϩά͕ͳ͍ͱݪҼΛ͑ͳ͍ • ϩάʹྲྀΕΔϝοηʔδ͔ΒɺΞϥʔτΛఆٛͯ͠ɺҟৗʹؾ͖͍ͮͨ • ා͍ਓʹ٧ΊΒΕͨͱ͖ʹɺূڌͱͳΔهΛग़ͨ͢Ί • ϩάΛஷΊΔ͜ͱ͕ɺސ٬ʹఏڙ͢ΔՁͰ͋Δ • e.x.) Papertrail, mackerel.io, etc…
ྫ͑͜Μͳཧ༝ • ࣮ࢪͨ͠ࢪࡦͷԠΛݟΔͨΊ • ϢʔβʔͲ͏Ԡͨ͠ͷ͔ʁΛΔ • ϓϩμΫτΛΑΓྑ͘ϒϥογϡΞοϓ͢ΔͨΊ • ϩάΛ input
ʹɺػցֶशͰ࡞ͬͨϞσϧΛڧ͍ͯ͘͘͠ • Τϥʔ͕͋ͬͨͱ͖ͷݪҼௐࠪ • ϩά͕ͳ͍ͱݪҼΛ͑ͳ͍ • ϩάʹྲྀΕΔϝοηʔδ͔ΒɺΞϥʔτΛఆٛͯ͠ɺҟৗʹؾ͖͍ͮͨ • ා͍ਓʹ٧ΊΒΕͨͱ͖ʹɺূڌͱͳΔهΛग़ͨ͢Ί • ϩάΛஷΊΔ͜ͱ͕ɺސ٬ʹఏڙ͢ΔՁͰ͋Δ • e.x.) Papertrail, mackerel.io, etc…
ຊͷ༰ • Gunosy ͰϩάΛͲ͏ूΊͯɺͲ͏׆༻ͯ͠ ͍Δ͔ • ੳ༻్Ͱ׆༻͢Δ • ϓϩμΫτվળ༻్Ͱ׆༻͢Δ
ੳ༻్
·ͣ Gunosy ͷུ֓ਤ(ࡶver) ػցֶश σʔλஔ͖ API αʔόʔ
ੳʹ͍ؔͯ͠Δ෦͜͜ ػցֶश σʔλஔ͖ API αʔόʔ
ϩάͲ͏ྲྀΕ͍͔ͯ͘ • fluentd Ͱ S3 or BQ • ETL
or όονͰ DB ʹอଘ • Re:dash ͰՄࢹԽ API server
Re:dash ͷ DataSource όονͰੜ raw log Redshift Re:dash formated
log parquet BigQuery batch ECS RDS
Re:dash ேձͰຖݟΔ
͜͜·Ͱͷ·ͱΊ • ීஈͷϩάͷҰ࣍ஔ͖ S3 ͱ BQ • ඞཁʹԠͯ͡ DB ʹϩʔυͯ͠ɺRe:dash
ͰՄ ࢹԽ • ՄࢹԽͨ͠ Re:dash ͷओͳ KPI ேձͰڞ༗ • (ߏࣗମׂͱී௨)
ϓϩμΫτվળ༻్
Ͳ͏͍͏ϩά͕Ͳ͏ΘΕΔʁ • Ͳ͏͍͏ϩά͕ʁ • Ϣʔβʔͷߦಈϩά • هࣄͷΫϦοΫϩά • ࠂͷΠϯϓϨογϣϯ •
ͦͷଞ৭ʑ… • Ͳ͏ΘΕΔʁ • Ϣʔβʔʹ࠷దͳใΛಧ͚ΔͨΊʹੜ͔͞ΕΔ • ϢʔβʔͻͱΓͻͱΓʹରͯ͠࠷దͳهࣄΛग़͢͜ͱʹΘΕΔ • Ϣʔβʔʹରͯ͠ڵຯͷ͋Δ Push ௨Λଧͭ
શମ૾ click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2 user
vector ͍͍ײ͡ʹ͢ΔϚϯ # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -ϫʔΫϑϩʔฤ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/gunosy-personalize-digdag-workflow)
هࣄใͷείΞϦϯά click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2
user vector # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -ϫʔΫϑϩʔฤ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/gunosy-personalize-digdag-workflow)
Ϣʔβʔใͱهࣄσʔλ user data • S3 ʹཷ·͍ͬͯΔϩά͚ͩͰͳͯ͘ɺRDS ͔ΒఆظతʹɺϢʔβʔͷଐੑใΛ S3 ʹஔ͍ͯ ͍Δ
• ϢʔβʔใͳͲͷٵ্͍͛ɺόον + ఆظ࣮ߦͰཧ • digdag on ECS Ͱಈ͔͍ͯ͠Δ • ٵ্͍͛ͨσʔλͱϩάσʔλͱͷ join + ൿີͳʹ͔ɺAirflow on ECS Ͱཧ log ECS RDS ECS EMR ‟vector ‟vector
Ϣʔβʔଐੑͷߋ৽ click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2 user
vector GET ͍͍ײ͡ʹ͢ΔϚϯ
ϦΞϧλΠϜͰϢʔβʔϕΫτϧ Λߋ৽ # ϢʔβʔߦಈͷཧϞσϧͱ ߴਪનγεςϜ (https://speakerdeck.com/mathetake/yusaxing-dong-falseshu-li-moteruto-gao-su-tui-jian-sisutemu) Click log stream Push/Trim
Trigger Put Put Batch Get Batch Get Click logger Article Vectorizer Crawler Article Vector User Vector • click log client ͔Β kinesis stream ʹૹΓɺ lambda Λ௨ͯ͠ dynamo
શମ૾(࠶ܝ) click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2
user vector # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -1ΫϦοΫͰ࢝·ΔύʔιφϥΠζ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/realtime-vectorization-with-dynamodb)
͜͜·Ͱͷ·ͱΊ • ϢʔβʔͷߦಈϩάΛͱʹɺϓϩμΫτͷվળΛߦͬ ͍ͯΔ • ಛʹϢʔβʔͷΫϦοΫϩά͔ΒɺϢʔβʔϕΫτϧߋ ৽Λ΄΅ϦΞϧλΠϜߦ͍ͬͯΔ • Ұ෦ϦΞϧλΠϜͳධՁΛߦ͍ͬͯͳͯ͘ɺόονͰ ରԠ
• ͜ΕΒͷόονɺdigdag on ECS Ͱߦ͍ͬͯΔ
·ͱΊ
ӡ༻͍ͯͯ͠ࢥ͏͜ͱ • ྑ͘ѱ͘ AWS ͲͬΓਁ͔ͬͯΔ • ϑϧϚωʔδυͰɺॳظಋೖɺӡ༻ίετ͍ͷͷɺ AWS ͷ༷ʹҾͬுΒΕ ͍ͯΔ෦৭ʑ͋Δͱࢥ͏
• ϩάੳ༻్ͷ DBɺS3 bucket ֤ॴʹࢄΒͬͯΔͷͰɺΧΦεײ • ͜ΕͬͯΔ͚ͬʁɺ࣮΄ͱΜͲͬͯͳ͍ɺΈ͍ͨͷ͍͔ͭ͋͘Δͣ • ੳ༻్Ͱ͏ϩάͷεΩʔϚཧʹ͍ͭͯ·͍͠ • json Ͱอଘ͠ɺඞཁʹԠͯ͡ json Λղͨ͠Ұ࣌ςʔϒϧΛ࡞ͬͯରॲ • σʔλࣗ༝ʹ৮ΕΔҰํɺύʔςΟγϣϯࢦఆͤͣɺϑϧεΩϟϯΫΤϦ࣮ߦ͠ଠ ͕ఆظతʹग़ݱ͢Δ • ϚωʔతͳҙຯͰ͏͔ͬΓ௧खΛ͏ͷආ͚͍͕ͨɺ͔ͩΒͱ͍ͬͯΞΫηεݖ Λୣ͏ͷҧ͏ؾ͕͢Δ
·ͱΊ • Gunosy Ͱѻ͍ͬͯΔϩάΛɺʮੳ༻్ʯʮϓϩμΫτվળ༻్ʯ͚ ͯͦΕͧΕ͝հ͠·ͨ͠ • ੳ༻్ͰɺηΦϦʔ௨Γͷߏ • S3 ->
தؒσʔλετϨʔδ -> Re:dash • ओཁͳ KPI ɺRe:dash ΛͬͯάϥϑԽ͠ɺேձͰຖڞ༗͍ͯ͠Δ • ϓϩμΫτվળ༻్Ͱɺόον + ϦΞϧλΠϜͰϩάΛ͍ͬͯ·͢ • AWS ʹେ͖͘ॿ͚ΒΕ͍ͯ·͢
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠