Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Gunosy のログ収集基盤
mgi166
June 23, 2018
1
610
Gunosy のログ収集基盤
mgi166
June 23, 2018
Tweet
Share
More Decks by mgi166
See All by mgi166
realtime_rails
mgi166
0
35
Music as Code
mgi166
0
490
Featured
See All Featured
Building Applications with DynamoDB
mza
83
4.7k
Support Driven Design
roundedbygravity
86
8.5k
The Brand Is Dead. Long Live the Brand.
mthomps
46
2.7k
Fireside Chat
paigeccino
11
1.3k
Why You Should Never Use an ORM
jnunemaker
PRO
47
7.2k
Building Adaptive Systems
keathley
25
1.1k
Design by the Numbers
sachag
271
17k
Scaling GitHub
holman
451
140k
Learning to Love Humans: Emotional Interface Design
aarron
261
37k
Designing the Hi-DPI Web
ddemaree
272
32k
WebSockets: Embracing the real-time Web
robhawkes
57
5.1k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
19
1.4k
Transcript
Gunosy ͷϩάऩूج൫ Architecture Night #1 ໜେເ@mgi166
ࣗݾհ • גࣜձࣾ Gunosy ։ൃɾӡ༻ਪਐ෦ • ໜେເ • ݩ Rails
ΤϯδχΞ • ࠓ Gunosy ͷ AWS पΓɺΠϯϑϥ पΓΛ୲
ຊͷ༰ • Gunosy ͷϩάपΓΛத৺ͱͨ͠ɺΠϯϑϥ ΞʔΩςΫνϟͷࣄྫΛհ͠·͢
ϩάΛूΊΔ
ͳΜͰϩάΛूΊΔͷʁ
ྫ͑͜Μͳཧ༝ • ࣮ࢪͨ͠ࢪࡦͷԠΛݟΔͨΊ • ϢʔβʔͲ͏Ԡͨ͠ͷ͔ʁΛΔ • ϓϩμΫτΛΑΓྑ͘ϒϥογϡΞοϓ͢ΔͨΊ • ϩάΛ input
ʹɺػցֶशͰ࡞ͬͨϞσϧΛڧ͍ͯ͘͘͠ • Τϥʔ͕͋ͬͨͱ͖ͷݪҼௐࠪ • ϩά͕ͳ͍ͱݪҼΛ͑ͳ͍ • ϩάʹྲྀΕΔϝοηʔδ͔ΒɺΞϥʔτΛఆٛͯ͠ɺҟৗʹؾ͖͍ͮͨ • ා͍ਓʹ٧ΊΒΕͨͱ͖ʹɺূڌͱͳΔهΛग़ͨ͢Ί • ϩάΛஷΊΔ͜ͱ͕ɺސ٬ʹఏڙ͢ΔՁͰ͋Δ • e.x.) Papertrail, mackerel.io, etc…
ྫ͑͜Μͳཧ༝ • ࣮ࢪͨ͠ࢪࡦͷԠΛݟΔͨΊ • ϢʔβʔͲ͏Ԡͨ͠ͷ͔ʁΛΔ • ϓϩμΫτΛΑΓྑ͘ϒϥογϡΞοϓ͢ΔͨΊ • ϩάΛ input
ʹɺػցֶशͰ࡞ͬͨϞσϧΛڧ͍ͯ͘͘͠ • Τϥʔ͕͋ͬͨͱ͖ͷݪҼௐࠪ • ϩά͕ͳ͍ͱݪҼΛ͑ͳ͍ • ϩάʹྲྀΕΔϝοηʔδ͔ΒɺΞϥʔτΛఆٛͯ͠ɺҟৗʹؾ͖͍ͮͨ • ා͍ਓʹ٧ΊΒΕͨͱ͖ʹɺূڌͱͳΔهΛग़ͨ͢Ί • ϩάΛஷΊΔ͜ͱ͕ɺސ٬ʹఏڙ͢ΔՁͰ͋Δ • e.x.) Papertrail, mackerel.io, etc…
ຊͷ༰ • Gunosy ͰϩάΛͲ͏ूΊͯɺͲ͏׆༻ͯ͠ ͍Δ͔ • ੳ༻్Ͱ׆༻͢Δ • ϓϩμΫτվળ༻్Ͱ׆༻͢Δ
ੳ༻్
·ͣ Gunosy ͷུ֓ਤ(ࡶver) ػցֶश σʔλஔ͖ API αʔόʔ
ੳʹ͍ؔͯ͠Δ෦͜͜ ػցֶश σʔλஔ͖ API αʔόʔ
ϩάͲ͏ྲྀΕ͍͔ͯ͘ • fluentd Ͱ S3 or BQ • ETL
or όονͰ DB ʹอଘ • Re:dash ͰՄࢹԽ API server
Re:dash ͷ DataSource όονͰੜ raw log Redshift Re:dash formated
log parquet BigQuery batch ECS RDS
Re:dash ேձͰຖݟΔ
͜͜·Ͱͷ·ͱΊ • ීஈͷϩάͷҰ࣍ஔ͖ S3 ͱ BQ • ඞཁʹԠͯ͡ DB ʹϩʔυͯ͠ɺRe:dash
ͰՄ ࢹԽ • ՄࢹԽͨ͠ Re:dash ͷओͳ KPI ேձͰڞ༗ • (ߏࣗମׂͱී௨)
ϓϩμΫτվળ༻్
Ͳ͏͍͏ϩά͕Ͳ͏ΘΕΔʁ • Ͳ͏͍͏ϩά͕ʁ • Ϣʔβʔͷߦಈϩά • هࣄͷΫϦοΫϩά • ࠂͷΠϯϓϨογϣϯ •
ͦͷଞ৭ʑ… • Ͳ͏ΘΕΔʁ • Ϣʔβʔʹ࠷దͳใΛಧ͚ΔͨΊʹੜ͔͞ΕΔ • ϢʔβʔͻͱΓͻͱΓʹରͯ͠࠷దͳهࣄΛग़͢͜ͱʹΘΕΔ • Ϣʔβʔʹରͯ͠ڵຯͷ͋Δ Push ௨Λଧͭ
શମ૾ click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2 user
vector ͍͍ײ͡ʹ͢ΔϚϯ # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -ϫʔΫϑϩʔฤ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/gunosy-personalize-digdag-workflow)
هࣄใͷείΞϦϯά click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2
user vector # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -ϫʔΫϑϩʔฤ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/gunosy-personalize-digdag-workflow)
Ϣʔβʔใͱهࣄσʔλ user data • S3 ʹཷ·͍ͬͯΔϩά͚ͩͰͳͯ͘ɺRDS ͔ΒఆظతʹɺϢʔβʔͷଐੑใΛ S3 ʹஔ͍ͯ ͍Δ
• ϢʔβʔใͳͲͷٵ্͍͛ɺόον + ఆظ࣮ߦͰཧ • digdag on ECS Ͱಈ͔͍ͯ͠Δ • ٵ্͍͛ͨσʔλͱϩάσʔλͱͷ join + ൿີͳʹ͔ɺAirflow on ECS Ͱཧ log ECS RDS ECS EMR ‟vector ‟vector
Ϣʔβʔଐੑͷߋ৽ click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2 user
vector GET ͍͍ײ͡ʹ͢ΔϚϯ
ϦΞϧλΠϜͰϢʔβʔϕΫτϧ Λߋ৽ # ϢʔβʔߦಈͷཧϞσϧͱ ߴਪનγεςϜ (https://speakerdeck.com/mathetake/yusaxing-dong-falseshu-li-moteruto-gao-su-tui-jian-sisutemu) Click log stream Push/Trim
Trigger Put Put Batch Get Batch Get Click logger Article Vectorizer Crawler Article Vector User Vector • click log client ͔Β kinesis stream ʹૹΓɺ lambda Λ௨ͯ͠ dynamo
શମ૾(࠶ܝ) click/imp log log API server ‟vector ͍͍ײ͡ʹ͢ΔϚϯ ͍͍ײ͡ʹ͢ΔϚϯ ‟vector2
user vector # GunosyͷύʔιφϥΠζΛࢧ͑Δٕज़ -1ΫϦοΫͰ࢝·ΔύʔιφϥΠζ- - Gunosy Tech Blog (https://tech.gunosy.io/entry/realtime-vectorization-with-dynamodb)
͜͜·Ͱͷ·ͱΊ • ϢʔβʔͷߦಈϩάΛͱʹɺϓϩμΫτͷվળΛߦͬ ͍ͯΔ • ಛʹϢʔβʔͷΫϦοΫϩά͔ΒɺϢʔβʔϕΫτϧߋ ৽Λ΄΅ϦΞϧλΠϜߦ͍ͬͯΔ • Ұ෦ϦΞϧλΠϜͳධՁΛߦ͍ͬͯͳͯ͘ɺόονͰ ରԠ
• ͜ΕΒͷόονɺdigdag on ECS Ͱߦ͍ͬͯΔ
·ͱΊ
ӡ༻͍ͯͯ͠ࢥ͏͜ͱ • ྑ͘ѱ͘ AWS ͲͬΓਁ͔ͬͯΔ • ϑϧϚωʔδυͰɺॳظಋೖɺӡ༻ίετ͍ͷͷɺ AWS ͷ༷ʹҾͬுΒΕ ͍ͯΔ෦৭ʑ͋Δͱࢥ͏
• ϩάੳ༻్ͷ DBɺS3 bucket ֤ॴʹࢄΒͬͯΔͷͰɺΧΦεײ • ͜ΕͬͯΔ͚ͬʁɺ࣮΄ͱΜͲͬͯͳ͍ɺΈ͍ͨͷ͍͔ͭ͋͘Δͣ • ੳ༻్Ͱ͏ϩάͷεΩʔϚཧʹ͍ͭͯ·͍͠ • json Ͱอଘ͠ɺඞཁʹԠͯ͡ json Λղͨ͠Ұ࣌ςʔϒϧΛ࡞ͬͯରॲ • σʔλࣗ༝ʹ৮ΕΔҰํɺύʔςΟγϣϯࢦఆͤͣɺϑϧεΩϟϯΫΤϦ࣮ߦ͠ଠ ͕ఆظతʹग़ݱ͢Δ • ϚωʔతͳҙຯͰ͏͔ͬΓ௧खΛ͏ͷආ͚͍͕ͨɺ͔ͩΒͱ͍ͬͯΞΫηεݖ Λୣ͏ͷҧ͏ؾ͕͢Δ
·ͱΊ • Gunosy Ͱѻ͍ͬͯΔϩάΛɺʮੳ༻్ʯʮϓϩμΫτվળ༻్ʯ͚ ͯͦΕͧΕ͝հ͠·ͨ͠ • ੳ༻్ͰɺηΦϦʔ௨Γͷߏ • S3 ->
தؒσʔλετϨʔδ -> Re:dash • ओཁͳ KPI ɺRe:dash ΛͬͯάϥϑԽ͠ɺேձͰຖڞ༗͍ͯ͠Δ • ϓϩμΫτվળ༻్Ͱɺόον + ϦΞϧλΠϜͰϩάΛ͍ͬͯ·͢ • AWS ʹେ͖͘ॿ͚ΒΕ͍ͯ·͢
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠