Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
マルチクラウドで構築する大規模解析サービス/戦略とポイントについて/
Search
tik-son
February 15, 2018
Technology
9
27k
マルチクラウドで構築する大規模解析サービス/戦略とポイントについて/
デブサミ2018でお話したこと。
http://event.shoeisha.jp/devsumi/20180215/session/1621/
tik-son
February 15, 2018
Tweet
Share
More Decks by tik-son
See All by tik-son
Microsoft の AI 技術 をベースに構成される Craft Copilot
tikson
0
310
KARTE CraftにおけるLLM活用~Craft Functions Copilot~
tikson
1
1.4k
AWS とKARTE で作る User State Driven System / user-state-driven-system-with-aws-and-karte
tikson
0
2.3k
User State Driven System with Amazon EventBridge & KARTE
tikson
1
3.6k
PLAID’s Journey with multi cloud
tikson
1
3.5k
KARTEを支えるAutoScaling~SpotInstance編
tikson
0
1.6k
Other Decks in Technology
See All in Technology
Deep dive into Nuxt Server Components
wattanx
1
1.5k
dbt-coreで実現するCore DataMartsのデータモデリング〜dbt編〜 / Core DataMarts Modeling with dbt-core
i125
0
180
S3上のログを分析したいだけなのに
mitsuo_sys
0
130
0x5F3759DF
ykozw
0
370
WebRTC と Wasm の関係を振り返ってみた
tetter27
0
190
VueとViteで作るUIコンポーネントライブラリ ~デザインシステムとプロダクトの理想的な分離を目指して~ / 20241019_cloudsign_VueFesJapan2024_1
bengo4com
8
4.9k
WebRTC と AI の組み合わせ
tnoho
0
390
Introduction to Indistinguishability/Ideal Obfuscation (iO)
sorasuegami
0
110
管理画面とユーザー機能の調和を取り戻す!~クエリパフォーマンス改善の成功物語~ / Restore harmony between administrative and user functions!
minisera
1
360
Amplify Gen 2ではじめる 生成AIアプリ開発入門
tsukuboshi
0
410
なぜ Rack を理解すべきかプレトーク / Why should you understand Rack - Pre-talk
hogelog
0
250
新入社員 オンボーディング改善プロジェクト - シンプルな仕組みで変革のきっかけを
enpipi
0
520
Featured
See All Featured
Bash Introduction
62gerente
608
210k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
504
140k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
364
22k
A Philosophy of Restraint
colly
203
16k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
20k
How STYLIGHT went responsive
nonsquared
95
5.1k
The Art of Programming - Codeland 2020
erikaheidi
51
13k
How to Think Like a Performance Engineer
csswizardry
18
1k
A designer walks into a library…
pauljervisheath
202
24k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
37
1.8k
Typedesign – Prime Four
hannesfritz
39
2.4k
Ruby is Unlike a Banana
tanoku
96
11k
Transcript
© 2018 PLAID,Inc. ϚϧνΫϥυͰߏங͢ΔେنղੳαʔϏε ઓུͱϙΠϯτʹ͍ͭͯ 'FCSVBSZ 1-"*% *OD /BPIJLP5BLFNVSB
© 2018 PLAID,Inc. 1SPpMF 2011ɹಉࢤࣾେֶֶ෦ଔۀ 2011͔Β2014·ͰɺNECʹͯΫϥυαʔϏεͷ্ཱͪ͛ʹैࣄ σʔληϯλʔͷϑΝγϦςΟ͔ΒωοτϫʔΫ/αʔόͷཧϨΠϠʔ͔ΒԾ ϨΠϠʔ·Ͱͷઃܭ/ߏங/ӡ༻Λ୲ 2014ɹϓϨΠυʹೖࣾ େنղੳαʔϏεKARTEͷཪଆΛΠϯϑϥ͔ΒόοΫΤϯυ·Ͱ෯͘୲
ଜɹঘ
© 2018 PLAID,Inc. 8IBU`T1-"*%
© 2018 PLAID,Inc. 4IPFT" 4IPFT# d d 4)011*/($"35
๚தͷϢʔβʔΛ ϦΞϧλΠϜղੳ ݸʑͷϢʔβʔʹ߹Θͤͨ ίϛϡχέʔγϣϯΛఏڙ 8IBU`T,"35&
© 2018 PLAID,Inc. 8IBU`T,"35& ๚தͷϢʔβʔΛϦΞϧλΠϜʹղੳ
© 2018 PLAID,Inc. 8IBU`T,"35& ϦΞϧλΠϜʹղੳͨ݁͠ՌΛͱʹ༷ʑͳΞΫγϣϯΛग़͚͠Δ͜ͱ͕Ͱ͖Δ
© 2018 PLAID,Inc. ಋೖΫϥΠΞϯτ ղੳUU DBͷ༰ྔ τϥοΩϯάΠϕϯτ TB events /
sec 6000 1430+ 500 ղੳ࣌ؒ ඵ 0.x UU 12.5ԯ ྲྀ௨ֹۚ ԁ / 5000ԯ 4UBUTPG,"35& ΫϥΠΞϯτ
© 2018 PLAID,Inc. Event Cost $PTUTPG,"35& 2લ ? ӡ༻ʹܞΘΔ Engineer
ݱࡏ
© 2018 PLAID,Inc. Event Cost *OGSBTUSBUFHZPG,"35& Ͱ͖Δ͚ͩγϯάϧΫϥυͰ γϯϓϧͳߏΛΉ ϓϩόΠμʹͱΒΘΕͣ ϚϧνΫϥυʹߏΛΉ
ઓུͷมߋ 2લ ӡ༻ʹܞΘΔ Engineer ݱࡏ
© 2018 PLAID,Inc. ϚϧνΫϥυΛ͍͜ͳͨ͢Ίͷ4ͭͷϙΠϯτ
© 2018 PLAID,Inc. શମߏ
© 2018 PLAID,Inc. 1. ֤ϓϩόΠμʔͷ͍͍ͱ͜औΓΛ͢Δ
© 2018 PLAID,Inc. ༷ʑͳϓϩόΠμ/αʔϏε͕ଘࡏ - ૯߹తͳϓϩόΠμɺҰಛԽܕͷϓϩόΠμ - ϓϩόΠμͷ࣭ͷ্ ϓϩόΠμ/αʔϏεʹಘҙ͕͋Δ -
AWS ͱ GCP ಘҙΛ͍͍ͱ͜ͲΓ͢ΔͱγϯάϧϓϩόΠμͰͷγεςϜΑΓڧ͘ͳͬͨ
© 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε
ղੳσʔλอଘ ղੳσʔλͷऔಘ DB Track
© 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε
ղੳσʔλอଘ ղੳσʔλͷऔಘ Track DB
© 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε
ղੳσʔλอଘ - ϨΠςϯγʔ͚ͩͲϑϧϚωʔδυͰͳ͍ - ಥવͷϝϯςφϯε - ύϑΥʔϚϯεͷݶք ղੳσʔλͷऔಘ Track DB
© 2018 PLAID,Inc. - DBͷ༷͕KARTEͱ૬ੑ͕ѱ͔ͬͨ - ΧλϩάεϖοΫ௨ΓʹύϑΥʔϚϯεग़ͳ͍ - ͏΄ͲΤΫεϙωϯγϟϧʹίετ͕͔͔Δ ϦΞϧλΠϜղੳͷྲྀΕ
ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ Ωϡʔ Analyze ղੳσʔλͷऔಘ DB Track - ϨΠςϯγʔ͚ͩͲϑϧϚωʔδυͰͳ͍ - ಥવͷϝϯςφϯε - ύϑΥʔϚϯεͷݶք
© 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε
ղੳσʔλอଘ ղੳσʔλͷऔಘ Track DB
© 2018 PLAID,Inc. redis enterpriseΛΫϥυͰఏڙ͍ͯ͠ΔαʔϏε ಛ - ߴ͍εέʔϥϏϦςΟ - ্༷σʔλ༰ྔɺίϯϐϡʔλϦιʔεͷ੍ݶ͕ͳ͘ɺෛՙʹԠͯࣗ͡ಈͰεέʔϧ
- ϨΠςϯγɺߴύϑΥʔϚϯε - Latency 0.1ms - ReadOps 3kOps/sec, WriteOps 11kOps/sec - ϑϧϚωʔδυαʔϏε - Δ͜ͱॳظઃఆͷΈ - ߴՄ༻ੑ - ೋؒେ͖ͳো͕ͳ͍ - DBͷσʔλྔʹΑΔैྔ՝ۚɻωοτϫʔΫίετͰ՝ۚ͞Εͳ͍ɻ ϨΠςϯγɺεέʔϥϒϧͳΩϡʔΛίετͰར༻Ͱ͖ΔΑ͏ʹ SFEJTDMPVE
© 2018 PLAID,Inc. Analyze ϦΞϧλΠϜղੳͷྲྀΕ ղੳσʔλͷऔಘ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε
ղੳσʔλอଘ Track
© 2018 PLAID,Inc. GoogleͷαʔϏεͰར༻͍ͯ͠Δٕज़ΛΫϥυͰఏڙ ಛ - ߴ͍εέʔϥϏϦςΟ - ্༷σʔλ༰ྔͷ੍ݶ͕ͳ͍ɻ140TB -
ύϑΥʔϚϯεΞοϓͷͨΊͷεέʔϧΞοϓखಈ͕ͩɺμϯλΠϜͳ͘ॠ࣌ʹ࣮ߦ - ߴՄ༻ੑ - ೋؒେ͖ͳো͕ͳ͍ - ϨΠςϯγɺߴύϑΥʔϚϯε - Latency -20ms - Readεϧʔϓοτ 1.3GB/sec, Writeεϧʔϓοτ 2.3MB/sec - ϑϧϚωʔδυαʔϏε $MPVE#JHUBCMF ΧλϩάεϖοΫҎ্ͷߴ͍ύϑΥʔϚϯε
© 2018 PLAID,Inc. Analyze - ύϑΥʔϚϯε/҆ఆੑ্͕ - αʔϏεར༻ίετ͕30%ݮ - ӡ༻ʹ͔͚Δ࣌ؒͷݮ
ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ ղੳσʔλͷऔಘ Track
© 2018 PLAID,Inc. 2. ϓϩόΠμͷোΛผϓϩόΠμͰΧόʔ͢Δ
© 2018 PLAID,Inc. ͲΕ͚ͩૉΒ͍͠αʔϏεͰোى͖Δ - 2017/3ͷS3ͷେنো(https://aws.amazon.com/jp/message/41926/) - 3࣌ؒɺUS-EAST-1ͷS3͕͑ͳ͘ͳΓɺଟ͘ͷαʔϏε͕μϯ - S399.99ͷSLA
োͷதʹγϯάϧΫϥυͰରԠ͠Α͏͕ͳ͍ͷଘࡏ͢Δ - ωοτϫʔΫͳͲڞ௨෦ͷো - Ϧʔδϣϯ/κʔϯͷ֓೦Λ࣋ͨͳ͍ϚωʔδυαʔϏεͷো
© 2018 PLAID,Inc. KARTE͓٬༷ͷαΠτͷҰ෦ʹͳΔͷͰՄ༻ੑͱͯॏཁ - ྫ͑chat ϚϧνΫϥυʹΑͬͯ͜ͷΑ͏ͳোΛ্ख͑͘Δ͜ͱ͕Ͱ͖͍ͯΔ
© 2018 PLAID,Inc. https://status.cloud.google.com/incident/compute/16020#5177473553661952 2࣌ؒɺόοΫΤϯυͷεςʔλεʹؔͳ͘ ϦΫΤετͷ1/3Ͱ502ͷϨεϙϯεΛฦͯ͠͠·ͬͨͱ͍͏LBͷো
© 2018 PLAID,Inc.
© 2018 PLAID,Inc. ߦಈϩάऩूपΓͷߏ ීஈRoute53ͷTrafficFlowͰ GCP : AWS = 8
: 2 Ͱ໊લղܾ
© 2018 PLAID,Inc. ߦಈϩάऩूपΓͷߏ ো࣌Route53ͷTrafficFlowͰGCP : AWS = 0 :
10 Ͱ໊લղܾ
© 2018 PLAID,Inc. ֤ϓϩόΠμʔࣅͨΑ͏ͳαʔϏεΛఏڙ͍ͯ͠ΔͷͰɺ༷ʑͳॴͰ࣮ݱՄೳ શͯͰ͜ͷߏΛͱΔͷ͍͠ͷͰɺಋೖ෦ʹ͍ͭͯݟۃΊ͕ඞཁ - γεςϜͱͯ͠ͷॏཁ͞ - ࣮ݱͷқ ཁ݅ʹԠͯ͡ɺͲ͜·Ͱ࡞ΓࠐΉ͔ߟ͑Δ
- ଟগίεύ͕ѱͯ͘ಈ͚͍͍ - खಈͰΓସ͑Ε͍͍ ϦΞϧλΠϜղੳʹؔΘΔॏཁͳ෦ ग़དྷΔ͚ͩෳͷϓϩόΠμʔͰಈ͔ͤΔΑ͏ʹ͍ͯ͠Δ GCS S3 CloudCDN CloudFront
© 2018 PLAID,Inc. 3. ϓϩόΠμ·ͨ͗ͷ௨৴ʹҙ͢Δ
© 2018 PLAID,Inc. 1. ௨৴ίετ͕ඦഒʹͳͬͨ 2. ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠Լͨ͠
© 2018 PLAID,Inc. ௨৴ίετ͕ඦഒʹͳͬͨ - ϦΞϧλΠϜղੳͰͷBigtableͱEC2ͷ௨৴Ͱର - BigtableͷසൟͳRead/WriteʹΑΓίετ૿ NBJO
© 2018 PLAID,Inc. ௨৴ίετ͕ඦഒʹͳͬͨ ผϓϩόΠμʔͷαʔϏεͷ௨৴ͷίετʹҙ - ϦΞϧλΠϜղੳͰͷBigtableͱEC2ͷ௨৴Ͱର - BigtableͷසൟͳRead/WriteʹΑΓίετ૿ -
Bigtableಉ͡zone͔Βͷ௨৴ίετ͔͔Βͳ͍ - Bigtableͱಉ͡zoneʹGCEΛߏங - ෭࣍తͳޮՌͱͯ͠ɺ҆ఆੑ/Latency্͕ - AWSGCPͷόοΫΞοϓܥʹ NBJO TUBOECZ NBJO
© 2018 PLAID,Inc. ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠Լͨ͠ - GCPͱAWSଓͷͨΊͷVPN෦ - GCE͔ΒMongo(EC2)ͷେྔͷRead - VPNͷଳҬ͕ανΓϘτϧωοΫ
- 1.5Gps/1VPNτϯωϧ(GCP) - ଳҬʹಛʹ੍ݶͳ͠(AWS)
© 2018 PLAID,Inc. ผϓϩόΠμͷ71/ܦ༝ͷ௨৴ͷଳҬʹҙ ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠Լͨ͠ - GCPʹReadReplicaΛ࡞ΓɺVPNͷ௨৴ྔݮ - ෳVPNτϯωϧʹIPϧʔςΟϯάͰෛՙࢄ -
ଳҬΛࢹ͠ɺඞཁʹԠͯ͡τϯωϧΛՃ - GCPͱAWSଓͷͨΊͷVPN෦ - GCE͔ΒMongo(EC2)ͷେྔͷRead - VPNͷଳҬ͕ανΓϘτϧωοΫ - 1.5Gps/1VPNτϯωϧ(GCP) - ଳҬʹಛʹ੍ݶͳ͠(AWS) IUUQUFDIQMBJEDPKQBXT@HDQ@NVMUJDMPVE@
© 2018 PLAID,Inc. ผϓϩόΠμʔͷαʔϏεͷ௨৴ͷίετ ผϓϩόΠμʔͷVPNܦ༝ͷ௨৴ͷଳҬ NBJO TUBOECZ
© 2018 PLAID,Inc. 4. ෳͷϓϩόΠμʔΛ౷ҰతͳΠϯλʔϑΣʔεͰӡ༻
© 2018 PLAID,Inc. ΫϥυαʔϏεͱ͍ͬͯࣗͨͪͰ໘Λݟͳ͍ͱ͍͚ͳ͍෦͕͋Δ - ߏཧ - ࢹ - ΞϓϦέʔγϣϯͷσϓϩΠ
ͳͲ ӡ༻Λิॿ͢ΔαʔϏε֤ϓϩόΠμͰͦΕͧΕಠࣗͷϑΥʔϚοτ - ྫ͑ߏཧ DeploymentManager CloudFormation
© 2018 PLAID,Inc. DeploymentManager
© 2018 PLAID,Inc. DeploymentManager CloudFormation
© 2018 PLAID,Inc. ౷ҰతͳΠϯλʔϑΣʔε
© 2018 PLAID,Inc. ࣗͨͪͰ։ൃʁ - ϓϩόΠμ͕૿͑ΔͭͲେ͖ͳ։ൃ͕ඞཁ - ରߏཧɺΞϓϦέʔγϣϯͷσϓϩΠɺࢹͳͲ - ίετେͰϓϩόΠμΛؾܰʹ૿ͤͳ͍
ಉ͡՝ײΛ๊͍͑ͯΔਓ݁ߏ͍Δ OSSɺαʔυύʔςΟͷαʔϏεΛ͏·͘ར༻͢Δ ౷ҰతͳΠϯλʔϑΣʔεΛͲ͏࡞Δ͔ʁ
© 2018 PLAID,Inc. ߏཧ ΞϓϦέʔγϣϯͷσϓϩΠ ࢹ
© 2018 PLAID,Inc. ϓϩϏδϣϯ WerckerΛτϦΨʔ ߏఆٛϑΝΠϧΛ PR & Merge ߏཧͷΈ
© 2018 PLAID,Inc. σϓϩΠఆٛϑΝΠϧ ΛPR & Merge WerckerΛτϦΨʔ Roer ΞϓϦέʔγϣϯͷσϓϩΠͷΈ
σϓϩΠఆٛΛϓϩϏδϣϯ
© 2018 PLAID,Inc. ఆٛࡁΈͷσϓϩΠͷapiΛୟ͘ σϓϩΠͷ࣮ߦ deployίϚϯυ Roer ΞϓϦέʔγϣϯͷσϓϩΠͷΈ σϓϩΠఆٛϑΝΠϧ ΛPR
& Merge WerckerΛτϦΨʔ σϓϩΠఆٛΛϓϩϏδϣϯ
© 2018 PLAID,Inc. ALERT MetricsͷӾཡ ి ௨ ALERT/Dashboardͷఆٛ Metricsͷू ࢹͷΈ
© 2018 PLAID,Inc. ߏཧ ΞϓϦέʔγϣϯͷσϓϩΠ ࢹ
© 2018 PLAID,Inc. ·ͱΊ
© 2018 PLAID,Inc. αʔϏεͷͱγϯάϧΫϥυͷݶք͔Β ʮϓϩόΠμʹͱΒΘΕͣϚϧνΫϥυʹߏΛΉʯઓུʹ ϙΠϯτ4ͭ 1.ϓϩόΠμʔͷ͍͍ͱ͜औΓΛ͢Δ 2.ϓϩόΠμͷোΛผϓϩόΠμͰΧόʔ͢Δ 3.ϓϩόΠμ·ͨ͗ͷ௨৴ʹҙ͢Δ 4.ෳͷϓϩόΠμʔΛ౷ҰతͳΠϯλʔϑΣʔεͰӡ༻͢Δ
© 2018 PLAID,Inc. ࠷ޙʹ
© 2018 PLAID,Inc. IUUQTDPEF[JOFKQBSUJDMFDPSOFS
© 2018 PLAID,Inc. IUUQUFDIQMBJEDPKQBXT@HDQ@NVMUJDMPVE@
© 2018 PLAID,Inc. IUUQTXXXXBOUFEMZDPNDPNQBOJFTQMBJE