マルチクラウドで構築する大規模解析サービス/戦略とポイントについて/

デブサミ2018でお話したこと。
http://event.shoeisha.jp/devsumi/20180215/session/1621/

Transcript

1. © 2018 PLAID,Inc. ϚϧνΫϥ΢υͰߏங͢Δେن໛ղੳαʔϏε
   ઓུͱϙΠϯτʹ͍ͭͯ
   'FCSVBSZ
   
   
   1-"*%
   *OD
   
   /BPIJLP
   5BLFNVSB
2. © 2018 PLAID,Inc. 1SPpMF 2011೥ɹಉࢤࣾେֶ޻ֶ෦ଔۀ 2011೥͔Β2014೥·ͰɺNECʹͯΫϥ΢υαʔϏεͷ্ཱͪ͛ʹैࣄ σʔληϯλʔͷϑΝγϦςΟ͔ΒωοτϫʔΫ/αʔόͷ෺ཧϨΠϠʔ͔ΒԾ ૝ϨΠϠʔ·Ͱͷઃܭ/ߏங/ӡ༻Λ୲౰ 2014೥ɹϓϨΠυʹೖࣾ େن໛ղੳαʔϏεKARTEͷཪଆΛΠϯϑϥ͔ΒόοΫΤϯυ·Ͱ෯޿͘୲౰ ஛ଜɹঘ඙
3. © 2018 PLAID,Inc. 8IBU`T
   1-"*%
   
4. © 2018 PLAID,Inc. 4IPFT" 4IPFT# d  d  4)011*/(
   $"35 ๚໰தͷϢʔβʔΛ ϦΞϧλΠϜղੳ ݸʑͷϢʔβʔʹ߹Θͤͨ ίϛϡχέʔγϣϯΛఏڙ 8IBU`T
   ,"35&
   
5. © 2018 PLAID,Inc. 8IBU`T
   ,"35&
   ๚໰தͷϢʔβʔΛϦΞϧλΠϜʹղੳ
6. © 2018 PLAID,Inc. 8IBU`T
   ,"35&
   ϦΞϧλΠϜʹղੳͨ݁͠ՌΛ΋ͱʹ༷ʑͳΞΫγϣϯΛग़͠෼͚Δ͜ͱ͕Ͱ͖Δ
7. © 2018 PLAID,Inc. ಋೖΫϥΠΞϯτ ղੳUU DBͷ༰ྔ τϥοΩϯάΠϕϯτ TB events / sec 6000 1430+ 500 ղੳ࣌ؒ ඵ 0.x UU 12.5ԯ ྲྀ௨ֹۚ ԁ / ೥ 5000ԯ 4UBUT
   PG
   ,"35& ΫϥΠΞϯτ
8. © 2018 PLAID,Inc. Event Cost $PTUT
   PG
   ,"35& 2೥લ ? ӡ༻ʹܞΘΔ Engineer ݱࡏ
9. © 2018 PLAID,Inc. Event Cost *OGSB
   TUSBUFHZ
   PG
   ,"35& Ͱ͖Δ͚ͩγϯάϧΫϥ΢υͰ γϯϓϧͳߏ੒Λ૊Ή ϓϩόΠμʹͱΒΘΕͣ ϚϧνΫϥ΢υʹߏ੒Λ૊Ή ઓུͷมߋ 2೥લ ӡ༻ʹܞΘΔ Engineer ݱࡏ
10. © 2018 PLAID,Inc. ϚϧνΫϥ΢υΛ࢖͍͜ͳͨ͢Ίͷ4ͭͷϙΠϯτ
11. © 2018 PLAID,Inc. શମߏ੒
12. © 2018 PLAID,Inc. 1. ֤ϓϩόΠμʔͷ͍͍ͱ͜औΓΛ͢Δ
13. © 2018 PLAID,Inc. ༷ʑͳϓϩόΠμ/αʔϏε͕ଘࡏ - ૯߹తͳϓϩόΠμɺҰ఺ಛԽܕͷϓϩόΠμ - ϓϩόΠμͷ࣭ͷ޲্ ϓϩόΠμ/αʔϏεʹ͸ಘҙ෼໺͕͋Δ - AWS ͱ GCP ಘҙ෼໺Λ͍͍ͱ͜ͲΓ͢ΔͱγϯάϧϓϩόΠμͰͷγεςϜΑΓڧ͘ͳͬͨ
14. © 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ ղੳσʔλͷऔಘ DB Track
15. © 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ ղੳσʔλͷऔಘ Track DB
16. © 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ - ௿ϨΠςϯγʔ͚ͩͲϑϧϚωʔδυͰ͸ͳ͍ - ಥવͷϝϯςφϯε - ύϑΥʔϚϯεͷݶք ղੳσʔλͷऔಘ Track DB
17. © 2018 PLAID,Inc. - DBͷ࢓༷͕KARTEͱ૬ੑ͕ѱ͔ͬͨ - ΧλϩάεϖοΫ௨ΓʹύϑΥʔϚϯεग़ͳ͍ - ࢖͏΄ͲΤΫεϙωϯγϟϧʹίετ͕͔͔Δ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ Ωϡʔ Analyze ղੳσʔλͷऔಘ DB Track - ௿ϨΠςϯγʔ͚ͩͲϑϧϚωʔδυͰ͸ͳ͍ - ಥવͷϝϯςφϯε - ύϑΥʔϚϯεͷݶք
18. © 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ ղੳσʔλͷऔಘ Track DB
19. © 2018 PLAID,Inc. redis enterpriseΛΫϥ΢υͰఏڙ͍ͯ͠ΔαʔϏε ಛ௃ - ߴ͍εέʔϥϏϦςΟ - ࢓্༷σʔλ༰ྔɺίϯϐϡʔλϦιʔεͷ੍ݶ͕ͳ͘ɺෛՙʹԠͯࣗ͡ಈͰεέʔϧ - ௿ϨΠςϯγɺߴύϑΥʔϚϯε - Latency 0.1ms - ReadOps 3kOps/sec, WriteOps 11kOps/sec - ϑϧϚωʔδυαʔϏε - ΍Δ͜ͱ͸ॳظઃఆͷΈ - ߴՄ༻ੑ - ೋ೥ؒେ͖ͳো֐͕ͳ͍ - DBͷσʔλྔʹΑΔैྔ՝ۚɻωοτϫʔΫίετͰ͸՝ۚ͞Εͳ͍ɻ ௿ϨΠςϯγɺεέʔϥϒϧͳΩϡʔΛ௿ίετͰར༻Ͱ͖ΔΑ͏ʹ SFEJT
    DMPVE
20. © 2018 PLAID,Inc. Analyze ϦΞϧλΠϜղੳͷྲྀΕ ղੳσʔλͷऔಘ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ Track
21. © 2018 PLAID,Inc. GoogleͷαʔϏε಺Ͱར༻͍ͯ͠Δٕज़ΛΫϥ΢υͰఏڙ ಛ௃ - ߴ͍εέʔϥϏϦςΟ - ࢓্༷σʔλ༰ྔͷ੍ݶ͕ͳ͍ɻ140TB - ύϑΥʔϚϯεΞοϓͷͨΊͷεέʔϧΞοϓ͸खಈ͕ͩɺμ΢ϯλΠϜͳ͘ॠ࣌ʹ࣮ߦ - ߴՄ༻ੑ - ೋ೥ؒେ͖ͳো֐͕ͳ͍ - ௿ϨΠςϯγɺߴύϑΥʔϚϯε - Latency -20ms - Readεϧʔϓοτ 1.3GB/sec, Writeεϧʔϓοτ 2.3MB/sec - ϑϧϚωʔδυαʔϏε $MPVE
    #JHUBCMF ΧλϩάεϖοΫҎ্ͷߴ͍ύϑΥʔϚϯε
22. © 2018 PLAID,Inc. Analyze - ύϑΥʔϚϯε/҆ఆੑ͕޲্ - αʔϏεར༻ίετ͕30%࡟ݮ - ӡ༻ʹ͔͚Δ࣌ؒͷ࡟ݮ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ ղੳσʔλͷऔಘ Track
23. © 2018 PLAID,Inc. 2. ϓϩόΠμͷো֐ΛผϓϩόΠμͰΧόʔ͢Δ
24. © 2018 PLAID,Inc. ͲΕ͚ͩૉ੖Β͍͠αʔϏεͰ΋ো֐͸ى͖Δ - 2017/3ͷS3ͷେن໛ো֐(https://aws.amazon.com/jp/message/41926/) - ໿3࣌ؒɺUS-EAST-1ͷS3͕࢖͑ͳ͘ͳΓɺଟ͘ͷαʔϏε͕μ΢ϯ - S3͸99.99ͷSLA ো֐ͷதʹ͸γϯάϧΫϥ΢υͰ͸ରԠ͠Α͏͕ͳ͍΋ͷ΋ଘࡏ͢Δ - ωοτϫʔΫͳͲڞ௨෦෼ͷো֐ - Ϧʔδϣϯ/κʔϯͷ֓೦Λ࣋ͨͳ͍ϚωʔδυαʔϏεͷো֐
25. © 2018 PLAID,Inc. KARTE͸͓٬༷ͷαΠτͷҰ෦ʹͳΔͷͰՄ༻ੑ͸ͱͯ΋ॏཁ - ྫ͑͹chat ϚϧνΫϥ΢υʹΑͬͯ͜ͷΑ͏ͳো֐Λ্ख͘཈͑Δ͜ͱ͕Ͱ͖͍ͯΔ
26. © 2018 PLAID,Inc. https://status.cloud.google.com/incident/compute/16020#5177473553661952 ໿2࣌ؒɺόοΫΤϯυͷεςʔλεʹؔ܎ͳ͘ ϦΫΤετͷ1/3Ͱ502ͷϨεϙϯεΛฦͯ͠͠·ͬͨͱ͍͏LBͷো֐
27. © 2018 PLAID,Inc.
28. © 2018 PLAID,Inc. ߦಈϩάऩूपΓͷߏ੒ ීஈ͸Route53ͷTrafficFlowͰ GCP : AWS = 8 : 2 Ͱ໊લղܾ
29. © 2018 PLAID,Inc. ߦಈϩάऩूपΓͷߏ੒ ো֐࣌͸Route53ͷTrafficFlowͰGCP : AWS = 0 : 10 Ͱ໊લղܾ
30. © 2018 PLAID,Inc. ֤ϓϩόΠμʔࣅͨΑ͏ͳαʔϏεΛఏڙ͍ͯ͠ΔͷͰɺ༷ʑͳ৔ॴͰ࣮ݱՄೳ શͯͰ͜ͷߏ੒ΛͱΔͷ͸೉͍͠ͷͰɺಋೖ෦෼ʹ͍ͭͯ͸ݟۃΊ͕ඞཁ - γεςϜͱͯ͠ͷॏཁ͞ - ࣮ݱͷ೉қ౓ ཁ݅ʹԠͯ͡ɺͲ͜·Ͱ࡞ΓࠐΉ͔ߟ͑Δ - ଟগίεύ͕ѱͯ͘΋ಈ͚͹͍͍ - खಈͰ੾Γସ͑Ε͹͍͍ ϦΞϧλΠϜղੳʹؔΘΔॏཁͳ෦෼͸ ग़དྷΔ͚ͩෳ਺ͷϓϩόΠμʔͰಈ͔ͤΔΑ͏ʹ͍ͯ͠Δ GCS S3 CloudCDN CloudFront
31. © 2018 PLAID,Inc. 3. ϓϩόΠμ·ͨ͗ͷ௨৴ʹ஫ҙ͢Δ
32. © 2018 PLAID,Inc. 1. ௨৴ίετ͕਺ඦഒʹͳͬͨ 2. ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠௿Լͨ͠
33. © 2018 PLAID,Inc. ௨৴ίετ͕਺ඦഒʹͳͬͨ࿩ - ϦΞϧλΠϜղੳͰͷBigtableͱEC2ͷ௨৴Ͱର৅ - Bigtable΁ͷසൟͳRead/WriteʹΑΓίετ૿ NBJO
34. © 2018 PLAID,Inc. ௨৴ίετ͕਺ඦഒʹͳͬͨ࿩ ผϓϩόΠμʔͷαʔϏε΁ͷ௨৴ͷίετʹ஫ҙ - ϦΞϧλΠϜղੳͰͷBigtableͱEC2ͷ௨৴Ͱର৅ - Bigtable΁ͷසൟͳRead/WriteʹΑΓίετ૿ - Bigtable͸ಉ͡zone͔Βͷ௨৴͸ίετ͸͔͔Βͳ͍ - Bigtableͱಉ͡zoneʹGCEΛߏங - ෭࣍తͳޮՌͱͯ͠ɺ҆ఆੑ/Latency͕޲্ - AWS͸GCPͷόοΫΞοϓܥʹ NBJO TUBOECZ NBJO
35. © 2018 PLAID,Inc. ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠௿Լͨ͠࿩ - GCPͱAWS઀ଓͷͨΊͷVPN෦෼ - GCE͔ΒMongo(EC2)΁ͷେྔͷRead - VPNͷଳҬ͕ανΓϘτϧωοΫ - 1.5Gps/1VPNτϯωϧ(GCP) - ଳҬʹ͸ಛʹ੍ݶͳ͠(AWS)
36. © 2018 PLAID,Inc. ผϓϩόΠμ΁ͷ71/ܦ༝ͷ௨৴ͷଳҬʹ஫ҙ ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠௿Լͨ͠࿩ - GCPʹReadReplicaΛ࡞ΓɺVPNͷ௨৴ྔ࡟ݮ - ෳ਺VPNτϯωϧʹIPϧʔςΟϯάͰෛՙ෼ࢄ - ଳҬΛ؂ࢹ͠ɺඞཁʹԠͯ͡τϯωϧΛ௥Ճ - GCPͱAWS઀ଓͷͨΊͷVPN෦෼ - GCE͔ΒMongo(EC2)΁ͷେྔͷRead - VPNͷଳҬ͕ανΓϘτϧωοΫ - 1.5Gps/1VPNτϯωϧ(GCP) - ଳҬʹ͸ಛʹ੍ݶͳ͠(AWS) IUUQUFDIQMBJEDPKQBXT@HDQ@NVMUJDMPVE@
37. © 2018 PLAID,Inc. ผϓϩόΠμʔͷαʔϏε΁ͷ௨৴ͷίετ ผϓϩόΠμʔ΁ͷVPNܦ༝ͷ௨৴ͷଳҬ NBJO TUBOECZ
38. © 2018 PLAID,Inc. 4. ෳ਺ͷϓϩόΠμʔΛ౷ҰతͳΠϯλʔϑΣʔεͰӡ༻
39. © 2018 PLAID,Inc. Ϋϥ΢υαʔϏεͱ͍ͬͯ΋ࣗ෼ͨͪͰ໘౗Λݟͳ͍ͱ͍͚ͳ͍෦෼͕͋Δ - ߏ੒؅ཧ - ؂ࢹ - ΞϓϦέʔγϣϯͷσϓϩΠ ͳͲ ӡ༻Λิॿ͢ΔαʔϏε͸֤ϓϩόΠμͰͦΕͧΕಠࣗͷϑΥʔϚοτ - ྫ͑͹ߏ੒؅ཧ DeploymentManager CloudFormation
40. © 2018 PLAID,Inc. DeploymentManager
41. © 2018 PLAID,Inc. DeploymentManager CloudFormation
42. © 2018 PLAID,Inc. ౷ҰతͳΠϯλʔϑΣʔε
43. © 2018 PLAID,Inc. ࣗ෼ͨͪͰ։ൃʁ - ϓϩόΠμ͕૿͑ΔͭͲେ͖ͳ։ൃ͕ඞཁ - ର৅͸ߏ੒؅ཧɺΞϓϦέʔγϣϯͷσϓϩΠɺ؂ࢹͳͲ - ίετେͰϓϩόΠμΛؾܰʹ૿΍ͤͳ͍ ಉ͡՝୊ײΛ๊͍͑ͯΔਓ͸݁ߏ͍Δ OSSɺαʔυύʔςΟͷαʔϏεΛ͏·͘ར༻͢Δ ౷ҰతͳΠϯλʔϑΣʔεΛͲ͏࡞Δ͔ʁ
44. © 2018 PLAID,Inc. ߏ੒؅ཧ ΞϓϦέʔγϣϯͷσϓϩΠ ؂ࢹ
45. © 2018 PLAID,Inc. ϓϩϏδϣϯ WerckerΛτϦΨʔ ߏ੒ఆٛϑΝΠϧΛ PR & Merge ߏ੒؅ཧͷ࢓૊Έ
46. © 2018 PLAID,Inc. σϓϩΠఆٛϑΝΠϧ ΛPR & Merge WerckerΛτϦΨʔ Roer ΞϓϦέʔγϣϯͷσϓϩΠͷ࢓૊Έ σϓϩΠఆٛΛϓϩϏδϣϯ
47. © 2018 PLAID,Inc. ఆٛࡁΈͷσϓϩΠͷapiΛୟ͘ σϓϩΠͷ࣮ߦ deployίϚϯυ Roer ΞϓϦέʔγϣϯͷσϓϩΠͷ࢓૊Έ σϓϩΠఆٛϑΝΠϧ ΛPR & Merge WerckerΛτϦΨʔ σϓϩΠఆٛΛϓϩϏδϣϯ
48. © 2018 PLAID,Inc. ALERT MetricsͷӾཡ ి࿩ ௨஌ ALERT/Dashboardͷఆٛ Metricsͷू໿ ؂ࢹͷ࢓૊Έ
49. © 2018 PLAID,Inc. ߏ੒؅ཧ ΞϓϦέʔγϣϯͷσϓϩΠ ؂ࢹ
50. © 2018 PLAID,Inc. ·ͱΊ
51. © 2018 PLAID,Inc. αʔϏεͷ੒௕ͱγϯάϧΫϥ΢υͷݶք͔Β ʮϓϩόΠμʹͱΒΘΕͣϚϧνΫϥ΢υʹߏ੒Λ૊Ήʯઓུʹ ϙΠϯτ͸4ͭ 1.ϓϩόΠμʔͷ͍͍ͱ͜औΓΛ͢Δ 2.ϓϩόΠμͷো֐ΛผϓϩόΠμͰΧόʔ͢Δ 3.ϓϩόΠμ·ͨ͗ͷ௨৴ʹ஫ҙ͢Δ 4.ෳ਺ͷϓϩόΠμʔΛ౷ҰతͳΠϯλʔϑΣʔεͰӡ༻͢Δ
52. © 2018 PLAID,Inc. ࠷ޙʹ
53. © 2018 PLAID,Inc. IUUQTDPEF[JOFKQBSUJDMFDPSOFS
54. © 2018 PLAID,Inc. IUUQUFDIQMBJEDPKQBXT@HDQ@NVMUJDMPVE@
55. © 2018 PLAID,Inc. IUUQTXXXXBOUFEMZDPNDPNQBOJFTQMBJE