マルチクラウドで構築する大規模解析サービス/戦略とポイントについて/

Transcript

1. © 2018 PLAID,Inc. ϚϧνΫϥ΢υͰߏங͢Δେن໛ղੳαʔϏε
   ઓུͱϙΠϯτʹ͍ͭͯ
   'FCSVBSZ
   
   
   1-"*%
   *OD
   
   /BPIJLP
   5BLFNVSB

2. © 2018 PLAID,Inc. 1SPpMF 2011೥ɹಉࢤࣾେֶ޻ֶ෦ଔۀ 2011೥͔Β2014೥·ͰɺNECʹͯΫϥ΢υαʔϏεͷ্ཱͪ͛ʹैࣄ σʔληϯλʔͷϑΝγϦςΟ͔ΒωοτϫʔΫ/αʔόͷ෺ཧϨΠϠʔ͔ΒԾ ૝ϨΠϠʔ·Ͱͷઃܭ/ߏங/ӡ༻Λ୲౰ 2014೥ɹϓϨΠυʹೖࣾ େن໛ղੳαʔϏεKARTEͷཪଆΛΠϯϑϥ͔ΒόοΫΤϯυ·Ͱ෯޿͘୲౰

   ஛ଜɹঘ඙

3. © 2018 PLAID,Inc. 8IBU`T
   1-"*%
   

4. © 2018 PLAID,Inc. 4IPFT" 4IPFT# d  d  4)011*/(
   $"35

   ๚໰தͷϢʔβʔΛ ϦΞϧλΠϜղੳ ݸʑͷϢʔβʔʹ߹Θͤͨ ίϛϡχέʔγϣϯΛఏڙ 8IBU`T
   ,"35&
   

5. © 2018 PLAID,Inc. 8IBU`T
   ,"35&
   ๚໰தͷϢʔβʔΛϦΞϧλΠϜʹղੳ

6. © 2018 PLAID,Inc. 8IBU`T
   ,"35&
   ϦΞϧλΠϜʹղੳͨ݁͠ՌΛ΋ͱʹ༷ʑͳΞΫγϣϯΛग़͠෼͚Δ͜ͱ͕Ͱ͖Δ

7. © 2018 PLAID,Inc. ಋೖΫϥΠΞϯτ ղੳUU DBͷ༰ྔ τϥοΩϯάΠϕϯτ TB events /

   sec 6000 1430+ 500 ղੳ࣌ؒ ඵ 0.x UU 12.5ԯ ྲྀ௨ֹۚ ԁ / ೥ 5000ԯ 4UBUT
   PG
   ,"35& ΫϥΠΞϯτ

8. © 2018 PLAID,Inc. Event Cost $PTUT
   PG
   ,"35& 2೥લ ? ӡ༻ʹܞΘΔ Engineer

   ݱࡏ

9. © 2018 PLAID,Inc. Event Cost *OGSB
   TUSBUFHZ
   PG
   ,"35& Ͱ͖Δ͚ͩγϯάϧΫϥ΢υͰ γϯϓϧͳߏ੒Λ૊Ή ϓϩόΠμʹͱΒΘΕͣ ϚϧνΫϥ΢υʹߏ੒Λ૊Ή

   ઓུͷมߋ 2೥લ ӡ༻ʹܞΘΔ Engineer ݱࡏ

10. © 2018 PLAID,Inc. ϚϧνΫϥ΢υΛ࢖͍͜ͳͨ͢Ίͷ4ͭͷϙΠϯτ

11. © 2018 PLAID,Inc. શମߏ੒

12. © 2018 PLAID,Inc. 1. ֤ϓϩόΠμʔͷ͍͍ͱ͜औΓΛ͢Δ

13. © 2018 PLAID,Inc. ༷ʑͳϓϩόΠμ/αʔϏε͕ଘࡏ - ૯߹తͳϓϩόΠμɺҰ఺ಛԽܕͷϓϩόΠμ - ϓϩόΠμͷ࣭ͷ޲্ ϓϩόΠμ/αʔϏεʹ͸ಘҙ෼໺͕͋Δ -

    AWS ͱ GCP ಘҙ෼໺Λ͍͍ͱ͜ͲΓ͢ΔͱγϯάϧϓϩόΠμͰͷγεςϜΑΓڧ͘ͳͬͨ

14. © 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε

    ղੳσʔλอଘ ղੳσʔλͷऔಘ DB Track

15. © 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε

    ղੳσʔλอଘ ղੳσʔλͷऔಘ Track DB

16. © 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε

    ղੳσʔλอଘ - ௿ϨΠςϯγʔ͚ͩͲϑϧϚωʔδυͰ͸ͳ͍ - ಥવͷϝϯςφϯε - ύϑΥʔϚϯεͷݶք ղੳσʔλͷऔಘ Track DB

17. © 2018 PLAID,Inc. - DBͷ࢓༷͕KARTEͱ૬ੑ͕ѱ͔ͬͨ - ΧλϩάεϖοΫ௨ΓʹύϑΥʔϚϯεग़ͳ͍ - ࢖͏΄ͲΤΫεϙωϯγϟϧʹίετ͕͔͔Δ ϦΞϧλΠϜղੳͷྲྀΕ

    ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ Ωϡʔ Analyze ղੳσʔλͷऔಘ DB Track - ௿ϨΠςϯγʔ͚ͩͲϑϧϚωʔδυͰ͸ͳ͍ - ಥવͷϝϯςφϯε - ύϑΥʔϚϯεͷݶք

18. © 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε

    ղੳσʔλอଘ ղੳσʔλͷऔಘ Track DB

19. © 2018 PLAID,Inc. redis enterpriseΛΫϥ΢υͰఏڙ͍ͯ͠ΔαʔϏε ಛ௃ - ߴ͍εέʔϥϏϦςΟ - ࢓্༷σʔλ༰ྔɺίϯϐϡʔλϦιʔεͷ੍ݶ͕ͳ͘ɺෛՙʹԠͯࣗ͡ಈͰεέʔϧ

    - ௿ϨΠςϯγɺߴύϑΥʔϚϯε - Latency 0.1ms - ReadOps 3kOps/sec, WriteOps 11kOps/sec - ϑϧϚωʔδυαʔϏε - ΍Δ͜ͱ͸ॳظઃఆͷΈ - ߴՄ༻ੑ - ೋ೥ؒେ͖ͳো֐͕ͳ͍ - DBͷσʔλྔʹΑΔैྔ՝ۚɻωοτϫʔΫίετͰ͸՝ۚ͞Εͳ͍ɻ ௿ϨΠςϯγɺεέʔϥϒϧͳΩϡʔΛ௿ίετͰར༻Ͱ͖ΔΑ͏ʹ SFEJT
    DMPVE

20. © 2018 PLAID,Inc. Analyze ϦΞϧλΠϜղੳͷྲྀΕ ղੳσʔλͷऔಘ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε

    ղੳσʔλอଘ Track

21. © 2018 PLAID,Inc. GoogleͷαʔϏε಺Ͱར༻͍ͯ͠Δٕज़ΛΫϥ΢υͰఏڙ ಛ௃ - ߴ͍εέʔϥϏϦςΟ - ࢓্༷σʔλ༰ྔͷ੍ݶ͕ͳ͍ɻ140TB -

    ύϑΥʔϚϯεΞοϓͷͨΊͷεέʔϧΞοϓ͸खಈ͕ͩɺμ΢ϯλΠϜͳ͘ॠ࣌ʹ࣮ߦ - ߴՄ༻ੑ - ೋ೥ؒେ͖ͳো֐͕ͳ͍ - ௿ϨΠςϯγɺߴύϑΥʔϚϯε - Latency -20ms - Readεϧʔϓοτ 1.3GB/sec, Writeεϧʔϓοτ 2.3MB/sec - ϑϧϚωʔδυαʔϏε $MPVE
    #JHUBCMF ΧλϩάεϖοΫҎ্ͷߴ͍ύϑΥʔϚϯε

22. © 2018 PLAID,Inc. Analyze - ύϑΥʔϚϯε/҆ఆੑ͕޲্ - αʔϏεར༻ίετ͕30%࡟ݮ - ӡ༻ʹ͔͚Δ࣌ؒͷ࡟ݮ

    ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ ղੳσʔλͷऔಘ Track

23. © 2018 PLAID,Inc. 2. ϓϩόΠμͷো֐ΛผϓϩόΠμͰΧόʔ͢Δ

24. © 2018 PLAID,Inc. ͲΕ͚ͩૉ੖Β͍͠αʔϏεͰ΋ো֐͸ى͖Δ - 2017/3ͷS3ͷେن໛ো֐(https://aws.amazon.com/jp/message/41926/) - ໿3࣌ؒɺUS-EAST-1ͷS3͕࢖͑ͳ͘ͳΓɺଟ͘ͷαʔϏε͕μ΢ϯ - S3͸99.99ͷSLA

    ো֐ͷதʹ͸γϯάϧΫϥ΢υͰ͸ରԠ͠Α͏͕ͳ͍΋ͷ΋ଘࡏ͢Δ - ωοτϫʔΫͳͲڞ௨෦෼ͷো֐ - Ϧʔδϣϯ/κʔϯͷ֓೦Λ࣋ͨͳ͍ϚωʔδυαʔϏεͷো֐

25. © 2018 PLAID,Inc. KARTE͸͓٬༷ͷαΠτͷҰ෦ʹͳΔͷͰՄ༻ੑ͸ͱͯ΋ॏཁ - ྫ͑͹chat ϚϧνΫϥ΢υʹΑͬͯ͜ͷΑ͏ͳো֐Λ্ख͘཈͑Δ͜ͱ͕Ͱ͖͍ͯΔ

26. © 2018 PLAID,Inc. https://status.cloud.google.com/incident/compute/16020#5177473553661952 ໿2࣌ؒɺόοΫΤϯυͷεςʔλεʹؔ܎ͳ͘ ϦΫΤετͷ1/3Ͱ502ͷϨεϙϯεΛฦͯ͠͠·ͬͨͱ͍͏LBͷো֐

27. © 2018 PLAID,Inc.

28. © 2018 PLAID,Inc. ߦಈϩάऩूपΓͷߏ੒ ීஈ͸Route53ͷTrafficFlowͰ GCP : AWS = 8

    : 2 Ͱ໊લղܾ

29. © 2018 PLAID,Inc. ߦಈϩάऩूपΓͷߏ੒ ো֐࣌͸Route53ͷTrafficFlowͰGCP : AWS = 0 :

    10 Ͱ໊લղܾ

30. © 2018 PLAID,Inc. ֤ϓϩόΠμʔࣅͨΑ͏ͳαʔϏεΛఏڙ͍ͯ͠ΔͷͰɺ༷ʑͳ৔ॴͰ࣮ݱՄೳ શͯͰ͜ͷߏ੒ΛͱΔͷ͸೉͍͠ͷͰɺಋೖ෦෼ʹ͍ͭͯ͸ݟۃΊ͕ඞཁ - γεςϜͱͯ͠ͷॏཁ͞ - ࣮ݱͷ೉қ౓ ཁ݅ʹԠͯ͡ɺͲ͜·Ͱ࡞ΓࠐΉ͔ߟ͑Δ

    - ଟগίεύ͕ѱͯ͘΋ಈ͚͹͍͍ - खಈͰ੾Γସ͑Ε͹͍͍ ϦΞϧλΠϜղੳʹؔΘΔॏཁͳ෦෼͸ ग़དྷΔ͚ͩෳ਺ͷϓϩόΠμʔͰಈ͔ͤΔΑ͏ʹ͍ͯ͠Δ GCS S3 CloudCDN CloudFront

31. © 2018 PLAID,Inc. 3. ϓϩόΠμ·ͨ͗ͷ௨৴ʹ஫ҙ͢Δ

32. © 2018 PLAID,Inc. 1. ௨৴ίετ͕਺ඦഒʹͳͬͨ 2. ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠௿Լͨ͠

33. © 2018 PLAID,Inc. ௨৴ίετ͕਺ඦഒʹͳͬͨ࿩ - ϦΞϧλΠϜղੳͰͷBigtableͱEC2ͷ௨৴Ͱର৅ - Bigtable΁ͷසൟͳRead/WriteʹΑΓίετ૿ NBJO

34. © 2018 PLAID,Inc. ௨৴ίετ͕਺ඦഒʹͳͬͨ࿩ ผϓϩόΠμʔͷαʔϏε΁ͷ௨৴ͷίετʹ஫ҙ - ϦΞϧλΠϜղੳͰͷBigtableͱEC2ͷ௨৴Ͱର৅ - Bigtable΁ͷසൟͳRead/WriteʹΑΓίετ૿ -

    Bigtable͸ಉ͡zone͔Βͷ௨৴͸ίετ͸͔͔Βͳ͍ - Bigtableͱಉ͡zoneʹGCEΛߏங - ෭࣍తͳޮՌͱͯ͠ɺ҆ఆੑ/Latency͕޲্ - AWS͸GCPͷόοΫΞοϓܥʹ NBJO TUBOECZ NBJO

35. © 2018 PLAID,Inc. ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠௿Լͨ͠࿩ - GCPͱAWS઀ଓͷͨΊͷVPN෦෼ - GCE͔ΒMongo(EC2)΁ͷେྔͷRead - VPNͷଳҬ͕ανΓϘτϧωοΫ

    - 1.5Gps/1VPNτϯωϧ(GCP) - ଳҬʹ͸ಛʹ੍ݶͳ͠(AWS)

36. © 2018 PLAID,Inc. ผϓϩόΠμ΁ͷ71/ܦ༝ͷ௨৴ͷଳҬʹ஫ҙ ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠௿Լͨ͠࿩ - GCPʹReadReplicaΛ࡞ΓɺVPNͷ௨৴ྔ࡟ݮ - ෳ਺VPNτϯωϧʹIPϧʔςΟϯάͰෛՙ෼ࢄ -

    ଳҬΛ؂ࢹ͠ɺඞཁʹԠͯ͡τϯωϧΛ௥Ճ - GCPͱAWS઀ଓͷͨΊͷVPN෦෼ - GCE͔ΒMongo(EC2)΁ͷେྔͷRead - VPNͷଳҬ͕ανΓϘτϧωοΫ - 1.5Gps/1VPNτϯωϧ(GCP) - ଳҬʹ͸ಛʹ੍ݶͳ͠(AWS) IUUQUFDIQMBJEDPKQBXT@HDQ@NVMUJDMPVE@

37. © 2018 PLAID,Inc. ผϓϩόΠμʔͷαʔϏε΁ͷ௨৴ͷίετ ผϓϩόΠμʔ΁ͷVPNܦ༝ͷ௨৴ͷଳҬ NBJO TUBOECZ

38. © 2018 PLAID,Inc. 4. ෳ਺ͷϓϩόΠμʔΛ౷ҰతͳΠϯλʔϑΣʔεͰӡ༻

39. © 2018 PLAID,Inc. Ϋϥ΢υαʔϏεͱ͍ͬͯ΋ࣗ෼ͨͪͰ໘౗Λݟͳ͍ͱ͍͚ͳ͍෦෼͕͋Δ - ߏ੒؅ཧ - ؂ࢹ - ΞϓϦέʔγϣϯͷσϓϩΠ

    ͳͲ ӡ༻Λิॿ͢ΔαʔϏε͸֤ϓϩόΠμͰͦΕͧΕಠࣗͷϑΥʔϚοτ - ྫ͑͹ߏ੒؅ཧ DeploymentManager CloudFormation

40. © 2018 PLAID,Inc. DeploymentManager

41. © 2018 PLAID,Inc. DeploymentManager CloudFormation

42. © 2018 PLAID,Inc. ౷ҰతͳΠϯλʔϑΣʔε

43. © 2018 PLAID,Inc. ࣗ෼ͨͪͰ։ൃʁ - ϓϩόΠμ͕૿͑ΔͭͲେ͖ͳ։ൃ͕ඞཁ - ର৅͸ߏ੒؅ཧɺΞϓϦέʔγϣϯͷσϓϩΠɺ؂ࢹͳͲ - ίετେͰϓϩόΠμΛؾܰʹ૿΍ͤͳ͍

    ಉ͡՝୊ײΛ๊͍͑ͯΔਓ͸݁ߏ͍Δ OSSɺαʔυύʔςΟͷαʔϏεΛ͏·͘ར༻͢Δ ౷ҰతͳΠϯλʔϑΣʔεΛͲ͏࡞Δ͔ʁ

44. © 2018 PLAID,Inc. ߏ੒؅ཧ ΞϓϦέʔγϣϯͷσϓϩΠ ؂ࢹ

45. © 2018 PLAID,Inc. ϓϩϏδϣϯ WerckerΛτϦΨʔ ߏ੒ఆٛϑΝΠϧΛ PR & Merge ߏ੒؅ཧͷ࢓૊Έ

46. © 2018 PLAID,Inc. σϓϩΠఆٛϑΝΠϧ ΛPR & Merge WerckerΛτϦΨʔ Roer ΞϓϦέʔγϣϯͷσϓϩΠͷ࢓૊Έ

    σϓϩΠఆٛΛϓϩϏδϣϯ

47. © 2018 PLAID,Inc. ఆٛࡁΈͷσϓϩΠͷapiΛୟ͘ σϓϩΠͷ࣮ߦ deployίϚϯυ Roer ΞϓϦέʔγϣϯͷσϓϩΠͷ࢓૊Έ σϓϩΠఆٛϑΝΠϧ ΛPR

    & Merge WerckerΛτϦΨʔ σϓϩΠఆٛΛϓϩϏδϣϯ

48. © 2018 PLAID,Inc. ALERT MetricsͷӾཡ ి࿩ ௨஌ ALERT/Dashboardͷఆٛ Metricsͷू໿ ؂ࢹͷ࢓૊Έ

49. © 2018 PLAID,Inc. ߏ੒؅ཧ ΞϓϦέʔγϣϯͷσϓϩΠ ؂ࢹ

50. © 2018 PLAID,Inc. ·ͱΊ

51. © 2018 PLAID,Inc. αʔϏεͷ੒௕ͱγϯάϧΫϥ΢υͷݶք͔Β ʮϓϩόΠμʹͱΒΘΕͣϚϧνΫϥ΢υʹߏ੒Λ૊Ήʯઓུʹ ϙΠϯτ͸4ͭ 1.ϓϩόΠμʔͷ͍͍ͱ͜औΓΛ͢Δ 2.ϓϩόΠμͷো֐ΛผϓϩόΠμͰΧόʔ͢Δ 3.ϓϩόΠμ·ͨ͗ͷ௨৴ʹ஫ҙ͢Δ 4.ෳ਺ͷϓϩόΠμʔΛ౷ҰతͳΠϯλʔϑΣʔεͰӡ༻͢Δ

52. © 2018 PLAID,Inc. ࠷ޙʹ

53. © 2018 PLAID,Inc. IUUQTDPEF[JOFKQBSUJDMFDPSOFS

54. © 2018 PLAID,Inc. IUUQUFDIQMBJEDPKQBXT@HDQ@NVMUJDMPVE@

55. © 2018 PLAID,Inc. IUUQTXXXXBOUFEMZDPNDPNQBOJFTQMBJE