Save 37% off PRO during our Black Friday Sale! »

マルチクラウドで構築する大規模解析サービス/戦略とポイントについて/

Fc099d9851ad91908e0c16a70aa2ddc1?s=47 tik-son
February 15, 2018

 マルチクラウドで構築する大規模解析サービス/戦略とポイントについて/

デブサミ2018でお話したこと。
http://event.shoeisha.jp/devsumi/20180215/session/1621/

Fc099d9851ad91908e0c16a70aa2ddc1?s=128

tik-son

February 15, 2018
Tweet

Transcript

  1. © 2018 PLAID,Inc. ϚϧνΫϥ΢υͰߏங͢Δେن໛ղੳαʔϏε ઓུͱϙΠϯτʹ͍ͭͯ 'FCSVBSZ  1-"*% *OD /BPIJLP5BLFNVSB

  2. © 2018 PLAID,Inc. 1SPpMF 2011೥ɹಉࢤࣾେֶ޻ֶ෦ଔۀ 2011೥͔Β2014೥·ͰɺNECʹͯΫϥ΢υαʔϏεͷ্ཱͪ͛ʹैࣄ σʔληϯλʔͷϑΝγϦςΟ͔ΒωοτϫʔΫ/αʔόͷ෺ཧϨΠϠʔ͔ΒԾ ૝ϨΠϠʔ·Ͱͷઃܭ/ߏங/ӡ༻Λ୲౰ 2014೥ɹϓϨΠυʹೖࣾ େن໛ղੳαʔϏεKARTEͷཪଆΛΠϯϑϥ͔ΒόοΫΤϯυ·Ͱ෯޿͘୲౰

    ஛ଜɹঘ඙
  3. © 2018 PLAID,Inc. 8IBU`T1-"*%

  4. © 2018 PLAID,Inc. 4IPFT" 4IPFT# d  d  4)011*/($"35

    ๚໰தͷϢʔβʔΛ ϦΞϧλΠϜղੳ ݸʑͷϢʔβʔʹ߹Θͤͨ ίϛϡχέʔγϣϯΛఏڙ 8IBU`T,"35&
  5. © 2018 PLAID,Inc. 8IBU`T,"35& ๚໰தͷϢʔβʔΛϦΞϧλΠϜʹղੳ

  6. © 2018 PLAID,Inc. 8IBU`T,"35& ϦΞϧλΠϜʹղੳͨ݁͠ՌΛ΋ͱʹ༷ʑͳΞΫγϣϯΛग़͠෼͚Δ͜ͱ͕Ͱ͖Δ

  7. © 2018 PLAID,Inc. ಋೖΫϥΠΞϯτ ղੳUU DBͷ༰ྔ τϥοΩϯάΠϕϯτ TB events /

    sec 6000 1430+ 500 ղੳ࣌ؒ ඵ 0.x UU 12.5ԯ ྲྀ௨ֹۚ ԁ / ೥ 5000ԯ 4UBUTPG,"35& ΫϥΠΞϯτ
  8. © 2018 PLAID,Inc. Event Cost $PTUTPG,"35& 2೥લ ? ӡ༻ʹܞΘΔ Engineer

    ݱࡏ
  9. © 2018 PLAID,Inc. Event Cost *OGSBTUSBUFHZPG,"35& Ͱ͖Δ͚ͩγϯάϧΫϥ΢υͰ γϯϓϧͳߏ੒Λ૊Ή ϓϩόΠμʹͱΒΘΕͣ ϚϧνΫϥ΢υʹߏ੒Λ૊Ή

    ઓུͷมߋ 2೥લ ӡ༻ʹܞΘΔ Engineer ݱࡏ
  10. © 2018 PLAID,Inc. ϚϧνΫϥ΢υΛ࢖͍͜ͳͨ͢Ίͷ4ͭͷϙΠϯτ

  11. © 2018 PLAID,Inc. શମߏ੒

  12. © 2018 PLAID,Inc. 1. ֤ϓϩόΠμʔͷ͍͍ͱ͜औΓΛ͢Δ

  13. © 2018 PLAID,Inc. ༷ʑͳϓϩόΠμ/αʔϏε͕ଘࡏ - ૯߹తͳϓϩόΠμɺҰ఺ಛԽܕͷϓϩόΠμ - ϓϩόΠμͷ࣭ͷ޲্ ϓϩόΠμ/αʔϏεʹ͸ಘҙ෼໺͕͋Δ -

    AWS ͱ GCP ಘҙ෼໺Λ͍͍ͱ͜ͲΓ͢ΔͱγϯάϧϓϩόΠμͰͷγεςϜΑΓڧ͘ͳͬͨ
  14. © 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε

    ղੳσʔλอଘ ղੳσʔλͷऔಘ DB Track
  15. © 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε

    ղੳσʔλอଘ ղੳσʔλͷऔಘ Track DB
  16. © 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε

    ղੳσʔλอଘ - ௿ϨΠςϯγʔ͚ͩͲϑϧϚωʔδυͰ͸ͳ͍ - ಥવͷϝϯςφϯε - ύϑΥʔϚϯεͷݶք ղੳσʔλͷऔಘ Track DB
  17. © 2018 PLAID,Inc. - DBͷ࢓༷͕KARTEͱ૬ੑ͕ѱ͔ͬͨ - ΧλϩάεϖοΫ௨ΓʹύϑΥʔϚϯεग़ͳ͍ - ࢖͏΄ͲΤΫεϙωϯγϟϧʹίετ͕͔͔Δ ϦΞϧλΠϜղੳͷྲྀΕ

    ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ Ωϡʔ Analyze ղੳσʔλͷऔಘ DB Track - ௿ϨΠςϯγʔ͚ͩͲϑϧϚωʔδυͰ͸ͳ͍ - ಥવͷϝϯςφϯε - ύϑΥʔϚϯεͷݶք
  18. © 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε

    ղੳσʔλอଘ ղੳσʔλͷऔಘ Track DB
  19. © 2018 PLAID,Inc. redis enterpriseΛΫϥ΢υͰఏڙ͍ͯ͠ΔαʔϏε ಛ௃ - ߴ͍εέʔϥϏϦςΟ - ࢓্༷σʔλ༰ྔɺίϯϐϡʔλϦιʔεͷ੍ݶ͕ͳ͘ɺෛՙʹԠͯࣗ͡ಈͰεέʔϧ

    - ௿ϨΠςϯγɺߴύϑΥʔϚϯε - Latency 0.1ms - ReadOps 3kOps/sec, WriteOps 11kOps/sec - ϑϧϚωʔδυαʔϏε - ΍Δ͜ͱ͸ॳظઃఆͷΈ - ߴՄ༻ੑ - ೋ೥ؒେ͖ͳো֐͕ͳ͍ - DBͷσʔλྔʹΑΔैྔ՝ۚɻωοτϫʔΫίετͰ͸՝ۚ͞Εͳ͍ɻ ௿ϨΠςϯγɺεέʔϥϒϧͳΩϡʔΛ௿ίετͰར༻Ͱ͖ΔΑ͏ʹ SFEJTDMPVE
  20. © 2018 PLAID,Inc. Analyze ϦΞϧλΠϜղੳͷྲྀΕ ղੳσʔλͷऔಘ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε

    ղੳσʔλอଘ Track
  21. © 2018 PLAID,Inc. GoogleͷαʔϏε಺Ͱར༻͍ͯ͠Δٕज़ΛΫϥ΢υͰఏڙ ಛ௃ - ߴ͍εέʔϥϏϦςΟ - ࢓্༷σʔλ༰ྔͷ੍ݶ͕ͳ͍ɻ140TB -

    ύϑΥʔϚϯεΞοϓͷͨΊͷεέʔϧΞοϓ͸खಈ͕ͩɺμ΢ϯλΠϜͳ͘ॠ࣌ʹ࣮ߦ - ߴՄ༻ੑ - ೋ೥ؒେ͖ͳো֐͕ͳ͍ - ௿ϨΠςϯγɺߴύϑΥʔϚϯε - Latency -20ms - Readεϧʔϓοτ 1.3GB/sec, Writeεϧʔϓοτ 2.3MB/sec - ϑϧϚωʔδυαʔϏε $MPVE#JHUBCMF ΧλϩάεϖοΫҎ্ͷߴ͍ύϑΥʔϚϯε
  22. © 2018 PLAID,Inc. Analyze - ύϑΥʔϚϯε/҆ఆੑ͕޲্ - αʔϏεར༻ίετ͕30%࡟ݮ - ӡ༻ʹ͔͚Δ࣌ؒͷ࡟ݮ

    ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ ղੳσʔλͷऔಘ Track
  23. © 2018 PLAID,Inc. 2. ϓϩόΠμͷো֐ΛผϓϩόΠμͰΧόʔ͢Δ

  24. © 2018 PLAID,Inc. ͲΕ͚ͩૉ੖Β͍͠αʔϏεͰ΋ো֐͸ى͖Δ - 2017/3ͷS3ͷେن໛ো֐(https://aws.amazon.com/jp/message/41926/) - ໿3࣌ؒɺUS-EAST-1ͷS3͕࢖͑ͳ͘ͳΓɺଟ͘ͷαʔϏε͕μ΢ϯ - S3͸99.99ͷSLA

    ো֐ͷதʹ͸γϯάϧΫϥ΢υͰ͸ରԠ͠Α͏͕ͳ͍΋ͷ΋ଘࡏ͢Δ - ωοτϫʔΫͳͲڞ௨෦෼ͷো֐ - Ϧʔδϣϯ/κʔϯͷ֓೦Λ࣋ͨͳ͍ϚωʔδυαʔϏεͷো֐
  25. © 2018 PLAID,Inc. KARTE͸͓٬༷ͷαΠτͷҰ෦ʹͳΔͷͰՄ༻ੑ͸ͱͯ΋ॏཁ - ྫ͑͹chat ϚϧνΫϥ΢υʹΑͬͯ͜ͷΑ͏ͳো֐Λ্ख͘཈͑Δ͜ͱ͕Ͱ͖͍ͯΔ

  26. © 2018 PLAID,Inc. https://status.cloud.google.com/incident/compute/16020#5177473553661952 ໿2࣌ؒɺόοΫΤϯυͷεςʔλεʹؔ܎ͳ͘ ϦΫΤετͷ1/3Ͱ502ͷϨεϙϯεΛฦͯ͠͠·ͬͨͱ͍͏LBͷো֐

  27. © 2018 PLAID,Inc.

  28. © 2018 PLAID,Inc. ߦಈϩάऩूपΓͷߏ੒ ීஈ͸Route53ͷTrafficFlowͰ GCP : AWS = 8

    : 2 Ͱ໊લղܾ
  29. © 2018 PLAID,Inc. ߦಈϩάऩूपΓͷߏ੒ ো֐࣌͸Route53ͷTrafficFlowͰGCP : AWS = 0 :

    10 Ͱ໊લղܾ
  30. © 2018 PLAID,Inc. ֤ϓϩόΠμʔࣅͨΑ͏ͳαʔϏεΛఏڙ͍ͯ͠ΔͷͰɺ༷ʑͳ৔ॴͰ࣮ݱՄೳ શͯͰ͜ͷߏ੒ΛͱΔͷ͸೉͍͠ͷͰɺಋೖ෦෼ʹ͍ͭͯ͸ݟۃΊ͕ඞཁ - γεςϜͱͯ͠ͷॏཁ͞ - ࣮ݱͷ೉қ౓ ཁ݅ʹԠͯ͡ɺͲ͜·Ͱ࡞ΓࠐΉ͔ߟ͑Δ

    - ଟগίεύ͕ѱͯ͘΋ಈ͚͹͍͍ - खಈͰ੾Γସ͑Ε͹͍͍ ϦΞϧλΠϜղੳʹؔΘΔॏཁͳ෦෼͸ ग़དྷΔ͚ͩෳ਺ͷϓϩόΠμʔͰಈ͔ͤΔΑ͏ʹ͍ͯ͠Δ GCS S3 CloudCDN CloudFront
  31. © 2018 PLAID,Inc. 3. ϓϩόΠμ·ͨ͗ͷ௨৴ʹ஫ҙ͢Δ

  32. © 2018 PLAID,Inc. 1. ௨৴ίετ͕਺ඦഒʹͳͬͨ 2. ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠௿Լͨ͠

  33. © 2018 PLAID,Inc. ௨৴ίετ͕਺ඦഒʹͳͬͨ࿩ - ϦΞϧλΠϜղੳͰͷBigtableͱEC2ͷ௨৴Ͱର৅ - Bigtable΁ͷසൟͳRead/WriteʹΑΓίετ૿ NBJO

  34. © 2018 PLAID,Inc. ௨৴ίετ͕਺ඦഒʹͳͬͨ࿩ ผϓϩόΠμʔͷαʔϏε΁ͷ௨৴ͷίετʹ஫ҙ - ϦΞϧλΠϜղੳͰͷBigtableͱEC2ͷ௨৴Ͱର৅ - Bigtable΁ͷසൟͳRead/WriteʹΑΓίετ૿ -

    Bigtable͸ಉ͡zone͔Βͷ௨৴͸ίετ͸͔͔Βͳ͍ - Bigtableͱಉ͡zoneʹGCEΛߏங - ෭࣍తͳޮՌͱͯ͠ɺ҆ఆੑ/Latency͕޲্ - AWS͸GCPͷόοΫΞοϓܥʹ NBJO TUBOECZ NBJO
  35. © 2018 PLAID,Inc. ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠௿Լͨ͠࿩ - GCPͱAWS઀ଓͷͨΊͷVPN෦෼ - GCE͔ΒMongo(EC2)΁ͷେྔͷRead - VPNͷଳҬ͕ανΓϘτϧωοΫ

    - 1.5Gps/1VPNτϯωϧ(GCP) - ଳҬʹ͸ಛʹ੍ݶͳ͠(AWS)
  36. © 2018 PLAID,Inc. ผϓϩόΠμ΁ͷ71/ܦ༝ͷ௨৴ͷଳҬʹ஫ҙ ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠௿Լͨ͠࿩ - GCPʹReadReplicaΛ࡞ΓɺVPNͷ௨৴ྔ࡟ݮ - ෳ਺VPNτϯωϧʹIPϧʔςΟϯάͰෛՙ෼ࢄ -

    ଳҬΛ؂ࢹ͠ɺඞཁʹԠͯ͡τϯωϧΛ௥Ճ - GCPͱAWS઀ଓͷͨΊͷVPN෦෼ - GCE͔ΒMongo(EC2)΁ͷେྔͷRead - VPNͷଳҬ͕ανΓϘτϧωοΫ - 1.5Gps/1VPNτϯωϧ(GCP) - ଳҬʹ͸ಛʹ੍ݶͳ͠(AWS) IUUQUFDIQMBJEDPKQBXT@HDQ@NVMUJDMPVE@
  37. © 2018 PLAID,Inc. ผϓϩόΠμʔͷαʔϏε΁ͷ௨৴ͷίετ ผϓϩόΠμʔ΁ͷVPNܦ༝ͷ௨৴ͷଳҬ NBJO TUBOECZ

  38. © 2018 PLAID,Inc. 4. ෳ਺ͷϓϩόΠμʔΛ౷ҰతͳΠϯλʔϑΣʔεͰӡ༻

  39. © 2018 PLAID,Inc. Ϋϥ΢υαʔϏεͱ͍ͬͯ΋ࣗ෼ͨͪͰ໘౗Λݟͳ͍ͱ͍͚ͳ͍෦෼͕͋Δ - ߏ੒؅ཧ - ؂ࢹ - ΞϓϦέʔγϣϯͷσϓϩΠ

    ͳͲ ӡ༻Λิॿ͢ΔαʔϏε͸֤ϓϩόΠμͰͦΕͧΕಠࣗͷϑΥʔϚοτ - ྫ͑͹ߏ੒؅ཧ DeploymentManager CloudFormation
  40. © 2018 PLAID,Inc. DeploymentManager

  41. © 2018 PLAID,Inc. DeploymentManager CloudFormation

  42. © 2018 PLAID,Inc. ౷ҰతͳΠϯλʔϑΣʔε

  43. © 2018 PLAID,Inc. ࣗ෼ͨͪͰ։ൃʁ - ϓϩόΠμ͕૿͑ΔͭͲେ͖ͳ։ൃ͕ඞཁ - ର৅͸ߏ੒؅ཧɺΞϓϦέʔγϣϯͷσϓϩΠɺ؂ࢹͳͲ - ίετେͰϓϩόΠμΛؾܰʹ૿΍ͤͳ͍

    ಉ͡՝୊ײΛ๊͍͑ͯΔਓ͸݁ߏ͍Δ OSSɺαʔυύʔςΟͷαʔϏεΛ͏·͘ར༻͢Δ ౷ҰతͳΠϯλʔϑΣʔεΛͲ͏࡞Δ͔ʁ
  44. © 2018 PLAID,Inc. ߏ੒؅ཧ ΞϓϦέʔγϣϯͷσϓϩΠ ؂ࢹ

  45. © 2018 PLAID,Inc. ϓϩϏδϣϯ WerckerΛτϦΨʔ ߏ੒ఆٛϑΝΠϧΛ PR & Merge ߏ੒؅ཧͷ࢓૊Έ

  46. © 2018 PLAID,Inc. σϓϩΠఆٛϑΝΠϧ ΛPR & Merge WerckerΛτϦΨʔ Roer ΞϓϦέʔγϣϯͷσϓϩΠͷ࢓૊Έ

    σϓϩΠఆٛΛϓϩϏδϣϯ
  47. © 2018 PLAID,Inc. ఆٛࡁΈͷσϓϩΠͷapiΛୟ͘ σϓϩΠͷ࣮ߦ deployίϚϯυ Roer ΞϓϦέʔγϣϯͷσϓϩΠͷ࢓૊Έ σϓϩΠఆٛϑΝΠϧ ΛPR

    & Merge WerckerΛτϦΨʔ σϓϩΠఆٛΛϓϩϏδϣϯ
  48. © 2018 PLAID,Inc. ALERT MetricsͷӾཡ ి࿩ ௨஌ ALERT/Dashboardͷఆٛ Metricsͷू໿ ؂ࢹͷ࢓૊Έ

  49. © 2018 PLAID,Inc. ߏ੒؅ཧ ΞϓϦέʔγϣϯͷσϓϩΠ ؂ࢹ

  50. © 2018 PLAID,Inc. ·ͱΊ

  51. © 2018 PLAID,Inc. αʔϏεͷ੒௕ͱγϯάϧΫϥ΢υͷݶք͔Β ʮϓϩόΠμʹͱΒΘΕͣϚϧνΫϥ΢υʹߏ੒Λ૊Ήʯઓུʹ ϙΠϯτ͸4ͭ 1.ϓϩόΠμʔͷ͍͍ͱ͜औΓΛ͢Δ 2.ϓϩόΠμͷো֐ΛผϓϩόΠμͰΧόʔ͢Δ 3.ϓϩόΠμ·ͨ͗ͷ௨৴ʹ஫ҙ͢Δ 4.ෳ਺ͷϓϩόΠμʔΛ౷ҰతͳΠϯλʔϑΣʔεͰӡ༻͢Δ

  52. © 2018 PLAID,Inc. ࠷ޙʹ

  53. © 2018 PLAID,Inc. IUUQTDPEF[JOFKQBSUJDMFDPSOFS

  54. © 2018 PLAID,Inc. IUUQUFDIQMBJEDPKQBXT@HDQ@NVMUJDMPVE@

  55. © 2018 PLAID,Inc. IUUQTXXXXBOUFEMZDPNDPNQBOJFTQMBJE