Tweet
Tweet

Slide 1

Slide 1 text

© 2018 PLAID,Inc. ϚϧνΫϥ΢υͰߏங͢Δେن໛ղੳαʔϏε ઓུͱϙΠϯτʹ͍ͭͯ 'FCSVBSZ  1-"*% *OD /BPIJLP5BLFNVSB

Slide 2

Slide 2 text

© 2018 PLAID,Inc. 1SPpMF 2011೥ɹಉࢤࣾେֶ޻ֶ෦ଔۀ 2011೥͔Β2014೥·ͰɺNECʹͯΫϥ΢υαʔϏεͷ্ཱͪ͛ʹैࣄ σʔληϯλʔͷϑΝγϦςΟ͔ΒωοτϫʔΫ/αʔόͷ෺ཧϨΠϠʔ͔ΒԾ ૝ϨΠϠʔ·Ͱͷઃܭ/ߏங/ӡ༻Λ୲౰ 2014೥ɹϓϨΠυʹೖࣾ େن໛ղੳαʔϏεKARTEͷཪଆΛΠϯϑϥ͔ΒόοΫΤϯυ·Ͱ෯޿͘୲౰ ஛ଜɹঘ඙

Slide 3

Slide 3 text

© 2018 PLAID,Inc. 8IBU`T1-"*%

Slide 4

Slide 4 text

© 2018 PLAID,Inc. 4IPFT" 4IPFT# d  d  4)011*/($"35 ๚໰தͷϢʔβʔΛ ϦΞϧλΠϜղੳ ݸʑͷϢʔβʔʹ߹Θͤͨ ίϛϡχέʔγϣϯΛఏڙ 8IBU`T,"35&

Slide 5

Slide 5 text

© 2018 PLAID,Inc. 8IBU`T,"35& ๚໰தͷϢʔβʔΛϦΞϧλΠϜʹղੳ

Slide 6

Slide 6 text

© 2018 PLAID,Inc. 8IBU`T,"35& ϦΞϧλΠϜʹղੳͨ݁͠ՌΛ΋ͱʹ༷ʑͳΞΫγϣϯΛग़͠෼͚Δ͜ͱ͕Ͱ͖Δ

Slide 7

Slide 7 text

© 2018 PLAID,Inc. ಋೖΫϥΠΞϯτ ղੳUU DBͷ༰ྔ τϥοΩϯάΠϕϯτ TB events / sec 6000 1430+ 500 ղੳ࣌ؒ ඵ 0.x UU 12.5ԯ ྲྀ௨ֹۚ ԁ / ೥ 5000ԯ 4UBUTPG,"35& ΫϥΠΞϯτ

Slide 8

Slide 8 text

© 2018 PLAID,Inc. Event Cost $PTUTPG,"35& 2೥લ ? ӡ༻ʹܞΘΔ Engineer ݱࡏ

Slide 9

Slide 9 text

© 2018 PLAID,Inc. Event Cost *OGSBTUSBUFHZPG,"35& Ͱ͖Δ͚ͩγϯάϧΫϥ΢υͰ γϯϓϧͳߏ੒Λ૊Ή ϓϩόΠμʹͱΒΘΕͣ ϚϧνΫϥ΢υʹߏ੒Λ૊Ή ઓུͷมߋ 2೥લ ӡ༻ʹܞΘΔ Engineer ݱࡏ

Slide 10

Slide 10 text

© 2018 PLAID,Inc. ϚϧνΫϥ΢υΛ࢖͍͜ͳͨ͢Ίͷ4ͭͷϙΠϯτ

Slide 11

Slide 11 text

© 2018 PLAID,Inc. શମߏ੒

Slide 12

Slide 12 text

© 2018 PLAID,Inc. 1. ֤ϓϩόΠμʔͷ͍͍ͱ͜औΓΛ͢Δ

Slide 13

Slide 13 text

© 2018 PLAID,Inc. ༷ʑͳϓϩόΠμ/αʔϏε͕ଘࡏ - ૯߹తͳϓϩόΠμɺҰ఺ಛԽܕͷϓϩόΠμ - ϓϩόΠμͷ࣭ͷ޲্ ϓϩόΠμ/αʔϏεʹ͸ಘҙ෼໺͕͋Δ - AWS ͱ GCP ಘҙ෼໺Λ͍͍ͱ͜ͲΓ͢ΔͱγϯάϧϓϩόΠμͰͷγεςϜΑΓڧ͘ͳͬͨ

Slide 14

Slide 14 text

© 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ ղੳσʔλͷऔಘ DB Track

Slide 15

Slide 15 text

© 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ ղੳσʔλͷऔಘ Track DB

Slide 16

Slide 16 text

© 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ - ௿ϨΠςϯγʔ͚ͩͲϑϧϚωʔδυͰ͸ͳ͍ - ಥવͷϝϯςφϯε - ύϑΥʔϚϯεͷݶք ղੳσʔλͷऔಘ Track DB

Slide 17

Slide 17 text

© 2018 PLAID,Inc. - DBͷ࢓༷͕KARTEͱ૬ੑ͕ѱ͔ͬͨ - ΧλϩάεϖοΫ௨ΓʹύϑΥʔϚϯεग़ͳ͍ - ࢖͏΄ͲΤΫεϙωϯγϟϧʹίετ͕͔͔Δ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ Ωϡʔ Analyze ղੳσʔλͷऔಘ DB Track - ௿ϨΠςϯγʔ͚ͩͲϑϧϚωʔδυͰ͸ͳ͍ - ಥવͷϝϯςφϯε - ύϑΥʔϚϯεͷݶք

Slide 18

Slide 18 text

© 2018 PLAID,Inc. Analyze Ωϡʔ ϦΞϧλΠϜղੳͷྲྀΕ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ ղੳσʔλͷऔಘ Track DB

Slide 19

Slide 19 text

© 2018 PLAID,Inc. redis enterpriseΛΫϥ΢υͰఏڙ͍ͯ͠ΔαʔϏε ಛ௃ - ߴ͍εέʔϥϏϦςΟ - ࢓্༷σʔλ༰ྔɺίϯϐϡʔλϦιʔεͷ੍ݶ͕ͳ͘ɺෛՙʹԠͯࣗ͡ಈͰεέʔϧ - ௿ϨΠςϯγɺߴύϑΥʔϚϯε - Latency 0.1ms - ReadOps 3kOps/sec, WriteOps 11kOps/sec - ϑϧϚωʔδυαʔϏε - ΍Δ͜ͱ͸ॳظઃఆͷΈ - ߴՄ༻ੑ - ೋ೥ؒେ͖ͳো֐͕ͳ͍ - DBͷσʔλྔʹΑΔैྔ՝ۚɻωοτϫʔΫίετͰ͸՝ۚ͞Εͳ͍ɻ ௿ϨΠςϯγɺεέʔϥϒϧͳΩϡʔΛ௿ίετͰར༻Ͱ͖ΔΑ͏ʹ SFEJTDMPVE

Slide 20

Slide 20 text

© 2018 PLAID,Inc. Analyze ϦΞϧλΠϜղੳͷྲྀΕ ղੳσʔλͷऔಘ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ Track

Slide 21

Slide 21 text

© 2018 PLAID,Inc. GoogleͷαʔϏε಺Ͱར༻͍ͯ͠Δٕज़ΛΫϥ΢υͰఏڙ ಛ௃ - ߴ͍εέʔϥϏϦςΟ - ࢓্༷σʔλ༰ྔͷ੍ݶ͕ͳ͍ɻ140TB - ύϑΥʔϚϯεΞοϓͷͨΊͷεέʔϧΞοϓ͸खಈ͕ͩɺμ΢ϯλΠϜͳ͘ॠ࣌ʹ࣮ߦ - ߴՄ༻ੑ - ೋ೥ؒେ͖ͳো֐͕ͳ͍ - ௿ϨΠςϯγɺߴύϑΥʔϚϯε - Latency -20ms - Readεϧʔϓοτ 1.3GB/sec, Writeεϧʔϓοτ 2.3MB/sec - ϑϧϚωʔδυαʔϏε $MPVE#JHUBCMF ΧλϩάεϖοΫҎ্ͷߴ͍ύϑΥʔϚϯε

Slide 22

Slide 22 text

© 2018 PLAID,Inc. Analyze - ύϑΥʔϚϯε/҆ఆੑ͕޲্ - αʔϏεར༻ίετ͕30%࡟ݮ - ӡ༻ʹ͔͚Δ࣌ؒͷ࡟ݮ ߦಈϩάऩूϓϩηε ෛՙίϯτϩʔϧͷͨΊͷ ΩϡʔΠϯά ղੳϓϩηε ղੳσʔλอଘ ղੳσʔλͷऔಘ Track

Slide 23

Slide 23 text

© 2018 PLAID,Inc. 2. ϓϩόΠμͷো֐ΛผϓϩόΠμͰΧόʔ͢Δ

Slide 24

Slide 24 text

© 2018 PLAID,Inc. ͲΕ͚ͩૉ੖Β͍͠αʔϏεͰ΋ো֐͸ى͖Δ - 2017/3ͷS3ͷେن໛ো֐(https://aws.amazon.com/jp/message/41926/) - ໿3࣌ؒɺUS-EAST-1ͷS3͕࢖͑ͳ͘ͳΓɺଟ͘ͷαʔϏε͕μ΢ϯ - S3͸99.99ͷSLA ো֐ͷதʹ͸γϯάϧΫϥ΢υͰ͸ରԠ͠Α͏͕ͳ͍΋ͷ΋ଘࡏ͢Δ - ωοτϫʔΫͳͲڞ௨෦෼ͷো֐ - Ϧʔδϣϯ/κʔϯͷ֓೦Λ࣋ͨͳ͍ϚωʔδυαʔϏεͷো֐

Slide 25

Slide 25 text

© 2018 PLAID,Inc. KARTE͸͓٬༷ͷαΠτͷҰ෦ʹͳΔͷͰՄ༻ੑ͸ͱͯ΋ॏཁ - ྫ͑͹chat ϚϧνΫϥ΢υʹΑͬͯ͜ͷΑ͏ͳো֐Λ্ख͘཈͑Δ͜ͱ͕Ͱ͖͍ͯΔ

Slide 26

Slide 26 text

© 2018 PLAID,Inc. https://status.cloud.google.com/incident/compute/16020#5177473553661952 ໿2࣌ؒɺόοΫΤϯυͷεςʔλεʹؔ܎ͳ͘ ϦΫΤετͷ1/3Ͱ502ͷϨεϙϯεΛฦͯ͠͠·ͬͨͱ͍͏LBͷো֐

Slide 27

Slide 27 text

© 2018 PLAID,Inc.

Slide 28

Slide 28 text

© 2018 PLAID,Inc. ߦಈϩάऩूपΓͷߏ੒ ීஈ͸Route53ͷTrafficFlowͰ GCP : AWS = 8 : 2 Ͱ໊લղܾ

Slide 29

Slide 29 text

© 2018 PLAID,Inc. ߦಈϩάऩूपΓͷߏ੒ ো֐࣌͸Route53ͷTrafficFlowͰGCP : AWS = 0 : 10 Ͱ໊લղܾ

Slide 30

Slide 30 text

© 2018 PLAID,Inc. ֤ϓϩόΠμʔࣅͨΑ͏ͳαʔϏεΛఏڙ͍ͯ͠ΔͷͰɺ༷ʑͳ৔ॴͰ࣮ݱՄೳ શͯͰ͜ͷߏ੒ΛͱΔͷ͸೉͍͠ͷͰɺಋೖ෦෼ʹ͍ͭͯ͸ݟۃΊ͕ඞཁ - γεςϜͱͯ͠ͷॏཁ͞ - ࣮ݱͷ೉қ౓ ཁ݅ʹԠͯ͡ɺͲ͜·Ͱ࡞ΓࠐΉ͔ߟ͑Δ - ଟগίεύ͕ѱͯ͘΋ಈ͚͹͍͍ - खಈͰ੾Γସ͑Ε͹͍͍ ϦΞϧλΠϜղੳʹؔΘΔॏཁͳ෦෼͸ ग़དྷΔ͚ͩෳ਺ͷϓϩόΠμʔͰಈ͔ͤΔΑ͏ʹ͍ͯ͠Δ GCS S3 CloudCDN CloudFront

Slide 31

Slide 31 text

© 2018 PLAID,Inc. 3. ϓϩόΠμ·ͨ͗ͷ௨৴ʹ஫ҙ͢Δ

Slide 32

Slide 32 text

© 2018 PLAID,Inc. 1. ௨৴ίετ͕਺ඦഒʹͳͬͨ 2. ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠௿Լͨ͠

Slide 33

Slide 33 text

© 2018 PLAID,Inc. ௨৴ίετ͕਺ඦഒʹͳͬͨ࿩ - ϦΞϧλΠϜղੳͰͷBigtableͱEC2ͷ௨৴Ͱର৅ - Bigtable΁ͷසൟͳRead/WriteʹΑΓίετ૿ NBJO

Slide 34

Slide 34 text

© 2018 PLAID,Inc. ௨৴ίετ͕਺ඦഒʹͳͬͨ࿩ ผϓϩόΠμʔͷαʔϏε΁ͷ௨৴ͷίετʹ஫ҙ - ϦΞϧλΠϜղੳͰͷBigtableͱEC2ͷ௨৴Ͱର৅ - Bigtable΁ͷසൟͳRead/WriteʹΑΓίετ૿ - Bigtable͸ಉ͡zone͔Βͷ௨৴͸ίετ͸͔͔Βͳ͍ - Bigtableͱಉ͡zoneʹGCEΛߏங - ෭࣍తͳޮՌͱͯ͠ɺ҆ఆੑ/Latency͕޲্ - AWS͸GCPͷόοΫΞοϓܥʹ NBJO TUBOECZ NBJO

Slide 35

Slide 35 text

© 2018 PLAID,Inc. ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠௿Լͨ͠࿩ - GCPͱAWS઀ଓͷͨΊͷVPN෦෼ - GCE͔ΒMongo(EC2)΁ͷେྔͷRead - VPNͷଳҬ͕ανΓϘτϧωοΫ - 1.5Gps/1VPNτϯωϧ(GCP) - ଳҬʹ͸ಛʹ੍ݶͳ͠(AWS)

Slide 36

Slide 36 text

© 2018 PLAID,Inc. ผϓϩόΠμ΁ͷ71/ܦ༝ͷ௨৴ͷଳҬʹ஫ҙ ωοτϫʔΫͷεϧʔϓοτ͕ஶ͘͠௿Լͨ͠࿩ - GCPʹReadReplicaΛ࡞ΓɺVPNͷ௨৴ྔ࡟ݮ - ෳ਺VPNτϯωϧʹIPϧʔςΟϯάͰෛՙ෼ࢄ - ଳҬΛ؂ࢹ͠ɺඞཁʹԠͯ͡τϯωϧΛ௥Ճ - GCPͱAWS઀ଓͷͨΊͷVPN෦෼ - GCE͔ΒMongo(EC2)΁ͷେྔͷRead - VPNͷଳҬ͕ανΓϘτϧωοΫ - 1.5Gps/1VPNτϯωϧ(GCP) - ଳҬʹ͸ಛʹ੍ݶͳ͠(AWS) IUUQUFDIQMBJEDPKQBXT@HDQ@NVMUJDMPVE@

Slide 37

Slide 37 text

© 2018 PLAID,Inc. ผϓϩόΠμʔͷαʔϏε΁ͷ௨৴ͷίετ ผϓϩόΠμʔ΁ͷVPNܦ༝ͷ௨৴ͷଳҬ NBJO TUBOECZ

Slide 38

Slide 38 text

© 2018 PLAID,Inc. 4. ෳ਺ͷϓϩόΠμʔΛ౷ҰతͳΠϯλʔϑΣʔεͰӡ༻

Slide 39

Slide 39 text

© 2018 PLAID,Inc. Ϋϥ΢υαʔϏεͱ͍ͬͯ΋ࣗ෼ͨͪͰ໘౗Λݟͳ͍ͱ͍͚ͳ͍෦෼͕͋Δ - ߏ੒؅ཧ - ؂ࢹ - ΞϓϦέʔγϣϯͷσϓϩΠ ͳͲ ӡ༻Λิॿ͢ΔαʔϏε͸֤ϓϩόΠμͰͦΕͧΕಠࣗͷϑΥʔϚοτ - ྫ͑͹ߏ੒؅ཧ DeploymentManager CloudFormation

Slide 40

Slide 40 text

© 2018 PLAID,Inc. DeploymentManager

Slide 41

Slide 41 text

© 2018 PLAID,Inc. DeploymentManager CloudFormation

Slide 42

Slide 42 text

© 2018 PLAID,Inc. ౷ҰతͳΠϯλʔϑΣʔε

Slide 43

Slide 43 text

© 2018 PLAID,Inc. ࣗ෼ͨͪͰ։ൃʁ - ϓϩόΠμ͕૿͑ΔͭͲେ͖ͳ։ൃ͕ඞཁ - ର৅͸ߏ੒؅ཧɺΞϓϦέʔγϣϯͷσϓϩΠɺ؂ࢹͳͲ - ίετେͰϓϩόΠμΛؾܰʹ૿΍ͤͳ͍ ಉ͡՝୊ײΛ๊͍͑ͯΔਓ͸݁ߏ͍Δ OSSɺαʔυύʔςΟͷαʔϏεΛ͏·͘ར༻͢Δ ౷ҰతͳΠϯλʔϑΣʔεΛͲ͏࡞Δ͔ʁ

Slide 44

Slide 44 text

© 2018 PLAID,Inc. ߏ੒؅ཧ ΞϓϦέʔγϣϯͷσϓϩΠ ؂ࢹ

Slide 45

Slide 45 text

© 2018 PLAID,Inc. ϓϩϏδϣϯ WerckerΛτϦΨʔ ߏ੒ఆٛϑΝΠϧΛ PR & Merge ߏ੒؅ཧͷ࢓૊Έ

Slide 46

Slide 46 text

© 2018 PLAID,Inc. σϓϩΠఆٛϑΝΠϧ ΛPR & Merge WerckerΛτϦΨʔ Roer ΞϓϦέʔγϣϯͷσϓϩΠͷ࢓૊Έ σϓϩΠఆٛΛϓϩϏδϣϯ

Slide 47

Slide 47 text

© 2018 PLAID,Inc. ఆٛࡁΈͷσϓϩΠͷapiΛୟ͘ σϓϩΠͷ࣮ߦ deployίϚϯυ Roer ΞϓϦέʔγϣϯͷσϓϩΠͷ࢓૊Έ σϓϩΠఆٛϑΝΠϧ ΛPR & Merge WerckerΛτϦΨʔ σϓϩΠఆٛΛϓϩϏδϣϯ

Slide 48

Slide 48 text

© 2018 PLAID,Inc. ALERT MetricsͷӾཡ ి࿩ ௨஌ ALERT/Dashboardͷఆٛ Metricsͷू໿ ؂ࢹͷ࢓૊Έ

Slide 49

Slide 49 text

© 2018 PLAID,Inc. ߏ੒؅ཧ ΞϓϦέʔγϣϯͷσϓϩΠ ؂ࢹ

Slide 50

Slide 50 text

© 2018 PLAID,Inc. ·ͱΊ

Slide 51

Slide 51 text

© 2018 PLAID,Inc. αʔϏεͷ੒௕ͱγϯάϧΫϥ΢υͷݶք͔Β ʮϓϩόΠμʹͱΒΘΕͣϚϧνΫϥ΢υʹߏ੒Λ૊Ήʯઓུʹ ϙΠϯτ͸4ͭ 1.ϓϩόΠμʔͷ͍͍ͱ͜औΓΛ͢Δ 2.ϓϩόΠμͷো֐ΛผϓϩόΠμͰΧόʔ͢Δ 3.ϓϩόΠμ·ͨ͗ͷ௨৴ʹ஫ҙ͢Δ 4.ෳ਺ͷϓϩόΠμʔΛ౷ҰతͳΠϯλʔϑΣʔεͰӡ༻͢Δ

Slide 52

Slide 52 text

© 2018 PLAID,Inc. ࠷ޙʹ

Slide 53

Slide 53 text

© 2018 PLAID,Inc. IUUQTDPEF[JOFKQBSUJDMFDPSOFS

Slide 54

Slide 54 text

© 2018 PLAID,Inc. IUUQUFDIQMBJEDPKQBXT@HDQ@NVMUJDMPVE@

Slide 55

Slide 55 text

© 2018 PLAID,Inc. IUUQTXXXXBOUFEMZDPNDPNQBOJFTQMBJE