Multicloud deploy with Spinnaker / builderscon tokyo 2018

Slide 1

Slide 1 text

Multi-cloud deploy with Spinnaker @ikemonn

Slide 2

Slide 2 text

@ikemonn SRE at

Slide 3

Slide 3 text

Spinnakerͷ͜ͱΛ ஌ͬͯΔਓ✋

Slide 4

Slide 4 text

SpinnakerΛࢼͯ͠ΈΑ͏ͱ ࢥͬͯ΋Β͏ ࠓ೔ͷΰʔϧ

Slide 5

Slide 5 text

CXϓϥοτϑΥʔϜ KARTE

Slide 6

Slide 6 text

4IPFT" 4IPFT# d d 4)011*/($"35 ๚໰தͷϢʔβʔΛ ϦΞϧλΠϜղੳ ݸʑͷϢʔβʔʹ߹Θͤͨ ίϛϡχέʔγϣϯΛఏڙ

Slide 7

Slide 7 text

ϢʔβΛϦΞϧλΠϜͰ֬ೝͰ͖Δ

Slide 8

Slide 8 text

ඵؒ࠷େΠϕϯτ਺ events / sec 22,000 1೔ͷ߹ܭΠϕϯτ਺ events / day 1billion ղੳ࣌ؒ sec 0.x

Slide 9

Slide 9 text

ϚϧνΫϥ΢υ(AWS/GCP)

Slide 10

Slide 10 text

ϦΞϧλΠϜੑ͕ٻΊΒΕΔɺ ߴෛՙͳେن໛෼ࢄγεςϜ

Slide 11

Slide 11 text

Agenda • σϓϩΠͷࢥ૝ • Spinnaker΁ͷҠߦ • Spinnakerӡ༻Ͱಘͨ஌ݟ • ·ͱΊ

Slide 12

Slide 12 text

Agenda • σϓϩΠͷࢥ૝ • Spinnaker΁ͷҠߦ • Spinnakerӡ༻Ͱಘͨ஌ݟ • ·ͱΊ

Slide 13

Slide 13 text

• খ͞ͳ୯ҐͰසൟʹσϓϩΠ͢Δ • ୭Ͱ΋σϓϩΠͰ͖Δ

Slide 14

Slide 14 text

• ໰୊ͷಛఆͱRollback͠΍͘͢͢ΔͨΊ • ࠷௿1೔1ճ͸σϓϩΠ • ଟ͍࣌͸10ճҎ্ খ͞ͳ୯ҐͰසൟʹσϓϩΠ͢Δ https://medium.com/@Zaiku/continuous-delivery-in-a-nutshell-29f4213dabda

Slide 15

Slide 15 text

• վળαΠΫϧͷߴ଎Խ & SRE:3ਓ͚ͩ • σϓϩΠͷࣗಈԽ • σϓϩΠ࣌ʹ໰୊͕͋ͬͨ࣌͸ࣗಈrollback ୭Ͱ΋σϓϩΠͰ͖Δ http://susunshun.hatenablog.com/entry/2015/10/30/011554

Slide 16

Slide 16 text

https://www.autorabit.com/salesforce-deployment-automation/ σϓϩΠπʔϧʹٻΊΔ͜ͱ

Slide 17

Slide 17 text

େن໛Πϯϑϥʹසൟʹ҆ఆͨ͠ ࣗಈσϓϩΠ͕ߦ͑Δ͜ͱ

Slide 18

Slide 18 text

Agenda • σϓϩΠͷࢥ૝ • Spinnaker΁ͷҠߦ • Spinnakerӡ༻Ͱಘͨ஌ݟ • ·ͱΊ

Slide 19

Slide 19 text

ϚϧνΫϥ΢υԽ

Slide 20

Slide 20 text

• CodeDeployΛແཧ΍Γ࢖͍ͬͯͨ • ϚϧνΫϥ΢υωΠςΟϒͷπʔϧͰ͸ͳ͍ͨ Ίɺ໰୊͕ى͖ҠߦΛܾఆ ϚϧνΫϥ΢υॳظ https://docs.aws.amazon.com/ja_jp/codedeploy/latest/userguide/instances-on-premises.html

Slide 21

Slide 21 text

ϚϧνΫϥ΢υରԠͷ σϓϩΠπʔϧΛ୳͢

Slide 22

Slide 22 text

https://cloudplatform.googleblog.com/2017/06/spinnaker-10-continuous-delivery.html

Slide 23

Slide 23 text

Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and conﬁdence. https://netﬂix.github.io/

Slide 24

Slide 24 text

https://www.spinnaker.io/

Slide 25

Slide 25 text

• ଞͷ࡞ۀ΋͠ͳ͕ΒɺSRE2ਓͰ3ϲ݄ • σϓϩΠϑϩʔͷઃܭ • όάͷճආํ๏ͷ໛ࡧɺPRͷϚʔδ଴ͪ Ҡߦίετ

Slide 26

Slide 26 text

Agenda • σϓϩΠͷࢥ૝ • Spinnaker΁ͷҠߦ • Spinnakerӡ༻Ͱಘͨ஌ݟ • ·ͱΊ

Slide 27

Slide 27 text

• ✅ϚϧνΫϥ΢υରԠ • ✅҆ఆͨ͠σϓϩΠ • ✅σϓϩΠϑϩʔͷίʔυԽ • ✅ࣗಈσϓϩΠ • ✅։ൃ͕׆ൃ ظ଴௨Γͩͬͨ͜ͱ

Slide 28

Slide 28 text

• ✅ϚϧνΫϥ΢υରԠ • ҆ఆͨ͠σϓϩΠ • σϓϩΠϑϩʔͷίʔυԽ • ࣗಈσϓϩΠ • ։ൃ͕׆ൃ ظ଴௨Γͩͬͨ͜ͱ

Slide 29

Slide 29 text

• ओཁΫϥ΢υϓϩόΠμ΍k8sͰ΋ར༻Մೳ ϚϧνΫϥ΢υରԠ Azure VM Openstack Amazon EC2 Amazon ECS

Slide 30

Slide 30 text

• ֤Ϋϥ΢υͷࠩ෼ΛҙࣝͤͣʹσϓϩΠͰ͖Δ ϚϧνΫϥ΢υରԠ

Slide 31

Slide 31 text

• ϚϧνΫϥ΢υରԠ • ✅҆ఆͨ͠σϓϩΠ • σϓϩΠϑϩʔͷίʔυԽ • ࣗಈσϓϩΠ • ։ൃ͕׆ൃ ظ଴௨Γͩͬͨ͜ͱ

Slide 32

Slide 32 text

• σϓϩΠͷਐḿΛ؅ཧը໘Ͱ֬ೝͰ͖Δ ҆ఆͨ͠σϓϩΠ

Slide 33

Slide 33 text

• stage, pipelineͰॊೈʹσϓϩΠϑϩʔΛ૊ ΊΔ ҆ఆͨ͠σϓϩΠ

Slide 34

Slide 34 text

• stageͷbuild-inػೳ͕ॆ࣮͓ͯ͠Γɺ΍Γͨ ͍͜ͱ͸େ఍Ͱ͖Δ(20+) ҆ఆͨ͠σϓϩΠ

Slide 35

Slide 35 text

• Blue-GreenσϓϩΠ΍ΧφϦΞσϓϩΠʹ΋ ରԠ(spinnaker/kayenta) ҆ఆͨ͠σϓϩΠ https://cloudplatform.googleblog.com/2018/04/introducing-Kayenta-an-open-automated-canary-analysis-tool-from-Google-and-Netﬂix.html

Slide 36

Slide 36 text

• Immutable Infraͷڧ੍ • ιʔείʔυͰ͸ͳ͘imageΛ੾Γସ͑Δࢥ૝ • ιʔείʔυ΍ઃఆ͕ҟͳΔαʔό͕ଘࡏ͠ͳ͍ • ෭࣍తͳޮՌͱͯ͠Spin-upͷߴ଎Խ • CodeDeployʹൺ΂ͯ5ഒ ҆ఆͨ͠σϓϩΠ

Slide 37

Slide 37 text

• ϚϧνΫϥ΢υରԠ • ҆ఆͨ͠σϓϩΠ • ✅σϓϩΠϑϩʔͷίʔυԽ • ࣗಈσϓϩΠ • ։ൃ͕׆ൃ ظ଴௨Γͩͬͨ͜ͱ

Slide 38

Slide 38 text

• roer/dcd-specͰpipelineͷઃఆΛ؅ཧ • ։ൃऴ͍ྃͯ͠ΔͷͰɺ͜Ε͔Β͸spin σϓϩΠϑϩʔͷίʔυԽ https://blog.spinnaker.io/spin-and-roer-managed-pipeline-templates-4fde2951c648

Slide 39

Slide 39 text

https://docs.google.com/document/d/17Ifm562aypjLSHnNJ02COFv_VMv3spicH-7zMYd-oXE/edit

Slide 40

Slide 40 text

• ϚϧνΫϥ΢υରԠ • ҆ఆͨ͠σϓϩΠ • σϓϩΠϑϩʔͷίʔυԽ • ✅ࣗಈσϓϩΠ • ։ൃ͕׆ൃ ظ଴௨Γͩͬͨ͜ͱ

Slide 41

Slide 41 text

• Gate͕REST APIΛఏڙ͍ͯ͠Δ • Pipelineͷ࣮ߦ΍݁Ռͷऔಘ౳͕Մೳ ࣗಈσϓϩΠ https://cloudacademy.com/blog/netﬂix-spinnaker/

Slide 42

Slide 42 text

• Pipelineͷىಈ/੒ޭ/ࣦഊ౳Λ௨஌Մೳ • Slack, Mail, HipChat, SMS… ࣗಈσϓϩΠ

Slide 43

Slide 43 text

• ϚϧνΫϥ΢υରԠ • ҆ఆͨ͠σϓϩΠ • σϓϩΠϑϩʔͷίʔυԽ • ࣗಈσϓϩΠ • ✅։ൃ͕׆ൃ ظ଴௨Γͩͬͨ͜ͱ

Slide 44

Slide 44 text

• Netﬂix, Google͕ੵۃతʹ։ൃ ։ൃ͕׆ൃ https://cloud.withgoogle.com/next18/sf/sessions/session/155951

Slide 45

Slide 45 text

• ❌ଞπʔϧͱͷ໾ׂ͕ඃͬͨ • ❌AWSͰσϓϩΠ͕ࣦഊ͕ͪ͠ • ❌ηϧϑϗεςΟϯά͠ͳ͍ͱ͍͚ͳ͍ • ❌όά͕ଟ͍ • ❌υΩϡϝϯτ͕গͳ͍ ظ଴ͱҧ͍ͬͯͨ͜ͱ

Slide 46

Slide 46 text

• ❌ଞπʔϧͱͷ໾ׂ͕ඃͬͨ • AWSͰσϓϩΠ͕ࣦഊ͕ͪ͠ • ηϧϑϗεςΟϯά͠ͳ͍ͱ͍͚ͳ͍ • όά͕ଟ͍ • υΩϡϝϯτ͕গͳ͍ ظ଴ͱҧ͍ͬͯͨ͜ͱ

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

SpinnakerͰ΋ Firewall, LB, ASGΛ࡞੒Մೳ

Slide 49

Slide 49 text

৽͍͠αʔόάϧʔϓΛ࡞੒͠ ͍ͯ͘ࢥ૝

Slide 50

Slide 50 text

'JSFXBMM -# "VUP4DBMJOH (SPVQ Ҡߦલ Ҡߦޙ

Slide 51

Slide 51 text

• ଞπʔϧͱͷ໾ׂ͕ඃͬͨ • ❌AWSͰσϓϩΠ͕ࣦഊ͕ͪ͠ • ηϧϑϗεςΟϯά͠ͳ͍ͱ͍͚ͳ͍ • όά͕ଟ͍ • υΩϡϝϯτ͕গͳ͍ ظ଴ͱҧ͍ͬͯͨ͜ͱ

Slide 52

Slide 52 text

• σϑΥϧτͷઃఆͩͱrate exceededͰσϓϩ Π͕ࣦഊ͕ͪ͠ AWSͰσϓϩΠ͕ࣦഊ͕ͪ͠ clouddriver.yml

Slide 53

Slide 53 text

• ଞπʔϧͱͷ໾ׂ͕ඃͬͨ • AWSͰσϓϩΠ͕ࣦഊ͕ͪ͠ • ❌ηϧϑϗεςΟϯά͠ͳ͍ͱ͍͚ͳ͍ • όά͕ଟ͍ • υΩϡϝϯτ͕গͳ͍ ظ଴ͱҧ͍ͬͯͨ͜ͱ

Slide 54

Slide 54 text

https://cloud.google.com/marketplace/ ϗεςΟϯά͠ͳ͍ͱ͍͚ͳ͍ • ϚωʔδυɾαʔϏε͕ແ͍ͷͰࣗ෼Ͱӡ༻ • GCSʹconﬁg৘ใΛอଘ • diskͷϚ΢ϯτ΍֤छઃఆ͸εΫϦϓτԽ

Slide 55

Slide 55 text

• ଞπʔϧͱͷ໾ׂ͕ඃͬͨ • AWSͰσϓϩΠ͕ࣦഊ͕ͪ͠ • ηϧϑϗεςΟϯά͠ͳ͍ͱ͍͚ͳ͍ • ❌όά͕ଟ͍ • υΩϡϝϯτ͕গͳ͍ ظ଴ͱҧ͍ͬͯͨ͜ͱ

Slide 56

Slide 56 text

• UI্ͷόάɺvupʹࣦഊ͢ΔɺCPU࢖༻཰͕ಥ વ্͕Δɺϓϩηε͕ಥવࢮ͵౳ όά͕ଟ͍

Slide 57

Slide 57 text

GCEʹσϓϩΠޙΠϯελϯεͷ ୆਺͕Ұؾʹmin·ͰݮΔ RPSͰεέʔϧ͍ͤͯ͞Δ৔߹ɺLB,Φʔτεέʔ ϥʔͱSpinnakerͷ૬ੑ͕ѱ͍ͷͰ10෼଴ͭ

Slide 58

Slide 58 text

• ଞπʔϧͱͷ໾ׂ͕ඃͬͨ • AWSͰσϓϩΠ͕ࣦഊ͕ͪ͠ • ηϧϑϗεςΟϯά͠ͳ͍ͱ͍͚ͳ͍ • όά͕ଟ͍ • ❌υΩϡϝϯτ͕গͳ͍ ظ଴ͱҧ͍ͬͯͨ͜ͱ

Slide 59

Slide 59 text

• υΩϡϝϯτ͕ॆ࣮ͯ͠ͳ͍ • ϒϩά౳ͷ৘ใ΋গͳ͍orݹ͘ͳ͍ͬͯΔ • GithubͷissueΈͨΓɺίʔυಡΜͩΓɺSlack Ͱ࣭໰ͨ͠Γ͢Δͷ͕ૣ͍ υΩϡϝϯτ͕গͳ͍

Slide 60

Slide 60 text

Agenda • σϓϩΠͷࢥ૝ • Spinnaker΁ͷҠߦ • Spinnakerӡ༻Ͱಘͨ஌ݟ • ·ͱΊ

Slide 61

Slide 61 text

ϚϧνΫϥ΢υͷIaaS΁ͷσϓϩΠ ʹ͸Spinnaker͕Φεεϝ

Slide 62

Slide 62 text

ϚϧνΫϥ΢υରԠ

Slide 63

Slide 63 text

σϓϩΠؔ࿈ͷػೳ͕ॆ࣮ https://sweetcode.io/13115-2/

Slide 64

Slide 64 text

Immutable Infraͷڧ੍ https://www.merriam-webster.com/assets/mw/word-of-the-day/social/7bcf417841f7ce37cd7e7e1427c0504f.jpg

Slide 65

Slide 65 text

Λ৐Γӽ͑Δඞཁ͋Γ όάͷଟ͞ υΩϡϝϯτͷগͳ͞

Slide 66

Slide 66 text

σϓϩΠπʔϧͰ೰ΜͰΔͳΒ ੋඇݕ౼͍ͯͩ͘͠͞ʂ

Slide 67

Slide 67 text

͓खݩͷQRίʔυ͔Β ͥͻϑΟʔυόοΫΛ͓ئ͍͠·͢

Slide 68

Slide 68 text

͓·͚

Slide 69

Slide 69 text

શମͷpipelineߏ੒

Slide 70

Slide 70 text

deploy-all deploy-aws deploy-gcp deploy-role1 deploy-role2 … Pipeline (શମ) deploy-role1 deploy-role2 …

Slide 71

Slide 71 text

Pipeline (֤Ϋϥ΢υ) αʔόͷroleຖʹσϓϩΠ & rollbackՄೳ

Slide 72

Slide 72 text

Pipeline (֤Ϋϥ΢υ) Pipeline parameter & conditional on ExpressionͰpipelineΛ੍ޚ

Slide 73

Slide 73 text

֤छϚΠΫϩαʔϏεͷઃఆ • ~/.hal/$DEPLOYMENT/proﬁles/ ҎԼʹϑ ΝΠϧΛ࡞੒ & hal deploy apply ɾ/opt/spinnaker/conﬁgʹ൓ө͞ΕΔ(v1.9)