Kubernetes で解決したこと新たに出てきた課題 /#jtf2017

Slide 1

Slide 1 text

,PEBJ4BLBCF!LPVEBJJJ8BOUFEMZ *OD Kubernetes Ͱղܾͨ͜͠ͱ ৽ͨʹग़͖ͯͨ՝୊ July Tech Festa 2017

Slide 2

Slide 2 text

ABOUT ME — Software Engineer — 2010/4- TIS ࡔ෦ ޿େ (KODAI SAKABE) 2015/8- Wantedly @koudaiii Site: https://koudaiii.com

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Kubernetes ෳ਺ͷϗετؒͰίϯςφԽ͞ΕͨΞϓϦέʔγϣϯΛ؅ཧ͢ΔͨΊͷOSS ΞϓϦέʔγϣϯͷdeployment, maintenance, scalingͷͨΊͷجຊతͳϝΧχζϜΛఏڙ https://github.com/kubernetes/kubernetes

Slide 5

Slide 5 text

Kubernetes Λར༻ͨ݁͠Ռ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͠΍͢͞ • εέʔϧͷ͠΍͢͞ • ՝୊ʹͳͬͨ͜ͱ • खಈͰ΍͖ͬͯͨ͜ͱͷݶք • Kubernetes ͷ Master ͕ SPOF • ϞχλϦϯάͷෳࡶ͞

Slide 6

Slide 6 text

Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ

Slide 7

Slide 7 text

ΞδΣϯμ ڈ೥·ͰͷৼΓฦΓ ࣮ફ Kubernetes ղܾͨ͜͠ͱ / ՝୊΁ͷऔΓ૊Έ τΠϧ๾໓ӡಈ The UNIX Philosophy ϚϧνϚελʔ ࣗಈϞχλϦϯά

Slide 8

Slide 8 text

ڈ೥·ͰͷৼΓฦΓ

Slide 9

Slide 9 text

͜Ε·ͰͷऔΓ૊Έ • Docker / Chef / Packer • Blue-Green Deployment • Terraform • CoreOS • cell

Slide 10

Slide 10 text

Docker / Chef / Packer • Docker image ͷதʹ Chef Λ࢖ͬͯΠϯϑϥΛߏங • Packer Ͱ AWS ͷ instance ͷ AMI Λ࡞Δ • Base ͱͳΔ template Λ༻ҙͯ͠ɺ৽͍͠αʔϏε͕ग़͖ͯͯ΋ྲྀ༻Ͱ ͖ΔΑ͏ʹͨ͠ • ΞϓϦέʔγϣϯͷߏங෦෼Λ Code Խ

Slide 11

Slide 11 text

Blue-Green Deployment • /healthcheck ͰϨεϙϯεΛฦͨ͠Β੾Γସ͑Δ UBHT OBNFBQQ BQQSBJMT DPMPSHSFFO UBHT OBNFBQQ BQQSBJMT DPMPSCMVF DBQEFQMPZTXJUDI@DPMPSCMVF QPSU QPSU

Slide 12

Slide 12 text

Terraform • AWS(S3/RDS/ELB etc) / dnsimple Λૢ࡞ IUUQTTQFBLFSEFDLDPNEUBOUFSSBGPSNBUXBOUFEMZUFDIDJSDMFOVNCFS

Slide 13

Slide 13 text

TAG ͱ AutoScaleGroup(ASG) • instance ͷ໾ׂΛɺ EC2 ͷ TAG Λ࢖ͬͯ؅ཧ • Կ୆ಈ͔͍͔ͨ͠ʁ => ASGͰઃఆ • ΠϯϑϥπʔϧΛ LaunchConﬁg ʹఆٛ • instance ্ཱ͕͕ͪΔͱ౰࣌ʹࣗ෼ʹ෇͚ΒΕͨ TAG Λ؍ͯɺඞཁͳ service Λ systemd ʹ൓ө͠ɺαʔϏεΛىಈ(cell)

Slide 14

Slide 14 text

cell • AWS TAG Λݩʹ systemd ʹαʔϏεΛొ࿥ • αʔϏεͷ status ͕ ok ʹͳͬͨΒ DNS ࣗ਎Λ௥Ճ

Slide 15

Slide 15 text

ٕज़ελοΫ • ৽نαʔϏεΛ্ཱͪ͛Δࡍ͸͜ͷٕज़ ελοΫΛશ෦༻ҙ͢Δ • Πϯϑϥͷ҆ఆԽͷͨΊͷ༷ʑπʔϧͱ ࢓૊ΈΛ༻ҙ • αʔόʔΛߏங͢Δͱ͜Ζ͔ΒσϓϩΠɺ cell ·ͰΛίʔυԽ Web Application Monitoring Logging Auto Scale Load Balancer Internal DNS

Slide 16

Slide 16 text

೉ղͳखॱ

Slide 17

Slide 17 text

࣮ફ Kubernetes

Slide 18

Slide 18 text

Kubernetes ಋೖ΁ αʔϏε͕Ͱ͖Δ౓ʹҰݸͣͭαʔόʔΛ༻ҙͯ͠ߏங͢Δํ๏ ෳ਺ͷαʔόʔΛҰͭͷେ͖ͳαʔόʔʹݟཱͯͯར༻͢Δ ಉ͡Α͏ͳΠϯϑϥ࡞ۀΛݮΒ͠ɺϦιʔεΛޮ཰తʹར༻͢Δ ࢀߟ: CodeZineʮKubernetesΛ࢖ͬͨมԽʹڧ͍Πϯϑϥ――WantedlyͷΠϯϑϥνʔϜ͕େ੾ʹ͍ͯ͠Δ͜ ͱʯhttps://codezine.jp/article/detail/10357

Slide 19

Slide 19 text

The Twelve Factor App •12 ݸͷΞϓϦέʔγϣϯͷํ๏࿦ https://12factor.net/ja/ •Herokuࣄଶ΋ίϯςφٕज़ɻDocker Λӡ༻͍ͯ͘͠࠷దͳํ๏࿦ •σʔλετΞΛίϯςφʹؚΊͳ͍ •։ൃ؀ڥͱຊ൪؀ڥͷ ࠩҟΛ؀ڥม਺Ͱઃఆ •ܧଓతσϓϩΠ

Slide 20

Slide 20 text

ͦͷଞ௥Ճϧʔϧ Docker ͷ image ͸ܰ͘ CI Λ௨ͯ͠ɺimage ࡞੒ɺdeploy Λߦͳ͏ ϦϙδτϦͱ namespace ͕ 1ର1 ʹͳΔΑ͏ʹ

Slide 21

Slide 21 text

docker image ͸ܰ͘ https://github.com/koudaiii/sltd/blob/master/Dockerﬁle Golang Ͱ͋Ε͹όΠφϦΛΆΜى͖ͩͱখ͍͞ 1 FROM alpine:3.6 2 3 RUN apk add --no-cache --update ca-certificates 4 5 COPY bin/sltd /sltd 6 7 ENTRYPOINT ["/sltd"] 8 CMD ["help"]

Slide 22

Slide 22 text

CI/CD .travis.yml ΑΓൈਮ 58 script: 59 - ./script/ci-test # Test 60 - ./script/ci-build # docker build 61 - ./script/ci-push # docker push 62 ɾɾɾɾɾɾɾɾɾɾɾ 69 deploy: 70 skip_cleanup: true 71 provider: script 72 script: ./script/ci-deploy # Release 73 on: 74 all_branches: true

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

ϦϙδτϦ = namespace ֤ϦϙδτϦʹ /kubernetes Λ഑ஔ $ tree kubernetes/ kubernetes/ ᵓᴷᴷ jobs ᴹ ᵓᴷᴷ create-blog-job.yaml ᴹ ᵓᴷᴷ destroy-blog-job.yaml ᴹ ᵋᴷᴷ update-monthly-ranking-job.yaml ᵓᴷᴷ namespace.yaml ᵓᴷᴷ rails-docker-sample-run.yaml ᵓᴷᴷ rails-docker-sample-hpa.yaml ᵋᴷᴷ rails-docker-sample-svc.yaml

Slide 25

Slide 25 text

ϚχϑΣετϑΝΠϧ Namespace ઐ༻ͷ໊લۭؒ Deployment ΞϓϦέʔγϣϯͷ࣮ߦ HorizontalPodAutoscaler cpu/memory ͔Β pod Φʔτεέʔϧ Pod one-oﬀ ίϯςφ༻(rails c ౳Λߦͳ͏༻) Service SSL෇͖ELB Job oneshot Ͱ࣮ߦ༻(rake db:migrate ౳) CronJob Cron ܗࣜͰλεΫΛ࣮ߦ(rake task ౳)

Slide 26

Slide 26 text

Namespace ઐ༻ͷ໊લۭؒΛ࡞੒ 1 apiVersion: v1 2 kind: Namespace 3 metadata: 4 name: hoge

Slide 27

Slide 27 text

Deployment 1 apiVersion: extensions/v1beta1 2 kind: Deployment ɾɾɾɾɾɾɾɾɾɾɾ 9 spec: 11 strategy: 12 type: RollingUpdate 13 rollingUpdate: 14 maxSurge: 50% # Ұ౓൒෼ೖΕସ͑ 15 maxUnavailable: 0 # replicas Λҡ࣋ ɾɾɾɾɾɾɾɾɾɾɾ 23 spec: 24 containers: 25 - image: wantedly/python:latest 26 name: python 27 ports: 28 - containerPort: 8000 29 readinessProbe: # ready ͷఆٛ 30 httpGet: 31 path: /ping 32 port: 8000 33 initialDelaySeconds: 10 34 timeoutSeconds: 1 35 envFrom: 36 - secretRef: 37 name: dotenv 38 command: ["script/server"]

Slide 28

Slide 28 text

HorizontalPodAutoscaler Pod ͷ autoscale 1 apiVersion: autoscaling/v1 2 kind: HorizontalPodAutoscaler 3 metadata: 4 name: hoge 5 namespace: hoge 6 spec: 7 maxReplicas: 30 8 minReplicas: 1 9 scaleTargetRef: 10 apiVersion: extensions/v1beta1 11 kind: Deployment 12 name: python 13 targetCPUUtilizationPercentage: 50

Slide 29

Slide 29 text

Pod one-oﬀ Container(ྫ: rails c) 1 apiVersion: v1 2 kind: Pod 3 metadata: 4 name: {USER} 5 namespace: hoge 6 labels: 7 role: console 8 spec: ɾɾɾɾɾɾɾɾɾɾɾ 11 containers: 12 - image: wantedly/bash:latest 13 imagePullPolicy: Always 14 name: {USER} 15 command: 16 - bash 17 stdin: true 18 stdinOnce: true 19 terminationMessagePath: /dev/termination-log 20 tty: true 21 envFrom: 22 - secretRef: 23 name: dotenv

Slide 30

Slide 30 text

Service SSLূ໌ॻ෇͖ ELB 1 apiVersion: v1 2 kind: Service 3 metadata: 4 name: hoge 6 annotations: 7 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:reagion:x:certificate/x 8 service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http 9 labels: 10 name: hoge 11 role: web 12 spec: 13 ports: 14 - port: 443 15 protocol: TCP 16 targetPort: 8000 17 selector: 18 name: hoge 19 role: web 20 type: LoadBalancer

Slide 31

Slide 31 text

Job (ྫ rake db:migrate) 1 apiVersion: batch/v1 2 kind: Job ɾɾɾɾɾɾɾɾɾɾɾ 16 spec: 17 restartPolicy: Never 18 containers: 19 - name: db-migrate-[REPLACE_WITH_DATETIME] 20 image: wantedly/rails:[REPLACE_WITH_TAG] 21 command: ["bundle", "exec", "rake", "db:migrate"] 22 envFrom: 23 - secretRef: 24 name: dotenv

Slide 32

Slide 32 text

CronJob (ྫ rake task) concurrencyPolicy Ͱલ࣮ߦ͕࢒ͬͨ৔߹ͷϋϯυϦϯά͕ग़དྷΔ 1 apiVersion: batch/v2alpha1 2 kind: CronJob 3 metadata: 4 namespace: hoge 5 name: create-post 6 labels: 7 job: create-post 9 spec: 10 schedule: "*/30 * * * *" 11 concurrencyPolicy: “Replace" # “Allow” or “Forbid” 12 successfulJobsHistoryLimit: 5 13 failedJobsHistoryLimit: 5 ɾɾɾɾɾɾɾɾɾɾɾ

Slide 33

Slide 33 text

ղܾͨ͜͠ͱ / ՝୊΁ͷऔΓ૊Έ

Slide 34

Slide 34 text

ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͠΍͢͞ • εέʔϧͷ͠΍͢͞

Slide 35

Slide 35 text

৽نαʔϏεͷग़͠΍͢͞ ͜Ε·Ͱ ݱࡏ

Slide 36

Slide 36 text

εέʔϧͷ͠΍͢͞ •͜Ε·Ͱ͸ instance ϨϕϧͰɺεέʔϧΛߦͳ͏࢓૊ΈΛ࡞Δඞཁ͕͋ͬͨ •ϔϧενΣοΫ •αʔϏεσΟεΧόϦ •DNSొ࿥ / Load Balancer •ࣗݾम෮ •Kubernetes ͔ΒɺԼهͷ఺͔Β༰қʹͳͬͨ •αʔϏεσΟεΧόϦͱDNSɺLoad Balancer ྑ͠ͳʹ͞Ε͍ͯΔ఺ •ίϯςφϨϕϧͷεέʔϧग़དྷΔ఺ $ kubectl autoscale deployment foo --min=2 —-max=10 —cpu-percent=50 # શମͷCPUUtilization 50% ᮢ஋Ͱ૿ݮ $ kubectl scale --replicas=3 deployment foo # ͦͷ৔Ͱ3ͭʹ͢Δ

Slide 37

Slide 37 text

՝୊ʹͳͬͨ͜ͱ 1. खಈͰ΍͖ͬͯͨ͜ͱͷݶք 2. Kubernetes ͷ Master ͕ SPOF 3. ϞχλϦϯάͷෳࡶ͞

Slide 38

Slide 38 text

՝୊΁ͷऔΓ૊Έ 1. τΠϧ๾໓ӡಈ 2. ϚϧνϚελʔ 3. ࣗಈϞχλϦϯά

Slide 39

Slide 39 text

“τΠϧͱ͸ɺϓϩμΫγϣϯαʔϏεΛಈ࡞ͤ͞Δ͜ͱʹ ؔ܎͢Δ࡞ۀͰɺख࡞ۀͰ܁Γฦ͠ߦΘΕɺࣗಈԽ͢Δ͜ͱ ͕ՄೳͰ͋Γɺઓज़తͰ௕ظతͳՁ஋Λ࣋ͨͣɺ࡞ۀྔ͕αʔ Ϗεͷ੒௕ʹൺྫ͢Δͱ͍ͬͨ܏޲Λ࣋ͭ΋ͷͰ͢ɻ” ൈਮɿ: Betsy Beyer “SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά”

Slide 40

Slide 40 text

ҰͭҎ্౰ͯ͸·ΔͱτΠϧͷՄೳੑ༗ • ख࡞ۀͰ͋Δ͜ͱ • ܁Γฦ͞ΕΔ͜ͱ • ࣗಈԽग़དྷΔ͜ͱ • ઓज़తͰ͋Δ͜ͱ(ઓུతͰ͋ͬͨΓ༧ଌʹجͮ͘΋ͷͰ͸ͳ͘ɺׂΓࠐΈͰ࢝·Γɺ໰୊ͳͲ ͕ੜͨ͜͡ͱ΁ͷରԠ) • ௕ظతͳՁ஋Λ࣋ͨͳ͍͜ͱ • αʔϏεͷ੒௕ʹରͯ͠O(n)Ͱ͋Δ͜ͱ

Slide 41

Slide 41 text

τΠϧ๾໓ӡಈ •test-build-push-release => CI ͷ template Խ •autoscale => HorizontalPodAutoscaler ϚχϑΣετΛॻ͘ •migrate => Job ϚχϑΣετΛॻ͘ •one-oﬀ => ઐ༻ͷ Pod ϚχϑΣετΛॻ͘ •SSL ͷखܰ͞ => Service ϚχϑΣετʹ annotation Λ௥ه͢Δͱઃఆ͞ΕΔ •deploy & rollback => The UNIX Philosophy ଇͬͨπʔϧ࡞੒ ࣮͸΄΅,VCFSOFUFTͰ࣮ݱ

Slide 42

Slide 42 text

The UNIX Philosophy

Slide 43

Slide 43 text

The UNIX Philosophy ൈਮ • Small is beautiful. খ͍͞΋ͷ͸ඒ͍͠ • Make each program do one thing well. 1ͭͷϓϩάϥϜʹ͸1ͭͷ͜ͱΛ͏·͘΍ ΒͤΔ • Choose portability over eﬃciency. ޮ཰ΑΓҠ২ੑΛ༏ઌ͢Δ • Use software leverage to your advantage. ιϑτ΢ΣΞΛᑏࢠ(ͯ͜)ͱͯ͠࢖͏ • Avoid captive user interfaces. ա౉ͷର࿩తΠϯλʔϑΣʔεΛආ͚Δ

Slide 44

Slide 44 text

Server Tools ͱ Client Tool • Ұͭͷ͜ͱΛ্ख͘΍Δπʔϧ࡞Γɺᑏࢠͱͯ͠ར༻ => ͍ͭͰ΋ަ׵Մೳ • ΫϥΠΞϯτଆ͸׳Ε਌͠Μͩύοέʔδ؅ཧ => homebrew Ͱ഑৴ • ΫϥΠΞϯτଆ͸πʔϧΛݺͼग़ͨ͢Ίͷϥούʔʹઐ೦ • ΫϥΠΞϯτଆʹग़དྷΔݶΓϩδοΫΛೖΕͳ͍ => ॲཧ͸ server ্ • ωοτ੾அΛ૝ఆ͢Δ => server ͱ container ͸ҡ࣋

Slide 45

Slide 45 text

homebrew Golang πʔϧΛ homebrew Ͱ binary Λ഑৴ ΫϩείϯύΠϥͰOSґଘ͕ۃྗͳ͘πʔϧ͕࢖͑ΔΑ͏ʹ

Slide 46

Slide 46 text

kube ࣾ಺πʔϧ

Slide 47

Slide 47 text

खಈσϓϩΠ ϚχϑΣετϑΝΠϧʹ annotations Λ௥ه͢Δ͜ͱͰखಈσϓϩΠ͕Ͱ͖Δ k8ship deploy [BRANCH|COMMIT_SHA1] 1 apiVersion: extensions/v1beta1 2 kind: Deployment 3 metadata: 4 name: rails 5 namespace: rails 6 labels: 7 name: rails 8 role: web 9 annotations: 10 wantedly.com/deploy-target: “true" # ௥ه 11 wantedly.com/deploy-target-container: rails # ௥ه 12 wantedly.com/github: rails=wantedly/rails # ௥ه https://github.com/dtan4/k8ship

Slide 48

Slide 48 text

kube deploy ίϚϯυ -dry-run ΦϓγϣϯͰ diﬀ Λग़͢ merge ޙ೚ҙͷλΠϛϯάͰ deploy ͍ͨ͠৔߹ʹར༻ $ kube qa deploy master --dry-run [dry-run] deploy to (deployment: "rails", container: "rails") [dry-run] before: wantedly/rails:d1cb608ee61cb18f8c397c2e27576573879b2fcf [dry-run] after: wantedly/rails:cb19269526f946222110fbdfb3d107c4b5a18fe7

Slide 49

Slide 49 text

ϚϧνϚελʔ

Slide 50

Slide 50 text

ϚϧνϚελʔ •Kubernetes ͷ version up ͸Ͳ͏͢Δͷʁ •Ϛελʔ͕ࢭ·ΔͱͲ͏ͳΔͷʁ •Ϋϥελͷఀࢭͳ͘ kops Λ࢖ͬͯ Rolling upgrade Ͱ͖ΔΑ͏ʹมߋ •ϚελʔΛࡾ୆༻ҙ͠ɺϚϧνϚελʔߏ੒ʹ͔͑ͯ SPOF վળ

Slide 51

Slide 51 text

ࣗಈϞχλϦϯά

Slide 52

Slide 52 text

ࣗಈϞχλϦϯά • 1ͭ1ͭઃఆ͢ΔͷͰ͸ͳ͘ɺ൚༻తʹద༻͞ΕΔΑ͏ʹ͢Δ • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}} • request ͱ status code औಘ • μογϡϘʔυʹ൓ө • Deployment Ͱఆٛͨ͠਺

Slide 53

Slide 53 text

ࣗಈͰಉ͡඼࣭ͷϞχλϦϯάΛ໨ࢦ͢ https://www.datadoghq.com/blog/monitoring-101-collecting-data/ http://qiita.com/koudaiii/items/bc89368e1279649f2498

Slide 54

Slide 54 text

Dashboard

Slide 55

Slide 55 text

ϞχλϦϯάͰ࢖༻͍ͯ͠Δ΋ͷ • kelseyhightower/konfd => PostgresqlͳͲͷ৘ใΛϦϙδτϦʹؚΊͳ͍࣌ʹར༻ • kubernetes/kube-state-metrics => deployment ౳ͷ available Λऔಘ • datadog/docker-dd-agent => docker ༻ɺ web ༻ɺ db ༻ͱͯ͠໾ׂผͰ഑උ • koudaiii/sltd => Service ͷ metadata Λݩʹ ELB ͷ tag ʹ௥Ճ • logentries => ΫϥελશͯͷϩάΛૹΔ

Slide 56

Slide 56 text

੒௕ਪҠ

Slide 57

Slide 57 text

αʔϏεਪҠ (2017/04Ἤ) Service 9 => 60 Container 700+ => 1,500 - 2,500+

Slide 58

Slide 58 text

·ͱΊ • kubernetes ͸ɺΠϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬʹͳͬͨ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͠΍͢͞ • εέʔϧͷ͠΍͢͞ • ՝୊͔ΒऔΓ૊Μͩ͜ͱ • खಈͰ΍͖ͬͯͨ͜ͱͷݶք => τΠϧ๾໓ӡಈ • Ϛελʔ͕ SPOF => ϚϧνϚελʔԽ • ϞχλϦϯάͷෳࡶ͞ => ࣗಈϞχλϦϯά