Kubernetes で解決したこと新たに出てきた課題 /#jtf2017

Transcript

1. ,PEBJ
   4BLBCF
   !LPVEBJJJ
   8BOUFEMZ
   *OD Kubernetes Ͱղܾͨ͜͠ͱ ৽ͨʹग़͖ͯͨ՝୊ July Tech Festa 2017

2. ABOUT ME — Software Engineer — 2010/4- TIS ࡔ෦ ޿େ

   (KODAI SAKABE) 2015/8- Wantedly @koudaiii Site: https://koudaiii.com
3. None

4. Kubernetes ෳ਺ͷϗετؒͰίϯςφԽ͞ΕͨΞϓϦέʔγϣϯΛ؅ཧ͢ΔͨΊͷOSS ΞϓϦέʔγϣϯͷdeployment, maintenance, scalingͷͨΊͷجຊతͳϝΧχζϜΛఏڙ https://github.com/kubernetes/kubernetes

5. Kubernetes Λར༻ͨ݁͠Ռ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͠΍͢͞ • εέʔϧͷ͠΍͢͞ • ՝୊ʹͳͬͨ͜ͱ

   • खಈͰ΍͖ͬͯͨ͜ͱͷݶք • Kubernetes ͷ Master ͕ SPOF • ϞχλϦϯάͷෳࡶ͞

6. Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ

7. ΞδΣϯμ ڈ೥·ͰͷৼΓฦΓ ࣮ફ Kubernetes ղܾͨ͜͠ͱ / ՝୊΁ͷऔΓ૊Έ τΠϧ๾໓ӡಈ The UNIX

   Philosophy ϚϧνϚελʔ ࣗಈϞχλϦϯά

8. ڈ೥·ͰͷৼΓฦΓ

9. ͜Ε·ͰͷऔΓ૊Έ • Docker / Chef / Packer • Blue-Green Deployment

   • Terraform • CoreOS • cell

10. Docker / Chef / Packer • Docker image ͷதʹ Chef

    Λ࢖ͬͯΠϯϑϥΛߏங • Packer Ͱ AWS ͷ instance ͷ AMI Λ࡞Δ • Base ͱͳΔ template Λ༻ҙͯ͠ɺ৽͍͠αʔϏε͕ग़͖ͯͯ΋ྲྀ༻Ͱ ͖ΔΑ͏ʹͨ͠ • ΞϓϦέʔγϣϯͷߏங෦෼Λ Code Խ

11. Blue-Green Deployment • /healthcheck ͰϨεϙϯεΛฦͨ͠Β੾Γସ͑Δ UBHT
    
    OBNF
    BQQ
    
    BQQ
    SBJMT
    
    DPMPS
    HSFFO UBHT

    
    
    OBNF
    BQQ
    
    BQQ
    SBJMT
    
    DPMPS
    CMVF DBQ
    EFQMPZTXJUDI@DPMPS
    CMVF QPSU QPSU

12. Terraform • AWS(S3/RDS/ELB etc) / dnsimple Λૢ࡞ IUUQTTQFBLFSEFDLDPNEUBOUFSSBGPSNBUXBOUFEMZUFDIDJSDMFOVNCFS

13. TAG ͱ AutoScaleGroup(ASG) • instance ͷ໾ׂΛɺ EC2 ͷ TAG Λ࢖ͬͯ؅ཧ

    • Կ୆ಈ͔͍͔ͨ͠ʁ => ASGͰઃఆ • ΠϯϑϥπʔϧΛ LaunchConfig ʹఆٛ • instance ্ཱ͕͕ͪΔͱ౰࣌ʹࣗ෼ʹ෇͚ΒΕͨ TAG Λ؍ͯɺඞཁͳ service Λ systemd ʹ൓ө͠ɺαʔϏεΛىಈ(cell)

14. cell • AWS TAG Λݩʹ systemd ʹαʔϏεΛొ࿥ • αʔϏεͷ status

    ͕ ok ʹͳͬͨΒ DNS ࣗ਎Λ௥Ճ

15. ٕज़ελοΫ • ৽نαʔϏεΛ্ཱͪ͛Δࡍ͸͜ͷٕज़ ελοΫΛશ෦༻ҙ͢Δ • Πϯϑϥͷ҆ఆԽͷͨΊͷ༷ʑπʔϧͱ ࢓૊ΈΛ༻ҙ • αʔόʔΛߏங͢Δͱ͜Ζ͔ΒσϓϩΠɺ cell

    ·ͰΛίʔυԽ Web Application Monitoring Logging Auto Scale Load Balancer Internal DNS

16. ೉ղͳखॱ

17. ࣮ફ Kubernetes

18. Kubernetes ಋೖ΁ αʔϏε͕Ͱ͖Δ౓ʹҰݸͣͭαʔόʔΛ༻ҙͯ͠ߏங͢Δํ๏ ෳ਺ͷαʔόʔΛҰͭͷେ͖ͳαʔόʔʹݟཱͯͯར༻͢Δ ಉ͡Α͏ͳΠϯϑϥ࡞ۀΛݮΒ͠ɺϦιʔεΛޮ཰తʹར༻͢Δ ࢀߟ: CodeZineʮKubernetesΛ࢖ͬͨมԽʹڧ͍Πϯϑϥ――WantedlyͷΠϯϑϥνʔϜ͕େ੾ʹ͍ͯ͠Δ͜ ͱʯhttps://codezine.jp/article/detail/10357

19. The Twelve Factor App •12 ݸͷΞϓϦέʔγϣϯͷํ๏࿦ https://12factor.net/ja/ •Herokuࣄଶ΋ίϯςφٕज़ɻDocker Λӡ༻͍ͯ͘͠࠷దͳํ๏࿦ •σʔλετΞΛίϯςφʹؚΊͳ͍

    •։ൃ؀ڥͱຊ൪؀ڥͷ ࠩҟΛ؀ڥม਺Ͱઃఆ •ܧଓతσϓϩΠ

20. ͦͷଞ௥Ճϧʔϧ Docker ͷ image ͸ܰ͘ CI Λ௨ͯ͠ɺimage ࡞੒ɺdeploy Λߦͳ͏ ϦϙδτϦͱ

    namespace ͕ 1ର1 ʹͳΔΑ͏ʹ

21. docker image ͸ܰ͘ https://github.com/koudaiii/sltd/blob/master/Dockerfile Golang Ͱ͋Ε͹όΠφϦΛΆΜى͖ͩͱখ͍͞ 1 FROM alpine:3.6 2

    3 RUN apk add --no-cache --update ca-certificates 4 5 COPY bin/sltd /sltd 6 7 ENTRYPOINT ["/sltd"] 8 CMD ["help"]

22. CI/CD .travis.yml ΑΓൈਮ 58 script: 59 - ./script/ci-test # Test

    60 - ./script/ci-build # docker build 61 - ./script/ci-push # docker push 62 ɾɾɾɾɾɾɾɾɾɾɾ 69 deploy: 70 skip_cleanup: true 71 provider: script 72 script: ./script/ci-deploy # Release 73 on: 74 all_branches: true
23. None

24. ϦϙδτϦ = namespace ֤ϦϙδτϦʹ /kubernetes Λ഑ஔ $ tree kubernetes/ kubernetes/

    ᵓᴷᴷ jobs ᴹ ᵓᴷᴷ create-blog-job.yaml ᴹ ᵓᴷᴷ destroy-blog-job.yaml ᴹ ᵋᴷᴷ update-monthly-ranking-job.yaml ᵓᴷᴷ namespace.yaml ᵓᴷᴷ rails-docker-sample-run.yaml ᵓᴷᴷ rails-docker-sample-hpa.yaml ᵋᴷᴷ rails-docker-sample-svc.yaml

25. ϚχϑΣετϑΝΠϧ Namespace ઐ༻ͷ໊લۭؒ Deployment ΞϓϦέʔγϣϯͷ࣮ߦ HorizontalPodAutoscaler cpu/memory ͔Β pod Φʔτεέʔϧ

    Pod one-off ίϯςφ༻(rails c ౳Λߦͳ͏༻) Service SSL෇͖ELB Job oneshot Ͱ࣮ߦ༻(rake db:migrate ౳) CronJob Cron ܗࣜͰλεΫΛ࣮ߦ(rake task ౳)

26. Namespace ઐ༻ͷ໊લۭؒΛ࡞੒ 1 apiVersion: v1 2 kind: Namespace 3 metadata:

    4 name: hoge

27. Deployment 1 apiVersion: extensions/v1beta1 2 kind: Deployment ɾɾɾɾɾɾɾɾɾɾɾ 9 spec:

    11 strategy: 12 type: RollingUpdate 13 rollingUpdate: 14 maxSurge: 50% # Ұ౓൒෼ೖΕସ͑ 15 maxUnavailable: 0 # replicas Λҡ࣋ ɾɾɾɾɾɾɾɾɾɾɾ 23 spec: 24 containers: 25 - image: wantedly/python:latest 26 name: python 27 ports: 28 - containerPort: 8000 29 readinessProbe: # ready ͷఆٛ 30 httpGet: 31 path: /ping 32 port: 8000 33 initialDelaySeconds: 10 34 timeoutSeconds: 1 35 envFrom: 36 - secretRef: 37 name: dotenv 38 command: ["script/server"]

28. HorizontalPodAutoscaler Pod ͷ autoscale 1 apiVersion: autoscaling/v1 2 kind: HorizontalPodAutoscaler

    3 metadata: 4 name: hoge 5 namespace: hoge 6 spec: 7 maxReplicas: 30 8 minReplicas: 1 9 scaleTargetRef: 10 apiVersion: extensions/v1beta1 11 kind: Deployment 12 name: python 13 targetCPUUtilizationPercentage: 50

29. Pod one-off Container(ྫ: rails c) 1 apiVersion: v1 2 kind:

    Pod 3 metadata: 4 name: {USER} 5 namespace: hoge 6 labels: 7 role: console 8 spec: ɾɾɾɾɾɾɾɾɾɾɾ 11 containers: 12 - image: wantedly/bash:latest 13 imagePullPolicy: Always 14 name: {USER} 15 command: 16 - bash 17 stdin: true 18 stdinOnce: true 19 terminationMessagePath: /dev/termination-log 20 tty: true 21 envFrom: 22 - secretRef: 23 name: dotenv

30. Service SSLূ໌ॻ෇͖ ELB 1 apiVersion: v1 2 kind: Service 3

    metadata: 4 name: hoge 6 annotations: 7 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:reagion:x:certificate/x 8 service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http 9 labels: 10 name: hoge 11 role: web 12 spec: 13 ports: 14 - port: 443 15 protocol: TCP 16 targetPort: 8000 17 selector: 18 name: hoge 19 role: web 20 type: LoadBalancer

31. Job (ྫ rake db:migrate) 1 apiVersion: batch/v1 2 kind: Job

    ɾɾɾɾɾɾɾɾɾɾɾ 16 spec: 17 restartPolicy: Never 18 containers: 19 - name: db-migrate-[REPLACE_WITH_DATETIME] 20 image: wantedly/rails:[REPLACE_WITH_TAG] 21 command: ["bundle", "exec", "rake", "db:migrate"] 22 envFrom: 23 - secretRef: 24 name: dotenv

32. CronJob (ྫ rake task) concurrencyPolicy Ͱલ࣮ߦ͕࢒ͬͨ৔߹ͷϋϯυϦϯά͕ग़དྷΔ 1 apiVersion: batch/v2alpha1 2

    kind: CronJob 3 metadata: 4 namespace: hoge 5 name: create-post 6 labels: 7 job: create-post 9 spec: 10 schedule: "*/30 * * * *" 11 concurrencyPolicy: “Replace" # “Allow” or “Forbid” 12 successfulJobsHistoryLimit: 5 13 failedJobsHistoryLimit: 5 ɾɾɾɾɾɾɾɾɾɾɾ

33. ղܾͨ͜͠ͱ / ՝୊΁ͷऔΓ૊Έ

34. ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͠΍͢͞ • εέʔϧͷ͠΍͢͞

35. ৽نαʔϏεͷग़͠΍͢͞ ͜Ε·Ͱ ݱࡏ

36. εέʔϧͷ͠΍͢͞ •͜Ε·Ͱ͸ instance ϨϕϧͰɺεέʔϧΛߦͳ͏࢓૊ΈΛ࡞Δඞཁ͕͋ͬͨ •ϔϧενΣοΫ •αʔϏεσΟεΧόϦ •DNSొ࿥ / Load Balancer

    •ࣗݾम෮ •Kubernetes ͔ΒɺԼهͷ఺͔Β༰қʹͳͬͨ •αʔϏεσΟεΧόϦͱDNSɺLoad Balancer ྑ͠ͳʹ͞Ε͍ͯΔ఺ •ίϯςφϨϕϧͷεέʔϧग़དྷΔ఺ $ kubectl autoscale deployment foo --min=2 —-max=10 —cpu-percent=50 # શମͷCPUUtilization 50% ᮢ஋Ͱ૿ݮ $ kubectl scale --replicas=3 deployment foo # ͦͷ৔Ͱ3ͭʹ͢Δ

37. ՝୊ʹͳͬͨ͜ͱ 1. खಈͰ΍͖ͬͯͨ͜ͱͷݶք 2. Kubernetes ͷ Master ͕ SPOF 3.

    ϞχλϦϯάͷෳࡶ͞

38. ՝୊΁ͷऔΓ૊Έ 1. τΠϧ๾໓ӡಈ 2. ϚϧνϚελʔ 3. ࣗಈϞχλϦϯά

39. “τΠϧͱ͸ɺϓϩμΫγϣϯαʔϏεΛಈ࡞ͤ͞Δ͜ͱʹ ؔ܎͢Δ࡞ۀͰɺख࡞ۀͰ܁Γฦ͠ߦΘΕɺࣗಈԽ͢Δ͜ͱ ͕ՄೳͰ͋Γɺઓज़తͰ௕ظతͳՁ஋Λ࣋ͨͣɺ࡞ۀྔ͕αʔ Ϗεͷ੒௕ʹൺྫ͢Δͱ͍ͬͨ܏޲Λ࣋ͭ΋ͷͰ͢ɻ” ൈਮɿ: Betsy Beyer “SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά”

40. ҰͭҎ্౰ͯ͸·ΔͱτΠϧͷՄೳੑ༗ • ख࡞ۀͰ͋Δ͜ͱ • ܁Γฦ͞ΕΔ͜ͱ • ࣗಈԽग़དྷΔ͜ͱ • ઓज़తͰ͋Δ͜ͱ(ઓུతͰ͋ͬͨΓ༧ଌʹجͮ͘΋ͷͰ͸ͳ͘ɺׂΓࠐΈͰ࢝·Γɺ໰୊ͳͲ ͕ੜͨ͜͡ͱ΁ͷରԠ)

    • ௕ظతͳՁ஋Λ࣋ͨͳ͍͜ͱ • αʔϏεͷ੒௕ʹରͯ͠O(n)Ͱ͋Δ͜ͱ

41. τΠϧ๾໓ӡಈ •test-build-push-release => CI ͷ template Խ •autoscale => HorizontalPodAutoscaler

    ϚχϑΣετΛॻ͘ •migrate => Job ϚχϑΣετΛॻ͘ •one-off => ઐ༻ͷ Pod ϚχϑΣετΛॻ͘ •SSL ͷखܰ͞ => Service ϚχϑΣετʹ annotation Λ௥ه͢Δͱઃఆ͞ΕΔ •deploy & rollback => The UNIX Philosophy ଇͬͨπʔϧ࡞੒ ࣮͸΄΅
    ,VCFSOFUFT
    Ͱ࣮ݱ

42. The UNIX Philosophy

43. The UNIX Philosophy ൈਮ • Small is beautiful. খ͍͞΋ͷ͸ඒ͍͠ •

    Make each program do one thing well. 1ͭͷϓϩάϥϜʹ͸1ͭͷ͜ͱΛ͏·͘΍ ΒͤΔ • Choose portability over efficiency. ޮ཰ΑΓҠ২ੑΛ༏ઌ͢Δ • Use software leverage to your advantage. ιϑτ΢ΣΞΛᑏࢠ(ͯ͜)ͱͯ͠࢖͏ • Avoid captive user interfaces. ա౉ͷର࿩తΠϯλʔϑΣʔεΛආ͚Δ

44. Server Tools ͱ Client Tool • Ұͭͷ͜ͱΛ্ख͘΍Δπʔϧ࡞Γɺᑏࢠͱͯ͠ར༻ => ͍ͭͰ΋ަ׵Մೳ •

    ΫϥΠΞϯτଆ͸׳Ε਌͠Μͩύοέʔδ؅ཧ => homebrew Ͱ഑৴ • ΫϥΠΞϯτଆ͸πʔϧΛݺͼग़ͨ͢Ίͷϥούʔʹઐ೦ • ΫϥΠΞϯτଆʹग़དྷΔݶΓϩδοΫΛೖΕͳ͍ => ॲཧ͸ server ্ • ωοτ੾அΛ૝ఆ͢Δ => server ͱ container ͸ҡ࣋

45. homebrew Golang πʔϧΛ homebrew Ͱ binary Λ഑৴ ΫϩείϯύΠϥͰOSґଘ͕ۃྗͳ͘πʔϧ͕࢖͑ΔΑ͏ʹ

46. kube ࣾ಺πʔϧ

47. खಈσϓϩΠ ϚχϑΣετϑΝΠϧʹ annotations Λ௥ه͢Δ͜ͱͰखಈσϓϩΠ͕Ͱ͖Δ k8ship deploy [BRANCH|COMMIT_SHA1] 1 apiVersion: extensions/v1beta1

    2 kind: Deployment 3 metadata: 4 name: rails 5 namespace: rails 6 labels: 7 name: rails 8 role: web 9 annotations: 10 wantedly.com/deploy-target: “true" # ௥ه 11 wantedly.com/deploy-target-container: rails # ௥ه 12 wantedly.com/github: rails=wantedly/rails # ௥ه https://github.com/dtan4/k8ship

48. kube deploy ίϚϯυ -dry-run ΦϓγϣϯͰ diff Λग़͢ merge ޙ೚ҙͷλΠϛϯάͰ deploy

    ͍ͨ͠৔߹ʹར༻ $ kube qa deploy master --dry-run [dry-run] deploy to (deployment: "rails", container: "rails") [dry-run] before: wantedly/rails:d1cb608ee61cb18f8c397c2e27576573879b2fcf [dry-run] after: wantedly/rails:cb19269526f946222110fbdfb3d107c4b5a18fe7

49. ϚϧνϚελʔ

50. ϚϧνϚελʔ •Kubernetes ͷ version up ͸Ͳ͏͢Δͷʁ •Ϛελʔ͕ࢭ·ΔͱͲ͏ͳΔͷʁ •Ϋϥελͷఀࢭͳ͘ kops Λ࢖ͬͯ

    Rolling upgrade Ͱ͖ΔΑ͏ʹมߋ •ϚελʔΛࡾ୆༻ҙ͠ɺϚϧνϚελʔߏ੒ʹ͔͑ͯ SPOF վળ

51. ࣗಈϞχλϦϯά

52. ࣗಈϞχλϦϯά • 1ͭ1ͭઃఆ͢ΔͷͰ͸ͳ͘ɺ൚༻తʹద༻͞ΕΔΑ͏ʹ͢Δ • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}}

    • request ͱ status code औಘ • μογϡϘʔυʹ൓ө • Deployment Ͱఆٛͨ͠਺

53. ࣗಈͰಉ͡඼࣭ͷϞχλϦϯάΛ໨ࢦ͢ https://www.datadoghq.com/blog/monitoring-101-collecting-data/ http://qiita.com/koudaiii/items/bc89368e1279649f2498

54. Dashboard

55. ϞχλϦϯάͰ࢖༻͍ͯ͠Δ΋ͷ • kelseyhightower/konfd => PostgresqlͳͲͷ৘ใΛϦϙδτϦʹؚΊͳ͍࣌ʹར༻ • kubernetes/kube-state-metrics => deployment ౳ͷ

    available Λऔಘ • datadog/docker-dd-agent => docker ༻ɺ web ༻ɺ db ༻ͱͯ͠໾ׂผͰ഑උ • koudaiii/sltd => Service ͷ metadata Λݩʹ ELB ͷ tag ʹ௥Ճ • logentries => ΫϥελશͯͷϩάΛૹΔ

56. ੒௕ਪҠ

57. αʔϏεਪҠ (2017/04Ἤ) Service 9 => 60 Container 700+ => 1,500

    - 2,500+

58. ·ͱΊ • kubernetes ͸ɺΠϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬʹͳͬͨ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͠΍͢͞ • εέʔϧͷ͠΍͢͞

    • ՝୊͔ΒऔΓ૊Μͩ͜ͱ • खಈͰ΍͖ͬͯͨ͜ͱͷݶք => τΠϧ๾໓ӡಈ • Ϛελʔ͕ SPOF => ϚϧνϚελʔԽ • ϞχλϦϯάͷෳࡶ͞ => ࣗಈϞχλϦϯά

59. Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ

60. IUUQTXXXXBOUFEMZDPNQSPKFDUT