Kubernetes で解決したこと新たに出てきた課題 /#jtf2017

1年を通して Kubernetes で解決したこと新たに出てきた課題をどう立ち向かったを発表しました

at July Tech Festa 2017

Transcript

1. ,PEBJ4BLBCF!LPVEBJJJ8BOUFEMZ *OD
   Kubernetes Ͱղܾͨ͜͠ͱ
   ৽ͨʹग़͖ͯͨ՝୊
   July Tech Festa 2017
   

   View Slide

2. ABOUT ME
   — Software Engineer —
   2010/4- TIS
   ࡔ෦ ޿େ (KODAI SAKABE)
   2015/8- Wantedly
   @koudaiii
   Site: https://koudaiii.com
   

   View Slide

3. View Slide

4. Kubernetes
   ෳ਺ͷϗετؒͰίϯςφԽ͞ΕͨΞϓϦέʔγϣϯΛ؅ཧ͢ΔͨΊͷOSS
   ΞϓϦέʔγϣϯͷdeployment, maintenance, scalingͷͨΊͷجຊతͳϝΧχζϜΛఏڙ
   https://github.com/kubernetes/kubernetes
   

   View Slide

5. Kubernetes Λར༻ͨ݁͠Ռ
   • ղܾͨ͜͠ͱ
   • ৽نαʔϏεͷग़͠΍͢͞
   • εέʔϧͷ͠΍͢͞
   • ՝୊ʹͳͬͨ͜ͱ
   • खಈͰ΍͖ͬͯͨ͜ͱͷݶք
   • Kubernetes ͷ Master ͕ SPOF
   • ϞχλϦϯάͷෳࡶ͞
   

   View Slide

6. Kubernetes Λར༻ͨ݁͠Ռ
   Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
   

   View Slide

7. ΞδΣϯμ
   ڈ೥·ͰͷৼΓฦΓ
   ࣮ફ Kubernetes
   ղܾͨ͜͠ͱ / ՝୊΁ͷऔΓ૊Έ
   τΠϧ๾໓ӡಈ
   The UNIX Philosophy
   ϚϧνϚελʔ
   ࣗಈϞχλϦϯά
   

   View Slide

8. ڈ೥·ͰͷৼΓฦΓ
   

   View Slide

9. ͜Ε·ͰͷऔΓ૊Έ
   • Docker / Chef / Packer
   • Blue-Green Deployment
   • Terraform
   • CoreOS
   • cell
   

   View Slide

10. Docker / Chef / Packer
    • Docker image ͷதʹ Chef Λ࢖ͬͯΠϯϑϥΛߏங
    • Packer Ͱ AWS ͷ instance ͷ AMI Λ࡞Δ
    • Base ͱͳΔ template Λ༻ҙͯ͠ɺ৽͍͠αʔϏε͕ग़͖ͯͯ΋ྲྀ༻Ͱ
    ͖ΔΑ͏ʹͨ͠
    • ΞϓϦέʔγϣϯͷߏங෦෼Λ Code Խ
    

    View Slide

11. Blue-Green Deployment
    • /healthcheck ͰϨεϙϯεΛฦͨ͠Β੾Γସ͑Δ
    UBHT
    OBNFBQQ
    BQQSBJMT
    DPMPSHSFFO
    UBHT
    OBNFBQQ
    BQQSBJMT
    DPMPSCMVF
    DBQEFQMPZTXJUDI@DPMPSCMVF
    QPSU
    QPSU
    

    View Slide

12. Terraform
    • AWS(S3/RDS/ELB etc) / dnsimple Λૢ࡞
    IUUQTTQFBLFSEFDLDPNEUBOUFSSBGPSNBUXBOUFEMZUFDIDJSDMFOVNCFS
    

    View Slide

13. TAG ͱ AutoScaleGroup(ASG)
    • instance ͷ໾ׂΛɺ EC2 ͷ TAG Λ࢖ͬͯ؅ཧ
    • Կ୆ಈ͔͍͔ͨ͠ʁ => ASGͰઃఆ
    • ΠϯϑϥπʔϧΛ LaunchConfig ʹఆٛ
    • instance ্ཱ͕͕ͪΔͱ౰࣌ʹࣗ෼ʹ෇͚ΒΕͨ TAG Λ؍ͯɺඞཁͳ
    service Λ systemd ʹ൓ө͠ɺαʔϏεΛىಈ(cell)
    

    View Slide

14. cell
    • AWS TAG Λݩʹ systemd ʹαʔϏεΛొ࿥
    • αʔϏεͷ status ͕ ok ʹͳͬͨΒ DNS ࣗ਎Λ௥Ճ
    

    View Slide

15. ٕज़ελοΫ
    • ৽نαʔϏεΛ্ཱͪ͛Δࡍ͸͜ͷٕज़
    ελοΫΛશ෦༻ҙ͢Δ
    • Πϯϑϥͷ҆ఆԽͷͨΊͷ༷ʑπʔϧͱ
    ࢓૊ΈΛ༻ҙ
    • αʔόʔΛߏங͢Δͱ͜Ζ͔ΒσϓϩΠɺ
    cell ·ͰΛίʔυԽ
    Web
    Application
    Monitoring
    Logging
    Auto Scale
    Load Balancer
    Internal DNS
    

    View Slide

16. ೉ղͳखॱ
    

    View Slide

17. ࣮ફ Kubernetes
    

    View Slide

18. Kubernetes ಋೖ΁
    αʔϏε͕Ͱ͖Δ౓ʹҰݸͣͭαʔόʔΛ༻ҙͯ͠ߏங͢Δํ๏
    ෳ਺ͷαʔόʔΛҰͭͷେ͖ͳαʔόʔʹݟཱͯͯར༻͢Δ
    ಉ͡Α͏ͳΠϯϑϥ࡞ۀΛݮΒ͠ɺϦιʔεΛޮ཰తʹར༻͢Δ
    ࢀߟ: CodeZineʮKubernetesΛ࢖ͬͨมԽʹڧ͍Πϯϑϥ――WantedlyͷΠϯϑϥνʔϜ͕େ੾ʹ͍ͯ͠Δ͜
    ͱʯhttps://codezine.jp/article/detail/10357
    

    View Slide

19. The Twelve Factor App
    •12 ݸͷΞϓϦέʔγϣϯͷํ๏࿦ https://12factor.net/ja/
    •Herokuࣄଶ΋ίϯςφٕज़ɻDocker Λӡ༻͍ͯ͘͠࠷దͳํ๏࿦
    •σʔλετΞΛίϯςφʹؚΊͳ͍
    •։ൃ؀ڥͱຊ൪؀ڥͷ ࠩҟΛ؀ڥม਺Ͱઃఆ
    •ܧଓతσϓϩΠ
    

    View Slide

20. ͦͷଞ௥Ճϧʔϧ
    Docker ͷ image ͸ܰ͘
    CI Λ௨ͯ͠ɺimage ࡞੒ɺdeploy Λߦͳ͏
    ϦϙδτϦͱ namespace ͕ 1ର1 ʹͳΔΑ͏ʹ
    

    View Slide

21. docker image ͸ܰ͘
    https://github.com/koudaiii/sltd/blob/master/Dockerfile
    Golang Ͱ͋Ε͹όΠφϦΛΆΜى͖ͩͱখ͍͞
    1 FROM alpine:3.6
    2
    3 RUN apk add --no-cache --update ca-certificates
    4
    5 COPY bin/sltd /sltd
    6
    7 ENTRYPOINT ["/sltd"]
    8 CMD ["help"]
    

    View Slide

22. CI/CD
    .travis.yml ΑΓൈਮ
    58 script:
    59 - ./script/ci-test # Test
    60 - ./script/ci-build # docker build
    61 - ./script/ci-push # docker push
    62
    ɾɾɾɾɾɾɾɾɾɾɾ
    69 deploy:
    70 skip_cleanup: true
    71 provider: script
    72 script: ./script/ci-deploy # Release
    73 on:
    74 all_branches: true
    

    View Slide

23. View Slide

24. ϦϙδτϦ = namespace
    ֤ϦϙδτϦʹ /kubernetes Λ഑ஔ
    $ tree kubernetes/
    kubernetes/
    ᵓᴷᴷ jobs
    ᴹ ᵓᴷᴷ create-blog-job.yaml
    ᴹ ᵓᴷᴷ destroy-blog-job.yaml
    ᴹ ᵋᴷᴷ update-monthly-ranking-job.yaml
    ᵓᴷᴷ namespace.yaml
    ᵓᴷᴷ rails-docker-sample-run.yaml
    ᵓᴷᴷ rails-docker-sample-hpa.yaml
    ᵋᴷᴷ rails-docker-sample-svc.yaml
    

    View Slide

25. ϚχϑΣετϑΝΠϧ
    Namespace ઐ༻ͷ໊લۭؒ
    Deployment ΞϓϦέʔγϣϯͷ࣮ߦ
    HorizontalPodAutoscaler cpu/memory ͔Β pod Φʔτεέʔϧ
    Pod one-off ίϯςφ༻(rails c ౳Λߦͳ͏༻)
    Service SSL෇͖ELB
    Job oneshot Ͱ࣮ߦ༻(rake db:migrate ౳)
    CronJob Cron ܗࣜͰλεΫΛ࣮ߦ(rake task ౳)
    

    View Slide

26. Namespace
    ઐ༻ͷ໊લۭؒΛ࡞੒
    1 apiVersion: v1
    2 kind: Namespace
    3 metadata:
    4 name: hoge
    

    View Slide

27. Deployment
    1 apiVersion: extensions/v1beta1
    2 kind: Deployment
    ɾɾɾɾɾɾɾɾɾɾɾ
    9 spec:
    11 strategy:
    12 type: RollingUpdate
    13 rollingUpdate:
    14 maxSurge: 50% # Ұ౓൒෼ೖΕସ͑
    15 maxUnavailable: 0 # replicas Λҡ࣋
    ɾɾɾɾɾɾɾɾɾɾɾ
    23 spec:
    24 containers:
    25 - image: wantedly/python:latest
    26 name: python
    27 ports:
    28 - containerPort: 8000
    29 readinessProbe: # ready ͷఆٛ
    30 httpGet:
    31 path: /ping
    32 port: 8000
    33 initialDelaySeconds: 10
    34 timeoutSeconds: 1
    35 envFrom:
    36 - secretRef:
    37 name: dotenv
    38 command: ["script/server"]
    

    View Slide

28. HorizontalPodAutoscaler
    Pod ͷ autoscale
    1 apiVersion: autoscaling/v1
    2 kind: HorizontalPodAutoscaler
    3 metadata:
    4 name: hoge
    5 namespace: hoge
    6 spec:
    7 maxReplicas: 30
    8 minReplicas: 1
    9 scaleTargetRef:
    10 apiVersion: extensions/v1beta1
    11 kind: Deployment
    12 name: python
    13 targetCPUUtilizationPercentage: 50
    

    View Slide

29. Pod one-off Container(ྫ: rails c)
    1 apiVersion: v1
    2 kind: Pod
    3 metadata:
    4 name: {USER}
    5 namespace: hoge
    6 labels:
    7 role: console
    8 spec:
    ɾɾɾɾɾɾɾɾɾɾɾ
    11 containers:
    12 - image: wantedly/bash:latest
    13 imagePullPolicy: Always
    14 name: {USER}
    15 command:
    16 - bash
    17 stdin: true
    18 stdinOnce: true
    19 terminationMessagePath: /dev/termination-log
    20 tty: true
    21 envFrom:
    22 - secretRef:
    23 name: dotenv
    

    View Slide

30. Service SSLূ໌ॻ෇͖ ELB
    1 apiVersion: v1
    2 kind: Service
    3 metadata:
    4 name: hoge
    6 annotations:
    7 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:reagion:x:certificate/x
    8 service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    9 labels:
    10 name: hoge
    11 role: web
    12 spec:
    13 ports:
    14 - port: 443
    15 protocol: TCP
    16 targetPort: 8000
    17 selector:
    18 name: hoge
    19 role: web
    20 type: LoadBalancer
    

    View Slide

31. Job (ྫ rake db:migrate)
    1 apiVersion: batch/v1
    2 kind: Job
    ɾɾɾɾɾɾɾɾɾɾɾ
    16 spec:
    17 restartPolicy: Never
    18 containers:
    19 - name: db-migrate-[REPLACE_WITH_DATETIME]
    20 image: wantedly/rails:[REPLACE_WITH_TAG]
    21 command: ["bundle", "exec", "rake", "db:migrate"]
    22 envFrom:
    23 - secretRef:
    24 name: dotenv
    

    View Slide

32. CronJob (ྫ rake task)
    concurrencyPolicy Ͱલ࣮ߦ͕࢒ͬͨ৔߹ͷϋϯυϦϯά͕ग़དྷΔ
    1 apiVersion: batch/v2alpha1
    2 kind: CronJob
    3 metadata:
    4 namespace: hoge
    5 name: create-post
    6 labels:
    7 job: create-post
    9 spec:
    10 schedule: "*/30 * * * *"
    11 concurrencyPolicy: “Replace" # “Allow” or “Forbid”
    12 successfulJobsHistoryLimit: 5
    13 failedJobsHistoryLimit: 5
    ɾɾɾɾɾɾɾɾɾɾɾ
    

    View Slide

33. ղܾͨ͜͠ͱ / ՝୊΁ͷऔΓ૊Έ
    

    View Slide

34. ղܾͨ͜͠ͱ
    • ৽نαʔϏεͷग़͠΍͢͞
    • εέʔϧͷ͠΍͢͞
    

    View Slide

35. ৽نαʔϏεͷग़͠΍͢͞
    ͜Ε·Ͱ ݱࡏ
    

    View Slide

36. εέʔϧͷ͠΍͢͞
    •͜Ε·Ͱ͸ instance ϨϕϧͰɺεέʔϧΛߦͳ͏࢓૊ΈΛ࡞Δඞཁ͕͋ͬͨ
    •ϔϧενΣοΫ
    •αʔϏεσΟεΧόϦ
    •DNSొ࿥ / Load Balancer
    •ࣗݾम෮
    •Kubernetes ͔ΒɺԼهͷ఺͔Β༰қʹͳͬͨ
    •αʔϏεσΟεΧόϦͱDNSɺLoad Balancer ྑ͠ͳʹ͞Ε͍ͯΔ఺
    •ίϯςφϨϕϧͷεέʔϧग़དྷΔ఺
    $ kubectl autoscale deployment foo --min=2 —-max=10 —cpu-percent=50
    # શମͷCPUUtilization 50% ᮢ஋Ͱ૿ݮ
    $ kubectl scale --replicas=3 deployment foo # ͦͷ৔Ͱ3ͭʹ͢Δ
    

    View Slide

37. ՝୊ʹͳͬͨ͜ͱ
    1. खಈͰ΍͖ͬͯͨ͜ͱͷݶք
    2. Kubernetes ͷ Master ͕ SPOF
    3. ϞχλϦϯάͷෳࡶ͞
    

    View Slide

38. ՝୊΁ͷऔΓ૊Έ
    1. τΠϧ๾໓ӡಈ
    2. ϚϧνϚελʔ
    3. ࣗಈϞχλϦϯά
    

    View Slide

39. “τΠϧͱ͸ɺϓϩμΫγϣϯαʔϏεΛಈ࡞ͤ͞Δ͜ͱʹ
    ؔ܎͢Δ࡞ۀͰɺख࡞ۀͰ܁Γฦ͠ߦΘΕɺࣗಈԽ͢Δ͜ͱ
    ͕ՄೳͰ͋Γɺઓज़తͰ௕ظతͳՁ஋Λ࣋ͨͣɺ࡞ۀྔ͕αʔ
    Ϗεͷ੒௕ʹൺྫ͢Δͱ͍ͬͨ܏޲Λ࣋ͭ΋ͷͰ͢ɻ”
    ൈਮɿ: Betsy Beyer “SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά”
    

    View Slide

40. ҰͭҎ্౰ͯ͸·ΔͱτΠϧͷՄೳੑ༗
    • ख࡞ۀͰ͋Δ͜ͱ
    • ܁Γฦ͞ΕΔ͜ͱ
    • ࣗಈԽग़དྷΔ͜ͱ
    • ઓज़తͰ͋Δ͜ͱ(ઓུతͰ͋ͬͨΓ༧ଌʹجͮ͘΋ͷͰ͸ͳ͘ɺׂΓࠐΈͰ࢝·Γɺ໰୊ͳͲ
    ͕ੜͨ͜͡ͱ΁ͷରԠ)
    • ௕ظతͳՁ஋Λ࣋ͨͳ͍͜ͱ
    • αʔϏεͷ੒௕ʹରͯ͠O(n)Ͱ͋Δ͜ͱ
    

    View Slide

41. τΠϧ๾໓ӡಈ
    •test-build-push-release => CI ͷ template Խ
    •autoscale => HorizontalPodAutoscaler ϚχϑΣετΛॻ͘
    •migrate => Job ϚχϑΣετΛॻ͘
    •one-off => ઐ༻ͷ Pod ϚχϑΣετΛॻ͘
    •SSL ͷखܰ͞ => Service ϚχϑΣετʹ annotation Λ௥ه͢Δͱઃఆ͞ΕΔ
    •deploy & rollback => The UNIX Philosophy ଇͬͨπʔϧ࡞੒
    ࣮͸΄΅,VCFSOFUFTͰ࣮ݱ
    

    View Slide

42. The UNIX Philosophy
    

    View Slide

43. The UNIX Philosophy ൈਮ
    • Small is beautiful. খ͍͞΋ͷ͸ඒ͍͠
    • Make each program do one thing well. 1ͭͷϓϩάϥϜʹ͸1ͭͷ͜ͱΛ͏·͘΍
    ΒͤΔ
    • Choose portability over efficiency. ޮ཰ΑΓҠ২ੑΛ༏ઌ͢Δ
    • Use software leverage to your advantage. ιϑτ΢ΣΞΛᑏࢠ(ͯ͜)ͱͯ͠࢖͏
    • Avoid captive user interfaces. ա౉ͷର࿩తΠϯλʔϑΣʔεΛආ͚Δ
    

    View Slide

44. Server Tools ͱ Client Tool
    • Ұͭͷ͜ͱΛ্ख͘΍Δπʔϧ࡞Γɺᑏࢠͱͯ͠ར༻ => ͍ͭͰ΋ަ׵Մೳ
    • ΫϥΠΞϯτଆ͸׳Ε਌͠Μͩύοέʔδ؅ཧ => homebrew Ͱ഑৴
    • ΫϥΠΞϯτଆ͸πʔϧΛݺͼग़ͨ͢Ίͷϥούʔʹઐ೦
    • ΫϥΠΞϯτଆʹग़དྷΔݶΓϩδοΫΛೖΕͳ͍ => ॲཧ͸ server ্
    • ωοτ੾அΛ૝ఆ͢Δ => server ͱ container ͸ҡ࣋
    

    View Slide

45. homebrew
    Golang πʔϧΛ homebrew Ͱ binary Λ഑৴
    ΫϩείϯύΠϥͰOSґଘ͕ۃྗͳ͘πʔϧ͕࢖͑ΔΑ͏ʹ
    

    View Slide

46. kube ࣾ಺πʔϧ
    

    View Slide

47. खಈσϓϩΠ
    ϚχϑΣετϑΝΠϧʹ annotations Λ௥ه͢Δ͜ͱͰखಈσϓϩΠ͕Ͱ͖Δ
    k8ship deploy [BRANCH|COMMIT_SHA1]
    1 apiVersion: extensions/v1beta1
    2 kind: Deployment
    3 metadata:
    4 name: rails
    5 namespace: rails
    6 labels:
    7 name: rails
    8 role: web
    9 annotations:
    10 wantedly.com/deploy-target: “true" # ௥ه
    11 wantedly.com/deploy-target-container: rails # ௥ه
    12 wantedly.com/github: rails=wantedly/rails # ௥ه
    https://github.com/dtan4/k8ship
    

    View Slide

48. kube deploy ίϚϯυ
    -dry-run ΦϓγϣϯͰ diff Λग़͢
    merge ޙ೚ҙͷλΠϛϯάͰ deploy ͍ͨ͠৔߹ʹར༻
    $ kube qa deploy master --dry-run
    [dry-run] deploy to (deployment: "rails", container: "rails")
    [dry-run] before: wantedly/rails:d1cb608ee61cb18f8c397c2e27576573879b2fcf
    [dry-run] after: wantedly/rails:cb19269526f946222110fbdfb3d107c4b5a18fe7
    

    View Slide

49. ϚϧνϚελʔ
    

    View Slide

50. ϚϧνϚελʔ
    •Kubernetes ͷ version up ͸Ͳ͏͢Δͷʁ
    •Ϛελʔ͕ࢭ·ΔͱͲ͏ͳΔͷʁ
    •Ϋϥελͷఀࢭͳ͘ kops Λ࢖ͬͯ Rolling upgrade Ͱ͖ΔΑ͏ʹมߋ
    •ϚελʔΛࡾ୆༻ҙ͠ɺϚϧνϚελʔߏ੒ʹ͔͑ͯ SPOF վળ
    

    View Slide

51. ࣗಈϞχλϦϯά
    

    View Slide

52. ࣗಈϞχλϦϯά
    • 1ͭ1ͭઃఆ͢ΔͷͰ͸ͳ͘ɺ൚༻తʹద༻͞ΕΔΑ͏ʹ͢Δ
    • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}}
    • request ͱ status code औಘ
    • μογϡϘʔυʹ൓ө
    • Deployment Ͱఆٛͨ͠਺
    

    View Slide

53. ࣗಈͰಉ͡඼࣭ͷϞχλϦϯάΛ໨ࢦ͢
    https://www.datadoghq.com/blog/monitoring-101-collecting-data/
    http://qiita.com/koudaiii/items/bc89368e1279649f2498
    

    View Slide

54. Dashboard
    

    View Slide

55. ϞχλϦϯάͰ࢖༻͍ͯ͠Δ΋ͷ
    • kelseyhightower/konfd => PostgresqlͳͲͷ৘ใΛϦϙδτϦʹؚΊͳ͍࣌ʹར༻
    • kubernetes/kube-state-metrics => deployment ౳ͷ available Λऔಘ
    • datadog/docker-dd-agent => docker ༻ɺ web ༻ɺ db ༻ͱͯ͠໾ׂผͰ഑උ
    • koudaiii/sltd => Service ͷ metadata Λݩʹ ELB ͷ tag ʹ௥Ճ
    • logentries => ΫϥελશͯͷϩάΛૹΔ
    

    View Slide

56. ੒௕ਪҠ
    

    View Slide

57. αʔϏεਪҠ (2017/04Ἤ)
    Service 9 => 60
    Container 700+ => 1,500 - 2,500+
    

    View Slide

58. ·ͱΊ
    • kubernetes ͸ɺΠϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬʹͳͬͨ
    • ղܾͨ͜͠ͱ
    • ৽نαʔϏεͷग़͠΍͢͞
    • εέʔϧͷ͠΍͢͞
    • ՝୊͔ΒऔΓ૊Μͩ͜ͱ
    • खಈͰ΍͖ͬͯͨ͜ͱͷݶք => τΠϧ๾໓ӡಈ
    • Ϛελʔ͕ SPOF => ϚϧνϚελʔԽ
    • ϞχλϦϯάͷෳࡶ͞ => ࣗಈϞχλϦϯά
    

    View Slide

59. Kubernetes Λར༻ͨ݁͠Ռ
    Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
    

    View Slide

60. IUUQTXXXXBOUFEMZDPNQSPKFDUT
    

    View Slide