$30 off During Our Annual Pro Sale. View Details »

Kubernetes で解決したこと新たに出てきた課題 /#jtf2017

Kubernetes で解決したこと新たに出てきた課題 /#jtf2017

1年を通して Kubernetes で解決したこと新たに出てきた課題をどう立ち向かったを発表しました

at July Tech Festa 2017

Kodai Sakabe

August 27, 2017
Tweet

More Decks by Kodai Sakabe

Other Decks in Technology

Transcript

  1. ,PEBJ4BLBCF!LPVEBJJJ8BOUFEMZ *OD
    Kubernetes Ͱղܾͨ͜͠ͱ
    ৽ͨʹग़͖ͯͨ՝୊
    July Tech Festa 2017

    View Slide

  2. ABOUT ME
    — Software Engineer —
    2010/4- TIS
    ࡔ෦ ޿େ (KODAI SAKABE)
    2015/8- Wantedly
    @koudaiii
    Site: https://koudaiii.com

    View Slide

  3. View Slide

  4. Kubernetes
    ෳ਺ͷϗετؒͰίϯςφԽ͞ΕͨΞϓϦέʔγϣϯΛ؅ཧ͢ΔͨΊͷOSS
    ΞϓϦέʔγϣϯͷdeployment, maintenance, scalingͷͨΊͷجຊతͳϝΧχζϜΛఏڙ
    https://github.com/kubernetes/kubernetes

    View Slide

  5. Kubernetes Λར༻ͨ݁͠Ռ
    • ղܾͨ͜͠ͱ
    • ৽نαʔϏεͷग़͠΍͢͞
    • εέʔϧͷ͠΍͢͞
    • ՝୊ʹͳͬͨ͜ͱ
    • खಈͰ΍͖ͬͯͨ͜ͱͷݶք
    • Kubernetes ͷ Master ͕ SPOF
    • ϞχλϦϯάͷෳࡶ͞

    View Slide

  6. Kubernetes Λར༻ͨ݁͠Ռ
    Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ

    View Slide

  7. ΞδΣϯμ
    ڈ೥·ͰͷৼΓฦΓ
    ࣮ફ Kubernetes
    ղܾͨ͜͠ͱ / ՝୊΁ͷऔΓ૊Έ
    τΠϧ๾໓ӡಈ
    The UNIX Philosophy
    ϚϧνϚελʔ
    ࣗಈϞχλϦϯά

    View Slide

  8. ڈ೥·ͰͷৼΓฦΓ

    View Slide

  9. ͜Ε·ͰͷऔΓ૊Έ
    • Docker / Chef / Packer
    • Blue-Green Deployment
    • Terraform
    • CoreOS
    • cell

    View Slide

  10. Docker / Chef / Packer
    • Docker image ͷதʹ Chef Λ࢖ͬͯΠϯϑϥΛߏங
    • Packer Ͱ AWS ͷ instance ͷ AMI Λ࡞Δ
    • Base ͱͳΔ template Λ༻ҙͯ͠ɺ৽͍͠αʔϏε͕ग़͖ͯͯ΋ྲྀ༻Ͱ
    ͖ΔΑ͏ʹͨ͠
    • ΞϓϦέʔγϣϯͷߏங෦෼Λ Code Խ

    View Slide

  11. Blue-Green Deployment
    • /healthcheck ͰϨεϙϯεΛฦͨ͠Β੾Γସ͑Δ
    UBHT
    OBNFBQQ
    BQQSBJMT
    DPMPSHSFFO
    UBHT
    OBNFBQQ
    BQQSBJMT
    DPMPSCMVF
    DBQEFQMPZTXJUDI@DPMPSCMVF
    QPSU
    QPSU

    View Slide

  12. Terraform
    • AWS(S3/RDS/ELB etc) / dnsimple Λૢ࡞
    IUUQTTQFBLFSEFDLDPNEUBOUFSSBGPSNBUXBOUFEMZUFDIDJSDMFOVNCFS

    View Slide

  13. TAG ͱ AutoScaleGroup(ASG)
    • instance ͷ໾ׂΛɺ EC2 ͷ TAG Λ࢖ͬͯ؅ཧ
    • Կ୆ಈ͔͍͔ͨ͠ʁ => ASGͰઃఆ
    • ΠϯϑϥπʔϧΛ LaunchConfig ʹఆٛ
    • instance ্ཱ͕͕ͪΔͱ౰࣌ʹࣗ෼ʹ෇͚ΒΕͨ TAG Λ؍ͯɺඞཁͳ
    service Λ systemd ʹ൓ө͠ɺαʔϏεΛىಈ(cell)

    View Slide

  14. cell
    • AWS TAG Λݩʹ systemd ʹαʔϏεΛొ࿥
    • αʔϏεͷ status ͕ ok ʹͳͬͨΒ DNS ࣗ਎Λ௥Ճ

    View Slide

  15. ٕज़ελοΫ
    • ৽نαʔϏεΛ্ཱͪ͛Δࡍ͸͜ͷٕज़
    ελοΫΛશ෦༻ҙ͢Δ
    • Πϯϑϥͷ҆ఆԽͷͨΊͷ༷ʑπʔϧͱ
    ࢓૊ΈΛ༻ҙ
    • αʔόʔΛߏங͢Δͱ͜Ζ͔ΒσϓϩΠɺ
    cell ·ͰΛίʔυԽ
    Web
    Application
    Monitoring
    Logging
    Auto Scale
    Load Balancer
    Internal DNS

    View Slide

  16. ೉ղͳखॱ

    View Slide

  17. ࣮ફ Kubernetes

    View Slide

  18. Kubernetes ಋೖ΁
    αʔϏε͕Ͱ͖Δ౓ʹҰݸͣͭαʔόʔΛ༻ҙͯ͠ߏங͢Δํ๏
    ෳ਺ͷαʔόʔΛҰͭͷେ͖ͳαʔόʔʹݟཱͯͯར༻͢Δ
    ಉ͡Α͏ͳΠϯϑϥ࡞ۀΛݮΒ͠ɺϦιʔεΛޮ཰తʹར༻͢Δ
    ࢀߟ: CodeZineʮKubernetesΛ࢖ͬͨมԽʹڧ͍Πϯϑϥ――WantedlyͷΠϯϑϥνʔϜ͕େ੾ʹ͍ͯ͠Δ͜
    ͱʯhttps://codezine.jp/article/detail/10357

    View Slide

  19. The Twelve Factor App
    •12 ݸͷΞϓϦέʔγϣϯͷํ๏࿦ https://12factor.net/ja/
    •Herokuࣄଶ΋ίϯςφٕज़ɻDocker Λӡ༻͍ͯ͘͠࠷దͳํ๏࿦
    •σʔλετΞΛίϯςφʹؚΊͳ͍
    •։ൃ؀ڥͱຊ൪؀ڥͷ ࠩҟΛ؀ڥม਺Ͱઃఆ
    •ܧଓతσϓϩΠ

    View Slide

  20. ͦͷଞ௥Ճϧʔϧ
    Docker ͷ image ͸ܰ͘
    CI Λ௨ͯ͠ɺimage ࡞੒ɺdeploy Λߦͳ͏
    ϦϙδτϦͱ namespace ͕ 1ର1 ʹͳΔΑ͏ʹ

    View Slide

  21. docker image ͸ܰ͘
    https://github.com/koudaiii/sltd/blob/master/Dockerfile
    Golang Ͱ͋Ε͹όΠφϦΛΆΜى͖ͩͱখ͍͞
    1 FROM alpine:3.6
    2
    3 RUN apk add --no-cache --update ca-certificates
    4
    5 COPY bin/sltd /sltd
    6
    7 ENTRYPOINT ["/sltd"]
    8 CMD ["help"]

    View Slide

  22. CI/CD
    .travis.yml ΑΓൈਮ
    58 script:
    59 - ./script/ci-test # Test
    60 - ./script/ci-build # docker build
    61 - ./script/ci-push # docker push
    62
    ɾɾɾɾɾɾɾɾɾɾɾ
    69 deploy:
    70 skip_cleanup: true
    71 provider: script
    72 script: ./script/ci-deploy # Release
    73 on:
    74 all_branches: true

    View Slide

  23. View Slide

  24. ϦϙδτϦ = namespace
    ֤ϦϙδτϦʹ /kubernetes Λ഑ஔ
    $ tree kubernetes/
    kubernetes/
    ᵓᴷᴷ jobs
    ᴹ ᵓᴷᴷ create-blog-job.yaml
    ᴹ ᵓᴷᴷ destroy-blog-job.yaml
    ᴹ ᵋᴷᴷ update-monthly-ranking-job.yaml
    ᵓᴷᴷ namespace.yaml
    ᵓᴷᴷ rails-docker-sample-run.yaml
    ᵓᴷᴷ rails-docker-sample-hpa.yaml
    ᵋᴷᴷ rails-docker-sample-svc.yaml

    View Slide

  25. ϚχϑΣετϑΝΠϧ
    Namespace ઐ༻ͷ໊લۭؒ
    Deployment ΞϓϦέʔγϣϯͷ࣮ߦ
    HorizontalPodAutoscaler cpu/memory ͔Β pod Φʔτεέʔϧ
    Pod one-off ίϯςφ༻(rails c ౳Λߦͳ͏༻)
    Service SSL෇͖ELB
    Job oneshot Ͱ࣮ߦ༻(rake db:migrate ౳)
    CronJob Cron ܗࣜͰλεΫΛ࣮ߦ(rake task ౳)

    View Slide

  26. Namespace
    ઐ༻ͷ໊લۭؒΛ࡞੒
    1 apiVersion: v1
    2 kind: Namespace
    3 metadata:
    4 name: hoge

    View Slide

  27. Deployment
    1 apiVersion: extensions/v1beta1
    2 kind: Deployment
    ɾɾɾɾɾɾɾɾɾɾɾ
    9 spec:
    11 strategy:
    12 type: RollingUpdate
    13 rollingUpdate:
    14 maxSurge: 50% # Ұ౓൒෼ೖΕସ͑
    15 maxUnavailable: 0 # replicas Λҡ࣋
    ɾɾɾɾɾɾɾɾɾɾɾ
    23 spec:
    24 containers:
    25 - image: wantedly/python:latest
    26 name: python
    27 ports:
    28 - containerPort: 8000
    29 readinessProbe: # ready ͷఆٛ
    30 httpGet:
    31 path: /ping
    32 port: 8000
    33 initialDelaySeconds: 10
    34 timeoutSeconds: 1
    35 envFrom:
    36 - secretRef:
    37 name: dotenv
    38 command: ["script/server"]

    View Slide

  28. HorizontalPodAutoscaler
    Pod ͷ autoscale
    1 apiVersion: autoscaling/v1
    2 kind: HorizontalPodAutoscaler
    3 metadata:
    4 name: hoge
    5 namespace: hoge
    6 spec:
    7 maxReplicas: 30
    8 minReplicas: 1
    9 scaleTargetRef:
    10 apiVersion: extensions/v1beta1
    11 kind: Deployment
    12 name: python
    13 targetCPUUtilizationPercentage: 50

    View Slide

  29. Pod one-off Container(ྫ: rails c)
    1 apiVersion: v1
    2 kind: Pod
    3 metadata:
    4 name: {USER}
    5 namespace: hoge
    6 labels:
    7 role: console
    8 spec:
    ɾɾɾɾɾɾɾɾɾɾɾ
    11 containers:
    12 - image: wantedly/bash:latest
    13 imagePullPolicy: Always
    14 name: {USER}
    15 command:
    16 - bash
    17 stdin: true
    18 stdinOnce: true
    19 terminationMessagePath: /dev/termination-log
    20 tty: true
    21 envFrom:
    22 - secretRef:
    23 name: dotenv

    View Slide

  30. Service SSLূ໌ॻ෇͖ ELB
    1 apiVersion: v1
    2 kind: Service
    3 metadata:
    4 name: hoge
    6 annotations:
    7 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:reagion:x:certificate/x
    8 service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    9 labels:
    10 name: hoge
    11 role: web
    12 spec:
    13 ports:
    14 - port: 443
    15 protocol: TCP
    16 targetPort: 8000
    17 selector:
    18 name: hoge
    19 role: web
    20 type: LoadBalancer

    View Slide

  31. Job (ྫ rake db:migrate)
    1 apiVersion: batch/v1
    2 kind: Job
    ɾɾɾɾɾɾɾɾɾɾɾ
    16 spec:
    17 restartPolicy: Never
    18 containers:
    19 - name: db-migrate-[REPLACE_WITH_DATETIME]
    20 image: wantedly/rails:[REPLACE_WITH_TAG]
    21 command: ["bundle", "exec", "rake", "db:migrate"]
    22 envFrom:
    23 - secretRef:
    24 name: dotenv

    View Slide

  32. CronJob (ྫ rake task)
    concurrencyPolicy Ͱલ࣮ߦ͕࢒ͬͨ৔߹ͷϋϯυϦϯά͕ग़དྷΔ
    1 apiVersion: batch/v2alpha1
    2 kind: CronJob
    3 metadata:
    4 namespace: hoge
    5 name: create-post
    6 labels:
    7 job: create-post
    9 spec:
    10 schedule: "*/30 * * * *"
    11 concurrencyPolicy: “Replace" # “Allow” or “Forbid”
    12 successfulJobsHistoryLimit: 5
    13 failedJobsHistoryLimit: 5
    ɾɾɾɾɾɾɾɾɾɾɾ

    View Slide

  33. ղܾͨ͜͠ͱ / ՝୊΁ͷऔΓ૊Έ

    View Slide

  34. ղܾͨ͜͠ͱ
    • ৽نαʔϏεͷग़͠΍͢͞
    • εέʔϧͷ͠΍͢͞

    View Slide

  35. ৽نαʔϏεͷग़͠΍͢͞
    ͜Ε·Ͱ ݱࡏ

    View Slide

  36. εέʔϧͷ͠΍͢͞
    •͜Ε·Ͱ͸ instance ϨϕϧͰɺεέʔϧΛߦͳ͏࢓૊ΈΛ࡞Δඞཁ͕͋ͬͨ
    •ϔϧενΣοΫ
    •αʔϏεσΟεΧόϦ
    •DNSొ࿥ / Load Balancer
    •ࣗݾम෮
    •Kubernetes ͔ΒɺԼهͷ఺͔Β༰қʹͳͬͨ
    •αʔϏεσΟεΧόϦͱDNSɺLoad Balancer ྑ͠ͳʹ͞Ε͍ͯΔ఺
    •ίϯςφϨϕϧͷεέʔϧग़དྷΔ఺
    $ kubectl autoscale deployment foo --min=2 —-max=10 —cpu-percent=50
    # શମͷCPUUtilization 50% ᮢ஋Ͱ૿ݮ
    $ kubectl scale --replicas=3 deployment foo # ͦͷ৔Ͱ3ͭʹ͢Δ

    View Slide

  37. ՝୊ʹͳͬͨ͜ͱ
    1. खಈͰ΍͖ͬͯͨ͜ͱͷݶք
    2. Kubernetes ͷ Master ͕ SPOF
    3. ϞχλϦϯάͷෳࡶ͞

    View Slide

  38. ՝୊΁ͷऔΓ૊Έ
    1. τΠϧ๾໓ӡಈ
    2. ϚϧνϚελʔ
    3. ࣗಈϞχλϦϯά

    View Slide

  39. “τΠϧͱ͸ɺϓϩμΫγϣϯαʔϏεΛಈ࡞ͤ͞Δ͜ͱʹ
    ؔ܎͢Δ࡞ۀͰɺख࡞ۀͰ܁Γฦ͠ߦΘΕɺࣗಈԽ͢Δ͜ͱ
    ͕ՄೳͰ͋Γɺઓज़తͰ௕ظతͳՁ஋Λ࣋ͨͣɺ࡞ۀྔ͕αʔ
    Ϗεͷ੒௕ʹൺྫ͢Δͱ͍ͬͨ܏޲Λ࣋ͭ΋ͷͰ͢ɻ”
    ൈਮɿ: Betsy Beyer “SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά”

    View Slide

  40. ҰͭҎ্౰ͯ͸·ΔͱτΠϧͷՄೳੑ༗
    • ख࡞ۀͰ͋Δ͜ͱ
    • ܁Γฦ͞ΕΔ͜ͱ
    • ࣗಈԽग़དྷΔ͜ͱ
    • ઓज़తͰ͋Δ͜ͱ(ઓུతͰ͋ͬͨΓ༧ଌʹجͮ͘΋ͷͰ͸ͳ͘ɺׂΓࠐΈͰ࢝·Γɺ໰୊ͳͲ
    ͕ੜͨ͜͡ͱ΁ͷରԠ)
    • ௕ظతͳՁ஋Λ࣋ͨͳ͍͜ͱ
    • αʔϏεͷ੒௕ʹରͯ͠O(n)Ͱ͋Δ͜ͱ

    View Slide

  41. τΠϧ๾໓ӡಈ
    •test-build-push-release => CI ͷ template Խ
    •autoscale => HorizontalPodAutoscaler ϚχϑΣετΛॻ͘
    •migrate => Job ϚχϑΣετΛॻ͘
    •one-off => ઐ༻ͷ Pod ϚχϑΣετΛॻ͘
    •SSL ͷखܰ͞ => Service ϚχϑΣετʹ annotation Λ௥ه͢Δͱઃఆ͞ΕΔ
    •deploy & rollback => The UNIX Philosophy ଇͬͨπʔϧ࡞੒
    ࣮͸΄΅,VCFSOFUFTͰ࣮ݱ

    View Slide

  42. The UNIX Philosophy

    View Slide

  43. The UNIX Philosophy ൈਮ
    • Small is beautiful. খ͍͞΋ͷ͸ඒ͍͠
    • Make each program do one thing well. 1ͭͷϓϩάϥϜʹ͸1ͭͷ͜ͱΛ͏·͘΍
    ΒͤΔ
    • Choose portability over efficiency. ޮ཰ΑΓҠ২ੑΛ༏ઌ͢Δ
    • Use software leverage to your advantage. ιϑτ΢ΣΞΛᑏࢠ(ͯ͜)ͱͯ͠࢖͏
    • Avoid captive user interfaces. ա౉ͷର࿩తΠϯλʔϑΣʔεΛආ͚Δ

    View Slide

  44. Server Tools ͱ Client Tool
    • Ұͭͷ͜ͱΛ্ख͘΍Δπʔϧ࡞Γɺᑏࢠͱͯ͠ར༻ => ͍ͭͰ΋ަ׵Մೳ
    • ΫϥΠΞϯτଆ͸׳Ε਌͠Μͩύοέʔδ؅ཧ => homebrew Ͱ഑৴
    • ΫϥΠΞϯτଆ͸πʔϧΛݺͼग़ͨ͢Ίͷϥούʔʹઐ೦
    • ΫϥΠΞϯτଆʹग़དྷΔݶΓϩδοΫΛೖΕͳ͍ => ॲཧ͸ server ্
    • ωοτ੾அΛ૝ఆ͢Δ => server ͱ container ͸ҡ࣋

    View Slide

  45. homebrew
    Golang πʔϧΛ homebrew Ͱ binary Λ഑৴
    ΫϩείϯύΠϥͰOSґଘ͕ۃྗͳ͘πʔϧ͕࢖͑ΔΑ͏ʹ

    View Slide

  46. kube ࣾ಺πʔϧ

    View Slide

  47. खಈσϓϩΠ
    ϚχϑΣετϑΝΠϧʹ annotations Λ௥ه͢Δ͜ͱͰखಈσϓϩΠ͕Ͱ͖Δ
    k8ship deploy [BRANCH|COMMIT_SHA1]
    1 apiVersion: extensions/v1beta1
    2 kind: Deployment
    3 metadata:
    4 name: rails
    5 namespace: rails
    6 labels:
    7 name: rails
    8 role: web
    9 annotations:
    10 wantedly.com/deploy-target: “true" # ௥ه
    11 wantedly.com/deploy-target-container: rails # ௥ه
    12 wantedly.com/github: rails=wantedly/rails # ௥ه
    https://github.com/dtan4/k8ship

    View Slide

  48. kube deploy ίϚϯυ
    -dry-run ΦϓγϣϯͰ diff Λग़͢
    merge ޙ೚ҙͷλΠϛϯάͰ deploy ͍ͨ͠৔߹ʹར༻
    $ kube qa deploy master --dry-run
    [dry-run] deploy to (deployment: "rails", container: "rails")
    [dry-run] before: wantedly/rails:d1cb608ee61cb18f8c397c2e27576573879b2fcf
    [dry-run] after: wantedly/rails:cb19269526f946222110fbdfb3d107c4b5a18fe7

    View Slide

  49. ϚϧνϚελʔ

    View Slide

  50. ϚϧνϚελʔ
    •Kubernetes ͷ version up ͸Ͳ͏͢Δͷʁ
    •Ϛελʔ͕ࢭ·ΔͱͲ͏ͳΔͷʁ
    •Ϋϥελͷఀࢭͳ͘ kops Λ࢖ͬͯ Rolling upgrade Ͱ͖ΔΑ͏ʹมߋ
    •ϚελʔΛࡾ୆༻ҙ͠ɺϚϧνϚελʔߏ੒ʹ͔͑ͯ SPOF վળ

    View Slide

  51. ࣗಈϞχλϦϯά

    View Slide

  52. ࣗಈϞχλϦϯά
    • 1ͭ1ͭઃఆ͢ΔͷͰ͸ͳ͘ɺ൚༻తʹద༻͞ΕΔΑ͏ʹ͢Δ
    • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}}
    • request ͱ status code औಘ
    • μογϡϘʔυʹ൓ө
    • Deployment Ͱఆٛͨ͠਺

    View Slide

  53. ࣗಈͰಉ͡඼࣭ͷϞχλϦϯάΛ໨ࢦ͢
    https://www.datadoghq.com/blog/monitoring-101-collecting-data/
    http://qiita.com/koudaiii/items/bc89368e1279649f2498

    View Slide

  54. Dashboard

    View Slide

  55. ϞχλϦϯάͰ࢖༻͍ͯ͠Δ΋ͷ
    • kelseyhightower/konfd => PostgresqlͳͲͷ৘ใΛϦϙδτϦʹؚΊͳ͍࣌ʹར༻
    • kubernetes/kube-state-metrics => deployment ౳ͷ available Λऔಘ
    • datadog/docker-dd-agent => docker ༻ɺ web ༻ɺ db ༻ͱͯ͠໾ׂผͰ഑උ
    • koudaiii/sltd => Service ͷ metadata Λݩʹ ELB ͷ tag ʹ௥Ճ
    • logentries => ΫϥελશͯͷϩάΛૹΔ

    View Slide

  56. ੒௕ਪҠ

    View Slide

  57. αʔϏεਪҠ (2017/04Ἤ)
    Service 9 => 60
    Container 700+ => 1,500 - 2,500+

    View Slide

  58. ·ͱΊ
    • kubernetes ͸ɺΠϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬʹͳͬͨ
    • ղܾͨ͜͠ͱ
    • ৽نαʔϏεͷग़͠΍͢͞
    • εέʔϧͷ͠΍͢͞
    • ՝୊͔ΒऔΓ૊Μͩ͜ͱ
    • खಈͰ΍͖ͬͯͨ͜ͱͷݶք => τΠϧ๾໓ӡಈ
    • Ϛελʔ͕ SPOF => ϚϧνϚελʔԽ
    • ϞχλϦϯάͷෳࡶ͞ => ࣗಈϞχλϦϯά

    View Slide

  59. Kubernetes Λར༻ͨ݁͠Ռ
    Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ

    View Slide

  60. IUUQTXXXXBOUFEMZDPNQSPKFDUT

    View Slide