Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Kubernetes in Production / #k8sjp

Using Kubernetes in Production / #k8sjp

Kubernetes Meetup Tokyo #5
https://k8sjp.connpass.com/event/56945/

Kubernetes を Production で運用してきたことについて話をしました。

Kodai Sakabe

June 28, 2017
Tweet

More Decks by Kodai Sakabe

Other Decks in Technology

Transcript

  1. ,PEBJ4BLBCF!LPVEBJJJ8BOUFEMZ *OD
    Using Kubernetes in Production

    View full-size slide

  2. ABOUT ME
    — Software Engineer —
    2010/4- TIS
    ࡔ෦ ޿େ (KODAI SAKABE)
    2015/8- Wantedly
    @koudaiii
    Site: https://koudaiii.com

    View full-size slide

  3. ΰʔϧ
    • Kubernetes Λ Production Ͱ࢖͏৔߹ʹ΍ͬͨ͜ͱ
    • ෳ਺Ϋϥελͱ Multi Master ͷ༻ҙ
    • खؒʹͳΔϞχλϦϯά΍؂ࢹͷઃఆΛࣗಈԽ
    • CI/CD ͷϧʔϧԽ

    View full-size slide

  4. αʔϏεͷਪҠ
    WantedlyશମͷαʔϏε਺ ͱ Kubernetes ͷ Namespace ͷ਺

    View full-size slide

  5. ΞδΣϯμ
    1೥ؒͷऔΓ૊Έͱ࡞੒ͨ͠πʔϧ
    CI/CD ͷϧʔϧԽ
    ϞχλϦϯά

    View full-size slide

  6. 1 ೥ؒͷऔΓ૊Έ

    View full-size slide

  7. Kubernetes v1.1 (2015/11 - 2016/04)
    •৽ن Web Application Λߏங͢Δͷʹखؒͱ࡞ۀऀͷଐਓԽ
    •ʮWantedly ্ʹ͍Δ User ͷܨ͕ΓΛߴ଎ʹฦ͢αʔϏεΛ࡞Δʯ͔Β
    ελʔτ
    •࣮ݧతʹ Go ͷΞϓϦέʔγϣϯΛ Kubernetes ্ʹཱͯΔ
    •Kubernetes ͷݕূ͕ελʔτͨ͠

    View full-size slide

  8. Production?
    •Production ʹࡌͤΔલʹҎԼͷ2ͭʹ͍ͭͯௐࠪΛߦ͍ͬͯͬͨ
    •ͦ΋ͦ΋ΫϥελͷϝϦοτͬͯͳΜͩΖ͏͔ʁ
    •Ϋϥελͦͷ΋ͷ͕յΕͨ࣌Ͳ͏͢Ε͹ྑ͍ͷ͔ʁ

    View full-size slide

  9. Production?
    •Ϋϥελ͕յΕͨ৔߹ɺ࠷ऴతʹݩʹ໭ͨ͢ΊʹͲ͏͢Ε͹ྑ͍ͷ͔?
    •VPC ຖʹΫϥελϦϯάΛ࡞੒Ͱ͖ΔΑ͏ʹ͢Δ => ࠷ѱҠಈՄೳ
    •1 Host : 1 ΞϓϦέʔγϣϯ ͱ 1ΫϥελϦϯά : ଞΞϓϦέʔγϣϯͷҧ͍
    •ޮ཰Խ͢Δ͜ͱʹΑΔϝϦοτͱσϝϦοτͷ·ͱΊ
    •Ϋϥελࣄଶͷ؂ࢹͱ؅ཧ͸Ͳ͏͢Δ΂͖͔
    •Datadog on Kubernetes(dd-agent)
    •Deployment ΍ Replication Controller Ͱ࡞੒͞Εͨ Pod
    • kubernetes/kube-state-metric Λ࢖ͬͯ unavailable ͷ਺ͱ available ͷ਺ΛݟΔ

    View full-size slide

  10. Kubernetes ಋೖظ v1.2.3 (2016/4- 2016/10)
    •Production Ͱ·ͣ͸ microservice ͷӡ༻͔Β࢝ΊΔ
    •API ʹ response ͕ͳͯ͘΋ຊମͷαʔϏε͕མͪͳ͍ઃܭ
    •·ͣ͸ΠϯϑϥνʔϜͰશͯϝϯςφϯε͢Δ
    •νʔϜͰ࢖͍ʹ͍͘෦෼Λվળ͢Δ
    •҆શੑͱརศੑΛߟ͑ͯɺ `repository == namespace` ͱ͍͏ܗͰӡ༻Λ։࢝
    •Datadog Λ DaemonSet ʹೖΕͯར༻։࢝
    •kubernetes ͷ Log ͱͯ͠ Logentries ΁ૹΔΑ͏ʹ DaemonSet ʹೖΕͯར༻։࢝

    View full-size slide

  11. ͜͜·Ͱ࡞੒ͨ͠πʔϧ
    •dtan4/k8sec
    •؀ڥม਺ʹ secret Λར༻͢Δࡍʹߋ৽ͱϦετͰ֬ೝ͢Δͷ͕खؒͩͬͨ
    •`heroku config` ͷΑ͏ͳπʔϧ
    •kube(ࣾ಺πʔϧ), wantedly/slack-notifier
    •heroku ίϚϯυͱಉ͡Α͏ʹͦͷϦϙδτϦͰͷ࡞ۀ͸ͦͷ namespace ͕σϑΥϧτͰೖΔ
    •࡞੒ͨ͠πʔϧͱ kubectl Λͦͷ··ར༻Մೳ
    •wantedly/dockertags
    •ίϚϯυϥΠϯ͔Β docker ͷ tag ҰཡΛݟΕΔΑ͏ʹͨ͠

    View full-size slide

  12. k8sec
    $ k8sec set test key=value key2=value2 --namespace=default
    test
    $ k8sec list test --namespace=default
    NAME TYPE KEY VALUE
    test Opaque key2 "value2"
    test Opaque key "value"
    $ k8sec set test key=hoge --namespace=default
    test
    $ k8sec list test --namespace=default
    NAME TYPE KEY VALUE
    test Opaque key "hoge"
    test Opaque key2 "value2"

    View full-size slide

  13. dockertags
    $ bin/dockertags quay.io/koudaiii/sltd
    latest
    v0.1.0

    View full-size slide

  14. kube
    $ kube
    CLI tool to execute kubectl over SSH
    Usage:
    kube [command]
    Available Commands:
    ɾɾɾɾ
    create kubectl create
    delete kubectl delete
    describe kubectl describe
    ɾɾɾɾ
    dotenv
    enable
    generate Generate manifest file
    get kubectl get
    prod Use Production cluster
    ps Conatainer status of all pods
    qa Use QA cluster
    qucli Quay.io CLI

    View full-size slide

  15. slack-notifier

    View full-size slide

  16. kube ࣾ಺πʔϧ

    View full-size slide

  17. Kubernetes v1.4.1 (2016/10 - 2016/11)
    • kube-up.sh Λ࢖ͬͯ AWS ্ʹߏங
    •ScheduledJob ͷௐࠪͱݕূ։࢝
    •Job ͸ࣦഊ͢ΔͱࣗಈͰ Retry ͢Δ
    •ႈ౳ੑΛอͭ Job Λॻ͘

    View full-size slide

  18. Kubernetes v1.4.5 (2016/11/2 -)
    •kube-dns ͕ܨ͕Βͳ͍౳ͷ໰୊͕͋Γɺͦͷվળ͞ΕͨλΠϛϯάͰ
    Upgrade
    •AWS ECR Ͱͷར༻։࢝
    •ScheduledJob ܦ༝Ͱ࡞ΒΕͨ Job ໊ͷ UUID ͕ॏෳ͠ɺJob ͕࡞ΒΕͳ
    ͍
    •ෳ਺ͷνʔϜͰར༻͕࢝·ΓɺCI/CD ͷϧʔϧͱςϯϓϨʔτΛܾΊΔ

    View full-size slide

  19. ͜͜·Ͱ࡞੒ͨ͠πʔϧ
    •run / blue-green deployment / rollback
    •֤ϦϙδτϦʹ script/ Λ࡞੒͠ɺͲͷΞϓϦέʔγϣϯͰ΋ಉ͡ϧʔϧͰӡ༻͢ΔʹऔΓܾΊ
    •࣮ଶ͸ kubectl Λ࢖ͬͨ؆୯ͳ shellscript
    •koudaiii/kubeps
    •pod ͕͍ͭɺͲͷ tag Ͱ࣮ߦ͞Ε͍ͯΔͷ͔෼͔Δ
    •dtan4/k8stail
    •ͦͷ namespace Ͱಈ͍͍ͯΔ pod ͷ log ΛҰׅ streaming ͢Δɻ wercker/stern ͱ΄΅ಉ࣌ظ
    •koudaiii/qucli
    •ΤϯδχΞ͕ cli Λ௨ͯ͠ docker ͷ private registry Λ࡞੒Ͱ͖ΔΑ͏ʹ

    View full-size slide

  20. k8stail
    $ k8stail
    Context: prod.cluster.wantedlyapp.com
    Namespace: default
    Labels:
    Press Ctrl-C to exit.
    ----------
    Pod:dd-agent-1wml1 Container:dd-agent has been detected
    Pod:dd-agent-61753 Container:dd-agent has been detected
    Pod:dd-agent-83chh Container:dd-agent has been detected
    Pod:dd-agent-8gg9x Container:dd-agent has been detected

    View full-size slide

  21. kubeps
    $ kubeps --namespace=default
    Namespace: default
    Labels:
    === Deployment ===
    NAME IMAGE NAMESPACE
    dd-agent datadog/docker-dd-agent:latest default
    === Pod ===
    NAME IMAGE STATUS READY RESTARTS START NAMESPACE
    dd-agent-1wml1 datadog/docker-dd-agent:latest Running 1/1 0 2017-06-27 15:20:24 +0900 JSTdefault

    View full-size slide

  22. qucli
    $ qucli create koudaiii/test
    Created! quay.io/koudaiii/test
    $ qucli get koudaiii/test
    Repository:
    quay.io/koudaiii/test
    Visibility:
    public
    Permissions:
    koudaiii(admin)

    View full-size slide

  23. Kubernetes v1.5.6 (2016/04 - 2017/06)
    •kube-dns ͷ autoscaler ͕ 1.5.0 ͰೖΓɺ SPOF ͱͳ͍ͬͯͨ෦෼͕ղܾ
    •kops ಋೖ
    •ෳ਺ Cluster(Production/QA) / sandbox ؀ڥΛߏங
    •horizontal pod auto scalers Ͱ pod ͷ auto scale Ұ෦ಋೖ
    •CronJob(ScheduledJob) ͷ UUID ॏෳͷղܾ
    •طଘαʔϏεͷόονॲཧΛશͯ Kubernetes ΁Ҡߦ
    •GC ͕ࣦഊ͠ଓ͚Δͱؾ͕͍ͭͨΒ disk ͕ᷓΕɺEvict ͞ΕͯԿ౓΋ Job Λ࠶࣮ߦ
    •pod Λ࡞੒͢Δࡍʹίϯςφͷ NIC ͷ create ͱ delete ͕ڝ߹ͯ͠ɺ NIC ͷͳ͍ίϯςφ͕ग़དྷ্͕Γ Evict ͞ΕΔ

    View full-size slide

  24. ͜͜·Ͱ࡞੒ͨ͠πʔϧ
    •dtan4/k8s-job-cleaner
    •ແݶʹ૿͑ଓ͚Δ Job Λফ͢
    •dtan4/k8s-pod-notifier
    •Job ͕ Fail ͨ͠Β slack ʹ௨஌͢Δ

    View full-size slide

  25. Multi Master (2017/06 - )
    •master ఀࢭແ͘ kops Ͱ upgrade Ͱ͖Δ
    •master ͷ SPOF վળ

    View full-size slide

  26. Kubernetes v1.6.6 (2017/06 - )
    •envFrom ಋೖ => ͜ΕʹΑΓ manifest file ʹ؀ڥม਺Λ௥ه͢Δඞཁ͕ͳ͘ͳͬͨ
    •Cronjob ʹ rotate ͱ͕ೖΔͷͱɺࣦഊճ਺ΛઃఆͰ͖ΔΑ͏ʹͳͬͨ
    •successfulJobsHistoryLimit
    •failedJobsHistoryLimit
    •koudaiii/sltd
    •Ͳͷ͘Β͍ request ͕དྷ͍ͯΔͷ͔? Http Status Λ؆୯ʹ஌ΔͨΊͷπʔϧ

    View full-size slide

  27. CI/CD ͷऔΓܾΊ

    View full-size slide

  28. CI/CD ͷςϯϓϨʔτ
    1. ϒϥϯνΛ੾ΓɺϓϧϦΫΤετΛૹΔ
    2. git push ͢Δ౓ʹςετ͕࣮ߦ͞ΕΔ
    3. ςετ͕௨Ε͹ QA ʹ deploy ͞Εɺϒϥ΢βͰ֬ೝ͢Δ
    4. ϦϦʔεग़དྷΔλΠϛϯάʹͳͬͨΒ master ʹϚʔδ͢Δ
    5. CI ্Ͱςετ͕૸Γɺςετ͕௨Ε͹ Production ʹϦϦʔε͢Δ

    View full-size slide

  29. •https://www.wantedly.com/companies/wantedly/post_articles/46089

    View full-size slide

  30. ͦͷଞͷऔΓܾΊ
    ΢ΣϒΞϓϦέʔγϣϯͷϔϧενΣοΫ΍εςʔλεͷURL͸౷Ұʹ͢Δ (/healthcheck ΍ /ping)
    ͢΂ͯͷϦϙδτϦʹ script σΟϨΫτϦΛ࡞੒͠ɺΠϯϑϥ࡞ۀΛಉ͡ํ๏Ͱ࣮ߦ
    Πϯετʔϧ script/bootstrap
    Ϗϧυεςοϓ script/ci-build
    σϓϩΠεςοϓ script/ci-deploy (Rolling Deploy ͔ blue-green Deploy ͔͸ࣗ༝ʹॻ͍ͯ΋ྑ͍)
    αʔόʔىಈ script/server
    ίϯιʔϧ(one-off container) script/console

    View full-size slide

  31. ϞχλϦϯά

    View full-size slide

  32. ϞχλϦϯά
    • ͲΜͲΜϚΠΫϩαʔϏεԽ͸ਐΉ => ֤͕ࣗ namespace Λ੾ͬͯར༻Ͱ͖ΔΑ͏ʹ͢Δ
    • 1ͭ1ͭઃఆ͢ΔΑΓ͸ɺϧʔϧΛܾΊͯ General ʹઃఆ͞ΕΔΑ͏ʹ͢Δ
    • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}}
    • kubernetes_state Λ࢖ͬͯɺ deployment ͕͔ͬ͠Γ available ʹͳ͍ͬͯΔ͜ͱΛݟΔ
    • WebΞϓϦέʔγϣϯ͕ࢧྲྀͳͷͰɺrequest ͱ status code ΋߹ͤͯͰ͖ΔΑ͏ʹ͢Δ
    • aws.elb.request_count ౳
    • ઐ༻μογϡϘʔυͰͲͷ͘Β͍ request ͱ SLA ͕୲อͰ͖͍ͯΔ͔ݟΕΔΑ͏ʹ

    View full-size slide

  33. ϞχλϦϯάͰ࢖༻͍ͯ͠Δ΋ͷ
    • kelseyhightower/konfd
    • secret ͔Β ConfigMap Λੜ੒͢Δ
    • repository ʹύεϫʔυͳͲΛؚΊͨ͘ͳ͍৔߹ʹར༻
    • kubernetes/kube-state-metrics
    • deployment ౳ͷ available Λऔಘ
    • datadog/docker-dd-agent
    • datadog ͷ agent Λ docker ༻ɺ http ༻ɺ DB ༻ͱrole ຖʹ deployment ·ͨ͸ DaemonSet Λ࡞͍ͬͯΔ
    • koudaiii/sltd
    • Service ͔Β࡞ΒΕΔ ELB ʹ labels Λ tag ʹ௥Ճɺ name ΍ namespace ౳΋߹ΘͤͯELB ʹ tag Λ͚ͭΔ

    View full-size slide

  34. ·ͱΊ
    • Kubernetes Λ Production Ͱ࢖͏৔߹ʹ΍ͬͨ͜ͱ
    • ෳ਺Ϋϥελͱ Multi Master ͷ༻ҙ
    • खؒʹͳΔϞχλϦϯά΍؂ࢹͷࣗಈԽ
    • CI/CD ͷϧʔϧԽ

    View full-size slide

  35. IUUQTXXXXBOUFEMZDPNQSPKFDUT

    View full-size slide