Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Kubernetes in Production / #k8sjp

Using Kubernetes in Production / #k8sjp

Kubernetes Meetup Tokyo #5
https://k8sjp.connpass.com/event/56945/

Kubernetes を Production で運用してきたことについて話をしました。

Kodai Sakabe

June 28, 2017
Tweet

More Decks by Kodai Sakabe

Other Decks in Technology

Transcript

  1. ,PEBJ4BLBCF!LPVEBJJJ8BOUFEMZ *OD
    Using Kubernetes in Production

    View Slide

  2. ABOUT ME
    — Software Engineer —
    2010/4- TIS
    ࡔ෦ ޿େ (KODAI SAKABE)
    2015/8- Wantedly
    @koudaiii
    Site: https://koudaiii.com

    View Slide

  3. View Slide

  4. ΰʔϧ
    • Kubernetes Λ Production Ͱ࢖͏৔߹ʹ΍ͬͨ͜ͱ
    • ෳ਺Ϋϥελͱ Multi Master ͷ༻ҙ
    • खؒʹͳΔϞχλϦϯά΍؂ࢹͷઃఆΛࣗಈԽ
    • CI/CD ͷϧʔϧԽ

    View Slide

  5. αʔϏεͷਪҠ
    WantedlyશମͷαʔϏε਺ ͱ Kubernetes ͷ Namespace ͷ਺

    View Slide

  6. ΞδΣϯμ
    1೥ؒͷऔΓ૊Έͱ࡞੒ͨ͠πʔϧ
    CI/CD ͷϧʔϧԽ
    ϞχλϦϯά

    View Slide

  7. 1 ೥ؒͷऔΓ૊Έ

    View Slide

  8. Kubernetes v1.1 (2015/11 - 2016/04)
    •৽ن Web Application Λߏங͢Δͷʹखؒͱ࡞ۀऀͷଐਓԽ
    •ʮWantedly ্ʹ͍Δ User ͷܨ͕ΓΛߴ଎ʹฦ͢αʔϏεΛ࡞Δʯ͔Β
    ελʔτ
    •࣮ݧతʹ Go ͷΞϓϦέʔγϣϯΛ Kubernetes ্ʹཱͯΔ
    •Kubernetes ͷݕূ͕ελʔτͨ͠

    View Slide

  9. Production?
    •Production ʹࡌͤΔલʹҎԼͷ2ͭʹ͍ͭͯௐࠪΛߦ͍ͬͯͬͨ
    •ͦ΋ͦ΋ΫϥελͷϝϦοτͬͯͳΜͩΖ͏͔ʁ
    •Ϋϥελͦͷ΋ͷ͕յΕͨ࣌Ͳ͏͢Ε͹ྑ͍ͷ͔ʁ

    View Slide

  10. Production?
    •Ϋϥελ͕յΕͨ৔߹ɺ࠷ऴతʹݩʹ໭ͨ͢ΊʹͲ͏͢Ε͹ྑ͍ͷ͔?
    •VPC ຖʹΫϥελϦϯάΛ࡞੒Ͱ͖ΔΑ͏ʹ͢Δ => ࠷ѱҠಈՄೳ
    •1 Host : 1 ΞϓϦέʔγϣϯ ͱ 1ΫϥελϦϯά : ଞΞϓϦέʔγϣϯͷҧ͍
    •ޮ཰Խ͢Δ͜ͱʹΑΔϝϦοτͱσϝϦοτͷ·ͱΊ
    •Ϋϥελࣄଶͷ؂ࢹͱ؅ཧ͸Ͳ͏͢Δ΂͖͔
    •Datadog on Kubernetes(dd-agent)
    •Deployment ΍ Replication Controller Ͱ࡞੒͞Εͨ Pod
    • kubernetes/kube-state-metric Λ࢖ͬͯ unavailable ͷ਺ͱ available ͷ਺ΛݟΔ

    View Slide

  11. Kubernetes ಋೖظ v1.2.3 (2016/4- 2016/10)
    •Production Ͱ·ͣ͸ microservice ͷӡ༻͔Β࢝ΊΔ
    •API ʹ response ͕ͳͯ͘΋ຊମͷαʔϏε͕མͪͳ͍ઃܭ
    •·ͣ͸ΠϯϑϥνʔϜͰશͯϝϯςφϯε͢Δ
    •νʔϜͰ࢖͍ʹ͍͘෦෼Λվળ͢Δ
    •҆શੑͱརศੑΛߟ͑ͯɺ `repository == namespace` ͱ͍͏ܗͰӡ༻Λ։࢝
    •Datadog Λ DaemonSet ʹೖΕͯར༻։࢝
    •kubernetes ͷ Log ͱͯ͠ Logentries ΁ૹΔΑ͏ʹ DaemonSet ʹೖΕͯར༻։࢝

    View Slide

  12. ͜͜·Ͱ࡞੒ͨ͠πʔϧ
    •dtan4/k8sec
    •؀ڥม਺ʹ secret Λར༻͢Δࡍʹߋ৽ͱϦετͰ֬ೝ͢Δͷ͕खؒͩͬͨ
    •`heroku config` ͷΑ͏ͳπʔϧ
    •kube(ࣾ಺πʔϧ), wantedly/slack-notifier
    •heroku ίϚϯυͱಉ͡Α͏ʹͦͷϦϙδτϦͰͷ࡞ۀ͸ͦͷ namespace ͕σϑΥϧτͰೖΔ
    •࡞੒ͨ͠πʔϧͱ kubectl Λͦͷ··ར༻Մೳ
    •wantedly/dockertags
    •ίϚϯυϥΠϯ͔Β docker ͷ tag ҰཡΛݟΕΔΑ͏ʹͨ͠

    View Slide

  13. k8sec
    $ k8sec set test key=value key2=value2 --namespace=default
    test
    $ k8sec list test --namespace=default
    NAME TYPE KEY VALUE
    test Opaque key2 "value2"
    test Opaque key "value"
    $ k8sec set test key=hoge --namespace=default
    test
    $ k8sec list test --namespace=default
    NAME TYPE KEY VALUE
    test Opaque key "hoge"
    test Opaque key2 "value2"

    View Slide

  14. dockertags
    $ bin/dockertags quay.io/koudaiii/sltd
    latest
    v0.1.0

    View Slide

  15. kube
    $ kube
    CLI tool to execute kubectl over SSH
    Usage:
    kube [command]
    Available Commands:
    ɾɾɾɾ
    create kubectl create
    delete kubectl delete
    describe kubectl describe
    ɾɾɾɾ
    dotenv
    enable
    generate Generate manifest file
    get kubectl get
    prod Use Production cluster
    ps Conatainer status of all pods
    qa Use QA cluster
    qucli Quay.io CLI

    View Slide

  16. slack-notifier

    View Slide

  17. kube ࣾ಺πʔϧ

    View Slide

  18. Kubernetes v1.4.1 (2016/10 - 2016/11)
    • kube-up.sh Λ࢖ͬͯ AWS ্ʹߏங
    •ScheduledJob ͷௐࠪͱݕূ։࢝
    •Job ͸ࣦഊ͢ΔͱࣗಈͰ Retry ͢Δ
    •ႈ౳ੑΛอͭ Job Λॻ͘

    View Slide

  19. Kubernetes v1.4.5 (2016/11/2 -)
    •kube-dns ͕ܨ͕Βͳ͍౳ͷ໰୊͕͋Γɺͦͷվળ͞ΕͨλΠϛϯάͰ
    Upgrade
    •AWS ECR Ͱͷར༻։࢝
    •ScheduledJob ܦ༝Ͱ࡞ΒΕͨ Job ໊ͷ UUID ͕ॏෳ͠ɺJob ͕࡞ΒΕͳ
    ͍
    •ෳ਺ͷνʔϜͰར༻͕࢝·ΓɺCI/CD ͷϧʔϧͱςϯϓϨʔτΛܾΊΔ

    View Slide

  20. ͜͜·Ͱ࡞੒ͨ͠πʔϧ
    •run / blue-green deployment / rollback
    •֤ϦϙδτϦʹ script/ Λ࡞੒͠ɺͲͷΞϓϦέʔγϣϯͰ΋ಉ͡ϧʔϧͰӡ༻͢ΔʹऔΓܾΊ
    •࣮ଶ͸ kubectl Λ࢖ͬͨ؆୯ͳ shellscript
    •koudaiii/kubeps
    •pod ͕͍ͭɺͲͷ tag Ͱ࣮ߦ͞Ε͍ͯΔͷ͔෼͔Δ
    •dtan4/k8stail
    •ͦͷ namespace Ͱಈ͍͍ͯΔ pod ͷ log ΛҰׅ streaming ͢Δɻ wercker/stern ͱ΄΅ಉ࣌ظ
    •koudaiii/qucli
    •ΤϯδχΞ͕ cli Λ௨ͯ͠ docker ͷ private registry Λ࡞੒Ͱ͖ΔΑ͏ʹ

    View Slide

  21. k8stail
    $ k8stail
    Context: prod.cluster.wantedlyapp.com
    Namespace: default
    Labels:
    Press Ctrl-C to exit.
    ----------
    Pod:dd-agent-1wml1 Container:dd-agent has been detected
    Pod:dd-agent-61753 Container:dd-agent has been detected
    Pod:dd-agent-83chh Container:dd-agent has been detected
    Pod:dd-agent-8gg9x Container:dd-agent has been detected

    View Slide

  22. kubeps
    $ kubeps --namespace=default
    Namespace: default
    Labels:
    === Deployment ===
    NAME IMAGE NAMESPACE
    dd-agent datadog/docker-dd-agent:latest default
    === Pod ===
    NAME IMAGE STATUS READY RESTARTS START NAMESPACE
    dd-agent-1wml1 datadog/docker-dd-agent:latest Running 1/1 0 2017-06-27 15:20:24 +0900 JSTdefault

    View Slide

  23. qucli
    $ qucli create koudaiii/test
    Created! quay.io/koudaiii/test
    $ qucli get koudaiii/test
    Repository:
    quay.io/koudaiii/test
    Visibility:
    public
    Permissions:
    koudaiii(admin)

    View Slide

  24. Kubernetes v1.5.6 (2016/04 - 2017/06)
    •kube-dns ͷ autoscaler ͕ 1.5.0 ͰೖΓɺ SPOF ͱͳ͍ͬͯͨ෦෼͕ղܾ
    •kops ಋೖ
    •ෳ਺ Cluster(Production/QA) / sandbox ؀ڥΛߏங
    •horizontal pod auto scalers Ͱ pod ͷ auto scale Ұ෦ಋೖ
    •CronJob(ScheduledJob) ͷ UUID ॏෳͷղܾ
    •طଘαʔϏεͷόονॲཧΛશͯ Kubernetes ΁Ҡߦ
    •GC ͕ࣦഊ͠ଓ͚Δͱؾ͕͍ͭͨΒ disk ͕ᷓΕɺEvict ͞ΕͯԿ౓΋ Job Λ࠶࣮ߦ
    •pod Λ࡞੒͢Δࡍʹίϯςφͷ NIC ͷ create ͱ delete ͕ڝ߹ͯ͠ɺ NIC ͷͳ͍ίϯςφ͕ग़དྷ্͕Γ Evict ͞ΕΔ

    View Slide

  25. ͜͜·Ͱ࡞੒ͨ͠πʔϧ
    •dtan4/k8s-job-cleaner
    •ແݶʹ૿͑ଓ͚Δ Job Λফ͢
    •dtan4/k8s-pod-notifier
    •Job ͕ Fail ͨ͠Β slack ʹ௨஌͢Δ

    View Slide

  26. Multi Master (2017/06 - )
    •master ఀࢭແ͘ kops Ͱ upgrade Ͱ͖Δ
    •master ͷ SPOF վળ

    View Slide

  27. Kubernetes v1.6.6 (2017/06 - )
    •envFrom ಋೖ => ͜ΕʹΑΓ manifest file ʹ؀ڥม਺Λ௥ه͢Δඞཁ͕ͳ͘ͳͬͨ
    •Cronjob ʹ rotate ͱ͕ೖΔͷͱɺࣦഊճ਺ΛઃఆͰ͖ΔΑ͏ʹͳͬͨ
    •successfulJobsHistoryLimit
    •failedJobsHistoryLimit
    •koudaiii/sltd
    •Ͳͷ͘Β͍ request ͕དྷ͍ͯΔͷ͔? Http Status Λ؆୯ʹ஌ΔͨΊͷπʔϧ

    View Slide

  28. CI/CD ͷऔΓܾΊ

    View Slide

  29. CI/CD ͷςϯϓϨʔτ
    1. ϒϥϯνΛ੾ΓɺϓϧϦΫΤετΛૹΔ
    2. git push ͢Δ౓ʹςετ͕࣮ߦ͞ΕΔ
    3. ςετ͕௨Ε͹ QA ʹ deploy ͞Εɺϒϥ΢βͰ֬ೝ͢Δ
    4. ϦϦʔεग़དྷΔλΠϛϯάʹͳͬͨΒ master ʹϚʔδ͢Δ
    5. CI ্Ͱςετ͕૸Γɺςετ͕௨Ε͹ Production ʹϦϦʔε͢Δ

    View Slide

  30. •https://www.wantedly.com/companies/wantedly/post_articles/46089

    View Slide

  31. ͦͷଞͷऔΓܾΊ
    ΢ΣϒΞϓϦέʔγϣϯͷϔϧενΣοΫ΍εςʔλεͷURL͸౷Ұʹ͢Δ (/healthcheck ΍ /ping)
    ͢΂ͯͷϦϙδτϦʹ script σΟϨΫτϦΛ࡞੒͠ɺΠϯϑϥ࡞ۀΛಉ͡ํ๏Ͱ࣮ߦ
    Πϯετʔϧ script/bootstrap
    Ϗϧυεςοϓ script/ci-build
    σϓϩΠεςοϓ script/ci-deploy (Rolling Deploy ͔ blue-green Deploy ͔͸ࣗ༝ʹॻ͍ͯ΋ྑ͍)
    αʔόʔىಈ script/server
    ίϯιʔϧ(one-off container) script/console

    View Slide

  32. ϞχλϦϯά

    View Slide

  33. ϞχλϦϯά
    • ͲΜͲΜϚΠΫϩαʔϏεԽ͸ਐΉ => ֤͕ࣗ namespace Λ੾ͬͯར༻Ͱ͖ΔΑ͏ʹ͢Δ
    • 1ͭ1ͭઃఆ͢ΔΑΓ͸ɺϧʔϧΛܾΊͯ General ʹઃఆ͞ΕΔΑ͏ʹ͢Δ
    • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}}
    • kubernetes_state Λ࢖ͬͯɺ deployment ͕͔ͬ͠Γ available ʹͳ͍ͬͯΔ͜ͱΛݟΔ
    • WebΞϓϦέʔγϣϯ͕ࢧྲྀͳͷͰɺrequest ͱ status code ΋߹ͤͯͰ͖ΔΑ͏ʹ͢Δ
    • aws.elb.request_count ౳
    • ઐ༻μογϡϘʔυͰͲͷ͘Β͍ request ͱ SLA ͕୲อͰ͖͍ͯΔ͔ݟΕΔΑ͏ʹ

    View Slide

  34. View Slide

  35. ϞχλϦϯάͰ࢖༻͍ͯ͠Δ΋ͷ
    • kelseyhightower/konfd
    • secret ͔Β ConfigMap Λੜ੒͢Δ
    • repository ʹύεϫʔυͳͲΛؚΊͨ͘ͳ͍৔߹ʹར༻
    • kubernetes/kube-state-metrics
    • deployment ౳ͷ available Λऔಘ
    • datadog/docker-dd-agent
    • datadog ͷ agent Λ docker ༻ɺ http ༻ɺ DB ༻ͱrole ຖʹ deployment ·ͨ͸ DaemonSet Λ࡞͍ͬͯΔ
    • koudaiii/sltd
    • Service ͔Β࡞ΒΕΔ ELB ʹ labels Λ tag ʹ௥Ճɺ name ΍ namespace ౳΋߹ΘͤͯELB ʹ tag Λ͚ͭΔ

    View Slide

  36. ·ͱΊ
    • Kubernetes Λ Production Ͱ࢖͏৔߹ʹ΍ͬͨ͜ͱ
    • ෳ਺Ϋϥελͱ Multi Master ͷ༻ҙ
    • खؒʹͳΔϞχλϦϯά΍؂ࢹͷࣗಈԽ
    • CI/CD ͷϧʔϧԽ

    View Slide

  37. IUUQTXXXXBOUFEMZDPNQSPKFDUT

    View Slide