Using Kubernetes in Production / #k8sjp

Kubernetes Meetup Tokyo #5
https://k8sjp.connpass.com/event/56945/

Kubernetes を Production で運用してきたことについて話をしました。

Transcript

1. ,PEBJ4BLBCF!LPVEBJJJ8BOUFEMZ *OD
   Using Kubernetes in Production
   

   View Slide

2. ABOUT ME
   — Software Engineer —
   2010/4- TIS
   ࡔ෦ ޿େ (KODAI SAKABE)
   2015/8- Wantedly
   @koudaiii
   Site: https://koudaiii.com
   

   View Slide

3. View Slide

4. ΰʔϧ
   • Kubernetes Λ Production Ͱ࢖͏৔߹ʹ΍ͬͨ͜ͱ
   • ෳ਺Ϋϥελͱ Multi Master ͷ༻ҙ
   • खؒʹͳΔϞχλϦϯά΍؂ࢹͷઃఆΛࣗಈԽ
   • CI/CD ͷϧʔϧԽ
   

   View Slide

5. αʔϏεͷਪҠ
   WantedlyશମͷαʔϏε਺ ͱ Kubernetes ͷ Namespace ͷ਺
   

   View Slide

6. ΞδΣϯμ
   1೥ؒͷऔΓ૊Έͱ࡞੒ͨ͠πʔϧ
   CI/CD ͷϧʔϧԽ
   ϞχλϦϯά
   

   View Slide

7. 1 ೥ؒͷऔΓ૊Έ
   

   View Slide

8. Kubernetes v1.1 (2015/11 - 2016/04)
   •৽ن Web Application Λߏங͢Δͷʹखؒͱ࡞ۀऀͷଐਓԽ
   •ʮWantedly ্ʹ͍Δ User ͷܨ͕ΓΛߴ଎ʹฦ͢αʔϏεΛ࡞Δʯ͔Β
   ελʔτ
   •࣮ݧతʹ Go ͷΞϓϦέʔγϣϯΛ Kubernetes ্ʹཱͯΔ
   •Kubernetes ͷݕূ͕ελʔτͨ͠
   

   View Slide

9. Production?
   •Production ʹࡌͤΔલʹҎԼͷ2ͭʹ͍ͭͯௐࠪΛߦ͍ͬͯͬͨ
   •ͦ΋ͦ΋ΫϥελͷϝϦοτͬͯͳΜͩΖ͏͔ʁ
   •Ϋϥελͦͷ΋ͷ͕յΕͨ࣌Ͳ͏͢Ε͹ྑ͍ͷ͔ʁ
   

   View Slide

10. Production?
    •Ϋϥελ͕յΕͨ৔߹ɺ࠷ऴతʹݩʹ໭ͨ͢ΊʹͲ͏͢Ε͹ྑ͍ͷ͔?
    •VPC ຖʹΫϥελϦϯάΛ࡞੒Ͱ͖ΔΑ͏ʹ͢Δ => ࠷ѱҠಈՄೳ
    •1 Host : 1 ΞϓϦέʔγϣϯ ͱ 1ΫϥελϦϯά : ଞΞϓϦέʔγϣϯͷҧ͍
    •ޮ཰Խ͢Δ͜ͱʹΑΔϝϦοτͱσϝϦοτͷ·ͱΊ
    •Ϋϥελࣄଶͷ؂ࢹͱ؅ཧ͸Ͳ͏͢Δ΂͖͔
    •Datadog on Kubernetes(dd-agent)
    •Deployment ΍ Replication Controller Ͱ࡞੒͞Εͨ Pod
    • kubernetes/kube-state-metric Λ࢖ͬͯ unavailable ͷ਺ͱ available ͷ਺ΛݟΔ
    

    View Slide

11. Kubernetes ಋೖظ v1.2.3 (2016/4- 2016/10)
    •Production Ͱ·ͣ͸ microservice ͷӡ༻͔Β࢝ΊΔ
    •API ʹ response ͕ͳͯ͘΋ຊମͷαʔϏε͕མͪͳ͍ઃܭ
    •·ͣ͸ΠϯϑϥνʔϜͰશͯϝϯςφϯε͢Δ
    •νʔϜͰ࢖͍ʹ͍͘෦෼Λվળ͢Δ
    •҆શੑͱརศੑΛߟ͑ͯɺ `repository == namespace` ͱ͍͏ܗͰӡ༻Λ։࢝
    •Datadog Λ DaemonSet ʹೖΕͯར༻։࢝
    •kubernetes ͷ Log ͱͯ͠ Logentries ΁ૹΔΑ͏ʹ DaemonSet ʹೖΕͯར༻։࢝
    

    View Slide

12. ͜͜·Ͱ࡞੒ͨ͠πʔϧ
    •dtan4/k8sec
    •؀ڥม਺ʹ secret Λར༻͢Δࡍʹߋ৽ͱϦετͰ֬ೝ͢Δͷ͕खؒͩͬͨ
    •`heroku config` ͷΑ͏ͳπʔϧ
    •kube(ࣾ಺πʔϧ), wantedly/slack-notifier
    •heroku ίϚϯυͱಉ͡Α͏ʹͦͷϦϙδτϦͰͷ࡞ۀ͸ͦͷ namespace ͕σϑΥϧτͰೖΔ
    •࡞੒ͨ͠πʔϧͱ kubectl Λͦͷ··ར༻Մೳ
    •wantedly/dockertags
    •ίϚϯυϥΠϯ͔Β docker ͷ tag ҰཡΛݟΕΔΑ͏ʹͨ͠
    

    View Slide

13. k8sec
    $ k8sec set test key=value key2=value2 --namespace=default
    test
    $ k8sec list test --namespace=default
    NAME TYPE KEY VALUE
    test Opaque key2 "value2"
    test Opaque key "value"
    $ k8sec set test key=hoge --namespace=default
    test
    $ k8sec list test --namespace=default
    NAME TYPE KEY VALUE
    test Opaque key "hoge"
    test Opaque key2 "value2"
    

    View Slide

14. dockertags
    $ bin/dockertags quay.io/koudaiii/sltd
    latest
    v0.1.0
    

    View Slide

15. kube
    $ kube
    CLI tool to execute kubectl over SSH
    Usage:
    kube [command]
    Available Commands:
    ɾɾɾɾ
    create kubectl create
    delete kubectl delete
    describe kubectl describe
    ɾɾɾɾ
    dotenv
    enable
    generate Generate manifest file
    get kubectl get
    prod Use Production cluster
    ps Conatainer status of all pods
    qa Use QA cluster
    qucli Quay.io CLI
    

    View Slide

16. slack-notifier
    

    View Slide

17. kube ࣾ಺πʔϧ
    

    View Slide

18. Kubernetes v1.4.1 (2016/10 - 2016/11)
    • kube-up.sh Λ࢖ͬͯ AWS ্ʹߏங
    •ScheduledJob ͷௐࠪͱݕূ։࢝
    •Job ͸ࣦഊ͢ΔͱࣗಈͰ Retry ͢Δ
    •ႈ౳ੑΛอͭ Job Λॻ͘
    

    View Slide

19. Kubernetes v1.4.5 (2016/11/2 -)
    •kube-dns ͕ܨ͕Βͳ͍౳ͷ໰୊͕͋Γɺͦͷվળ͞ΕͨλΠϛϯάͰ
    Upgrade
    •AWS ECR Ͱͷར༻։࢝
    •ScheduledJob ܦ༝Ͱ࡞ΒΕͨ Job ໊ͷ UUID ͕ॏෳ͠ɺJob ͕࡞ΒΕͳ
    ͍
    •ෳ਺ͷνʔϜͰར༻͕࢝·ΓɺCI/CD ͷϧʔϧͱςϯϓϨʔτΛܾΊΔ
    

    View Slide

20. ͜͜·Ͱ࡞੒ͨ͠πʔϧ
    •run / blue-green deployment / rollback
    •֤ϦϙδτϦʹ script/ Λ࡞੒͠ɺͲͷΞϓϦέʔγϣϯͰ΋ಉ͡ϧʔϧͰӡ༻͢ΔʹऔΓܾΊ
    •࣮ଶ͸ kubectl Λ࢖ͬͨ؆୯ͳ shellscript
    •koudaiii/kubeps
    •pod ͕͍ͭɺͲͷ tag Ͱ࣮ߦ͞Ε͍ͯΔͷ͔෼͔Δ
    •dtan4/k8stail
    •ͦͷ namespace Ͱಈ͍͍ͯΔ pod ͷ log ΛҰׅ streaming ͢Δɻ wercker/stern ͱ΄΅ಉ࣌ظ
    •koudaiii/qucli
    •ΤϯδχΞ͕ cli Λ௨ͯ͠ docker ͷ private registry Λ࡞੒Ͱ͖ΔΑ͏ʹ
    

    View Slide

21. k8stail
    $ k8stail
    Context: prod.cluster.wantedlyapp.com
    Namespace: default
    Labels:
    Press Ctrl-C to exit.
    ----------
    Pod:dd-agent-1wml1 Container:dd-agent has been detected
    Pod:dd-agent-61753 Container:dd-agent has been detected
    Pod:dd-agent-83chh Container:dd-agent has been detected
    Pod:dd-agent-8gg9x Container:dd-agent has been detected
    

    View Slide

22. kubeps
    $ kubeps --namespace=default
    Namespace: default
    Labels:
    === Deployment ===
    NAME IMAGE NAMESPACE
    dd-agent datadog/docker-dd-agent:latest default
    === Pod ===
    NAME IMAGE STATUS READY RESTARTS START NAMESPACE
    dd-agent-1wml1 datadog/docker-dd-agent:latest Running 1/1 0 2017-06-27 15:20:24 +0900 JSTdefault
    

    View Slide

23. qucli
    $ qucli create koudaiii/test
    Created! quay.io/koudaiii/test
    $ qucli get koudaiii/test
    Repository:
    quay.io/koudaiii/test
    Visibility:
    public
    Permissions:
    koudaiii(admin)
    

    View Slide

24. Kubernetes v1.5.6 (2016/04 - 2017/06)
    •kube-dns ͷ autoscaler ͕ 1.5.0 ͰೖΓɺ SPOF ͱͳ͍ͬͯͨ෦෼͕ղܾ
    •kops ಋೖ
    •ෳ਺ Cluster(Production/QA) / sandbox ؀ڥΛߏங
    •horizontal pod auto scalers Ͱ pod ͷ auto scale Ұ෦ಋೖ
    •CronJob(ScheduledJob) ͷ UUID ॏෳͷղܾ
    •طଘαʔϏεͷόονॲཧΛશͯ Kubernetes ΁Ҡߦ
    •GC ͕ࣦഊ͠ଓ͚Δͱؾ͕͍ͭͨΒ disk ͕ᷓΕɺEvict ͞ΕͯԿ౓΋ Job Λ࠶࣮ߦ
    •pod Λ࡞੒͢Δࡍʹίϯςφͷ NIC ͷ create ͱ delete ͕ڝ߹ͯ͠ɺ NIC ͷͳ͍ίϯςφ͕ग़དྷ্͕Γ Evict ͞ΕΔ
    

    View Slide

25. ͜͜·Ͱ࡞੒ͨ͠πʔϧ
    •dtan4/k8s-job-cleaner
    •ແݶʹ૿͑ଓ͚Δ Job Λফ͢
    •dtan4/k8s-pod-notifier
    •Job ͕ Fail ͨ͠Β slack ʹ௨஌͢Δ
    

    View Slide

26. Multi Master (2017/06 - )
    •master ఀࢭແ͘ kops Ͱ upgrade Ͱ͖Δ
    •master ͷ SPOF վળ
    

    View Slide

27. Kubernetes v1.6.6 (2017/06 - )
    •envFrom ಋೖ => ͜ΕʹΑΓ manifest file ʹ؀ڥม਺Λ௥ه͢Δඞཁ͕ͳ͘ͳͬͨ
    •Cronjob ʹ rotate ͱ͕ೖΔͷͱɺࣦഊճ਺ΛઃఆͰ͖ΔΑ͏ʹͳͬͨ
    •successfulJobsHistoryLimit
    •failedJobsHistoryLimit
    •koudaiii/sltd
    •Ͳͷ͘Β͍ request ͕དྷ͍ͯΔͷ͔? Http Status Λ؆୯ʹ஌ΔͨΊͷπʔϧ
    

    View Slide

28. CI/CD ͷऔΓܾΊ
    

    View Slide

29. CI/CD ͷςϯϓϨʔτ
    1. ϒϥϯνΛ੾ΓɺϓϧϦΫΤετΛૹΔ
    2. git push ͢Δ౓ʹςετ͕࣮ߦ͞ΕΔ
    3. ςετ͕௨Ε͹ QA ʹ deploy ͞Εɺϒϥ΢βͰ֬ೝ͢Δ
    4. ϦϦʔεग़དྷΔλΠϛϯάʹͳͬͨΒ master ʹϚʔδ͢Δ
    5. CI ্Ͱςετ͕૸Γɺςετ͕௨Ε͹ Production ʹϦϦʔε͢Δ
    

    View Slide

30. •https://www.wantedly.com/companies/wantedly/post_articles/46089
    

    View Slide

31. ͦͷଞͷऔΓܾΊ
    ΢ΣϒΞϓϦέʔγϣϯͷϔϧενΣοΫ΍εςʔλεͷURL͸౷Ұʹ͢Δ (/healthcheck ΍ /ping)
    ͢΂ͯͷϦϙδτϦʹ script σΟϨΫτϦΛ࡞੒͠ɺΠϯϑϥ࡞ۀΛಉ͡ํ๏Ͱ࣮ߦ
    Πϯετʔϧ script/bootstrap
    Ϗϧυεςοϓ script/ci-build
    σϓϩΠεςοϓ script/ci-deploy (Rolling Deploy ͔ blue-green Deploy ͔͸ࣗ༝ʹॻ͍ͯ΋ྑ͍)
    αʔόʔىಈ script/server
    ίϯιʔϧ(one-off container) script/console
    

    View Slide

32. ϞχλϦϯά
    

    View Slide

33. ϞχλϦϯά
    • ͲΜͲΜϚΠΫϩαʔϏεԽ͸ਐΉ => ֤͕ࣗ namespace Λ੾ͬͯར༻Ͱ͖ΔΑ͏ʹ͢Δ
    • 1ͭ1ͭઃఆ͢ΔΑΓ͸ɺϧʔϧΛܾΊͯ General ʹઃఆ͞ΕΔΑ͏ʹ͢Δ
    • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}}
    • kubernetes_state Λ࢖ͬͯɺ deployment ͕͔ͬ͠Γ available ʹͳ͍ͬͯΔ͜ͱΛݟΔ
    • WebΞϓϦέʔγϣϯ͕ࢧྲྀͳͷͰɺrequest ͱ status code ΋߹ͤͯͰ͖ΔΑ͏ʹ͢Δ
    • aws.elb.request_count ౳
    • ઐ༻μογϡϘʔυͰͲͷ͘Β͍ request ͱ SLA ͕୲อͰ͖͍ͯΔ͔ݟΕΔΑ͏ʹ
    

    View Slide

34. View Slide

35. ϞχλϦϯάͰ࢖༻͍ͯ͠Δ΋ͷ
    • kelseyhightower/konfd
    • secret ͔Β ConfigMap Λੜ੒͢Δ
    • repository ʹύεϫʔυͳͲΛؚΊͨ͘ͳ͍৔߹ʹར༻
    • kubernetes/kube-state-metrics
    • deployment ౳ͷ available Λऔಘ
    • datadog/docker-dd-agent
    • datadog ͷ agent Λ docker ༻ɺ http ༻ɺ DB ༻ͱrole ຖʹ deployment ·ͨ͸ DaemonSet Λ࡞͍ͬͯΔ
    • koudaiii/sltd
    • Service ͔Β࡞ΒΕΔ ELB ʹ labels Λ tag ʹ௥Ճɺ name ΍ namespace ౳΋߹ΘͤͯELB ʹ tag Λ͚ͭΔ
    

    View Slide

36. ·ͱΊ
    • Kubernetes Λ Production Ͱ࢖͏৔߹ʹ΍ͬͨ͜ͱ
    • ෳ਺Ϋϥελͱ Multi Master ͷ༻ҙ
    • खؒʹͳΔϞχλϦϯά΍؂ࢹͷࣗಈԽ
    • CI/CD ͷϧʔϧԽ
    

    View Slide

37. IUUQTXXXXBOUFEMZDPNQSPKFDUT
    

    View Slide