Using Kubernetes in Production / #k8sjp

Kubernetes Meetup Tokyo #5
https://k8sjp.connpass.com/event/56945/

Kubernetes を Production で運用してきたことについて話をしました。

Transcript

1. ,PEBJ
   4BLBCF
   !LPVEBJJJ
   8BOUFEMZ
   *OD Using Kubernetes in Production
2. ABOUT ME — Software Engineer — 2010/4- TIS ࡔ෦ ޿େ (KODAI SAKABE) 2015/8- Wantedly @koudaiii Site: https://koudaiii.com
4. ΰʔϧ • Kubernetes Λ Production Ͱ࢖͏৔߹ʹ΍ͬͨ͜ͱ • ෳ਺Ϋϥελͱ Multi Master ͷ༻ҙ • खؒʹͳΔϞχλϦϯά΍؂ࢹͷઃఆΛࣗಈԽ • CI/CD ͷϧʔϧԽ
5. αʔϏεͷਪҠ WantedlyશମͷαʔϏε਺ ͱ Kubernetes ͷ Namespace ͷ਺
6. ΞδΣϯμ 1೥ؒͷऔΓ૊Έͱ࡞੒ͨ͠πʔϧ CI/CD ͷϧʔϧԽ ϞχλϦϯά
7. 1 ೥ؒͷऔΓ૊Έ
8. Kubernetes v1.1 (2015/11 - 2016/04) •৽ن Web Application Λߏங͢Δͷʹखؒͱ࡞ۀऀͷଐਓԽ •ʮWantedly ্ʹ͍Δ User ͷܨ͕ΓΛߴ଎ʹฦ͢αʔϏεΛ࡞Δʯ͔Β ελʔτ •࣮ݧతʹ Go ͷΞϓϦέʔγϣϯΛ Kubernetes ্ʹཱͯΔ •Kubernetes ͷݕূ͕ελʔτͨ͠
9. Production? •Production ʹࡌͤΔલʹҎԼͷ2ͭʹ͍ͭͯௐࠪΛߦ͍ͬͯͬͨ •ͦ΋ͦ΋ΫϥελͷϝϦοτͬͯͳΜͩΖ͏͔ʁ •Ϋϥελͦͷ΋ͷ͕յΕͨ࣌Ͳ͏͢Ε͹ྑ͍ͷ͔ʁ
10. Production? •Ϋϥελ͕յΕͨ৔߹ɺ࠷ऴతʹݩʹ໭ͨ͢ΊʹͲ͏͢Ε͹ྑ͍ͷ͔? •VPC ຖʹΫϥελϦϯάΛ࡞੒Ͱ͖ΔΑ͏ʹ͢Δ => ࠷ѱҠಈՄೳ •1 Host : 1 ΞϓϦέʔγϣϯ ͱ 1ΫϥελϦϯά : ଞΞϓϦέʔγϣϯͷҧ͍ •ޮ཰Խ͢Δ͜ͱʹΑΔϝϦοτͱσϝϦοτͷ·ͱΊ •Ϋϥελࣄଶͷ؂ࢹͱ؅ཧ͸Ͳ͏͢Δ΂͖͔ •Datadog on Kubernetes(dd-agent) •Deployment ΍ Replication Controller Ͱ࡞੒͞Εͨ Pod • kubernetes/kube-state-metric Λ࢖ͬͯ unavailable ͷ਺ͱ available ͷ਺ΛݟΔ
11. Kubernetes ಋೖظ v1.2.3 (2016/4- 2016/10) •Production Ͱ·ͣ͸ microservice ͷӡ༻͔Β࢝ΊΔ •API ʹ response ͕ͳͯ͘΋ຊମͷαʔϏε͕མͪͳ͍ઃܭ •·ͣ͸ΠϯϑϥνʔϜͰશͯϝϯςφϯε͢Δ •νʔϜͰ࢖͍ʹ͍͘෦෼Λվળ͢Δ •҆શੑͱརศੑΛߟ͑ͯɺ `repository == namespace` ͱ͍͏ܗͰӡ༻Λ։࢝ •Datadog Λ DaemonSet ʹೖΕͯར༻։࢝ •kubernetes ͷ Log ͱͯ͠ Logentries ΁ૹΔΑ͏ʹ DaemonSet ʹೖΕͯར༻։࢝
12. ͜͜·Ͱ࡞੒ͨ͠πʔϧ •dtan4/k8sec •؀ڥม਺ʹ secret Λར༻͢Δࡍʹߋ৽ͱϦετͰ֬ೝ͢Δͷ͕खؒͩͬͨ •`heroku config` ͷΑ͏ͳπʔϧ •kube(ࣾ಺πʔϧ), wantedly/slack-notifier •heroku ίϚϯυͱಉ͡Α͏ʹͦͷϦϙδτϦͰͷ࡞ۀ͸ͦͷ namespace ͕σϑΥϧτͰೖΔ •࡞੒ͨ͠πʔϧͱ kubectl Λͦͷ··ར༻Մೳ •wantedly/dockertags •ίϚϯυϥΠϯ͔Β docker ͷ tag ҰཡΛݟΕΔΑ͏ʹͨ͠
13. k8sec $ k8sec set test key=value key2=value2 --namespace=default test $ k8sec list test --namespace=default NAME TYPE KEY VALUE test Opaque key2 "value2" test Opaque key "value" $ k8sec set test key=hoge --namespace=default test $ k8sec list test --namespace=default NAME TYPE KEY VALUE test Opaque key "hoge" test Opaque key2 "value2"
14. dockertags $ bin/dockertags quay.io/koudaiii/sltd latest v0.1.0
15. kube $ kube CLI tool to execute kubectl over SSH Usage: kube [command] Available Commands: ɾɾɾɾ create kubectl create delete kubectl delete describe kubectl describe ɾɾɾɾ dotenv enable generate Generate manifest file get kubectl get prod Use Production cluster ps Conatainer status of all pods qa Use QA cluster qucli Quay.io CLI
16. slack-notifier
17. kube ࣾ಺πʔϧ
18. Kubernetes v1.4.1 (2016/10 - 2016/11) • kube-up.sh Λ࢖ͬͯ AWS ্ʹߏங •ScheduledJob ͷௐࠪͱݕূ։࢝ •Job ͸ࣦഊ͢ΔͱࣗಈͰ Retry ͢Δ •ႈ౳ੑΛอͭ Job Λॻ͘
19. Kubernetes v1.4.5 (2016/11/2 -) •kube-dns ͕ܨ͕Βͳ͍౳ͷ໰୊͕͋Γɺͦͷվળ͞ΕͨλΠϛϯάͰ Upgrade •AWS ECR Ͱͷར༻։࢝ •ScheduledJob ܦ༝Ͱ࡞ΒΕͨ Job ໊ͷ UUID ͕ॏෳ͠ɺJob ͕࡞ΒΕͳ ͍ •ෳ਺ͷνʔϜͰར༻͕࢝·ΓɺCI/CD ͷϧʔϧͱςϯϓϨʔτΛܾΊΔ
20. ͜͜·Ͱ࡞੒ͨ͠πʔϧ •run / blue-green deployment / rollback •֤ϦϙδτϦʹ script/ Λ࡞੒͠ɺͲͷΞϓϦέʔγϣϯͰ΋ಉ͡ϧʔϧͰӡ༻͢ΔʹऔΓܾΊ •࣮ଶ͸ kubectl Λ࢖ͬͨ؆୯ͳ shellscript •koudaiii/kubeps •pod ͕͍ͭɺͲͷ tag Ͱ࣮ߦ͞Ε͍ͯΔͷ͔෼͔Δ •dtan4/k8stail •ͦͷ namespace Ͱಈ͍͍ͯΔ pod ͷ log ΛҰׅ streaming ͢Δɻ wercker/stern ͱ΄΅ಉ࣌ظ •koudaiii/qucli •ΤϯδχΞ͕ cli Λ௨ͯ͠ docker ͷ private registry Λ࡞੒Ͱ͖ΔΑ͏ʹ
21. k8stail $ k8stail Context: prod.cluster.wantedlyapp.com Namespace: default Labels: Press Ctrl-C to exit. ---------- Pod:dd-agent-1wml1 Container:dd-agent has been detected Pod:dd-agent-61753 Container:dd-agent has been detected Pod:dd-agent-83chh Container:dd-agent has been detected Pod:dd-agent-8gg9x Container:dd-agent has been detected
22. kubeps $ kubeps --namespace=default Namespace: default Labels: === Deployment === NAME IMAGE NAMESPACE dd-agent datadog/docker-dd-agent:latest default === Pod === NAME IMAGE STATUS READY RESTARTS START NAMESPACE dd-agent-1wml1 datadog/docker-dd-agent:latest Running 1/1 0 2017-06-27 15:20:24 +0900 JSTdefault
23. qucli $ qucli create koudaiii/test Created! quay.io/koudaiii/test $ qucli get koudaiii/test Repository: quay.io/koudaiii/test Visibility: public Permissions: koudaiii(admin)
24. Kubernetes v1.5.6 (2016/04 - 2017/06) •kube-dns ͷ autoscaler ͕ 1.5.0 ͰೖΓɺ SPOF ͱͳ͍ͬͯͨ෦෼͕ղܾ •kops ಋೖ •ෳ਺ Cluster(Production/QA) / sandbox ؀ڥΛߏங •horizontal pod auto scalers Ͱ pod ͷ auto scale Ұ෦ಋೖ •CronJob(ScheduledJob) ͷ UUID ॏෳͷղܾ •طଘαʔϏεͷόονॲཧΛશͯ Kubernetes ΁Ҡߦ •GC ͕ࣦഊ͠ଓ͚Δͱؾ͕͍ͭͨΒ disk ͕ᷓΕɺEvict ͞ΕͯԿ౓΋ Job Λ࠶࣮ߦ •pod Λ࡞੒͢Δࡍʹίϯςφͷ NIC ͷ create ͱ delete ͕ڝ߹ͯ͠ɺ NIC ͷͳ͍ίϯςφ͕ग़དྷ্͕Γ Evict ͞ΕΔ
25. ͜͜·Ͱ࡞੒ͨ͠πʔϧ •dtan4/k8s-job-cleaner •ແݶʹ૿͑ଓ͚Δ Job Λফ͢ •dtan4/k8s-pod-notifier •Job ͕ Fail ͨ͠Β slack ʹ௨஌͢Δ
26. Multi Master (2017/06 - ) •master ఀࢭແ͘ kops Ͱ upgrade Ͱ͖Δ •master ͷ SPOF վળ
27. Kubernetes v1.6.6 (2017/06 - ) •envFrom ಋೖ => ͜ΕʹΑΓ manifest file ʹ؀ڥม਺Λ௥ه͢Δඞཁ͕ͳ͘ͳͬͨ •Cronjob ʹ rotate ͱ͕ೖΔͷͱɺࣦഊճ਺ΛઃఆͰ͖ΔΑ͏ʹͳͬͨ •successfulJobsHistoryLimit •failedJobsHistoryLimit •koudaiii/sltd •Ͳͷ͘Β͍ request ͕དྷ͍ͯΔͷ͔? Http Status Λ؆୯ʹ஌ΔͨΊͷπʔϧ
28. CI/CD ͷऔΓܾΊ
29. CI/CD ͷςϯϓϨʔτ 1. ϒϥϯνΛ੾ΓɺϓϧϦΫΤετΛૹΔ 2. git push ͢Δ౓ʹςετ͕࣮ߦ͞ΕΔ 3. ςετ͕௨Ε͹ QA ʹ deploy ͞Εɺϒϥ΢βͰ֬ೝ͢Δ 4. ϦϦʔεग़དྷΔλΠϛϯάʹͳͬͨΒ master ʹϚʔδ͢Δ 5. CI ্Ͱςετ͕૸Γɺςετ͕௨Ε͹ Production ʹϦϦʔε͢Δ
30. •https://www.wantedly.com/companies/wantedly/post_articles/46089
31. ͦͷଞͷऔΓܾΊ ΢ΣϒΞϓϦέʔγϣϯͷϔϧενΣοΫ΍εςʔλεͷURL͸౷Ұʹ͢Δ (/healthcheck ΍ /ping) ͢΂ͯͷϦϙδτϦʹ script σΟϨΫτϦΛ࡞੒͠ɺΠϯϑϥ࡞ۀΛಉ͡ํ๏Ͱ࣮ߦ Πϯετʔϧ script/bootstrap Ϗϧυεςοϓ script/ci-build σϓϩΠεςοϓ script/ci-deploy (Rolling Deploy ͔ blue-green Deploy ͔͸ࣗ༝ʹॻ͍ͯ΋ྑ͍) αʔόʔىಈ script/server ίϯιʔϧ(one-off container) script/console
32. ϞχλϦϯά
33. ϞχλϦϯά • ͲΜͲΜϚΠΫϩαʔϏεԽ͸ਐΉ => ֤͕ࣗ namespace Λ੾ͬͯར༻Ͱ͖ΔΑ͏ʹ͢Δ • 1ͭ1ͭઃఆ͢ΔΑΓ͸ɺϧʔϧΛܾΊͯ General ʹઃఆ͞ΕΔΑ͏ʹ͢Δ • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}} • kubernetes_state Λ࢖ͬͯɺ deployment ͕͔ͬ͠Γ available ʹͳ͍ͬͯΔ͜ͱΛݟΔ • WebΞϓϦέʔγϣϯ͕ࢧྲྀͳͷͰɺrequest ͱ status code ΋߹ͤͯͰ͖ΔΑ͏ʹ͢Δ • aws.elb.request_count ౳ • ઐ༻μογϡϘʔυͰͲͷ͘Β͍ request ͱ SLA ͕୲อͰ͖͍ͯΔ͔ݟΕΔΑ͏ʹ
35. ϞχλϦϯάͰ࢖༻͍ͯ͠Δ΋ͷ • kelseyhightower/konfd • secret ͔Β ConfigMap Λੜ੒͢Δ • repository ʹύεϫʔυͳͲΛؚΊͨ͘ͳ͍৔߹ʹར༻ • kubernetes/kube-state-metrics • deployment ౳ͷ available Λऔಘ • datadog/docker-dd-agent • datadog ͷ agent Λ docker ༻ɺ http ༻ɺ DB ༻ͱrole ຖʹ deployment ·ͨ͸ DaemonSet Λ࡞͍ͬͯΔ • koudaiii/sltd • Service ͔Β࡞ΒΕΔ ELB ʹ labels Λ tag ʹ௥Ճɺ name ΍ namespace ౳΋߹ΘͤͯELB ʹ tag Λ͚ͭΔ
36. ·ͱΊ • Kubernetes Λ Production Ͱ࢖͏৔߹ʹ΍ͬͨ͜ͱ • ෳ਺Ϋϥελͱ Multi Master ͷ༻ҙ • खؒʹͳΔϞχλϦϯά΍؂ࢹͷࣗಈԽ • CI/CD ͷϧʔϧԽ
37. IUUQTXXXXBOUFEMZDPNQSPKFDUT