Kube - The core tool at Wantedly

Transcript

1. ©2019 Wantedly, Inc. Kube the core tool at Wantedly CloudNative Days Tokyo Meetup w/ Melanie Cebula Jul 23, 2019 - Shimpei Otsubo @potsbo
2. ©2019 Wantedly, Inc. Shimpei Otsubo @potsbo Visit Recommendation Squad, Wantedly, Inc. 2018 New Grads Kubernetes Dev Tools CI / CD AuthN / Z Productivity Dvorak
3. ©2019 Wantedly, Inc. Why this talk? Provide another case study K tool at Airbnb is similar to ours But in different ways
4. ©2019 Wantedly, Inc. Wantedly Scale 40 Engineers 6 Infrastructure Engineers 5 Kubernetes Clusters 300 namespaces in total 99% of services are on Kubernetes 1 cluster for one env - prod / qa / sandbox
5. ©2019 Wantedly, Inc. Motivation To know Kubernetes is NOT their job The concept is a little bit too complicated Engineers switch around namespaces Automatic namespace management is required Operation without knowing Kubernetes Tool to reduce operation mistakes
6. ©2019 Wantedly, Inc. implement configure This is NOT how we do App Engineers Products infrastructure Infra Engineers
7. ©2019 Wantedly, Inc. implement configure ask / help This is NOT how we do App Engineers Products infrastructure Infra Engineers
8. ©2019 Wantedly, Inc. App Engineers Infra Engineers implement Access through tools and APIs Products infrastructure Provide tools automate This is how we do
9. ©2019 Wantedly, Inc. Manifest Generator Generates namespace, deploy, service, and hpa Convention solves almost everything [~/src/github.com/wantedly/new-service] $ kube generate autoscale generated ~/src/github.com/wantedly/new-service/kubernetes/deployment-with-newrelic.yaml generated ~/src/github.com/wantedly/new-service/kubernetes/deployment.yaml generated ~/src/github.com/wantedly/new-service/kubernetes/service.yaml generated ~/src/github.com/wantedly/new-service/kubernetes/hpa.yaml generated ~/src/github.com/wantedly/new-service/kubernetes/namespace.yaml No configuration required One namespace for one repo
10. ©2019 Wantedly, Inc. Kubectl Wrapper Kube is fully compatible with kubectl $ kube <cluster name> <args> $ kubectl <args> Automatic Configuration completion -n <namespace> $ kube <cluster name> get po $ kubectl get po -n some-service Credential filled Inside a repo called some-service
11. ©2019 Wantedly, Inc. Kubectl Wrapper G Token Token Teams Groups RBAC!! genmon TokenReview Results Token Authentication is automatically managed HR team assigns people to correct team Infrastructure team manages only RBAC config
12. ©2019 Wantedly, Inc. Handy Commands to Deploy $ kube prod deploy master # branch name $ kube prod deploy bebb47 # any commit hash $ kube prod deploy -c # current branch $ kube prod lock # lock deployment $ kube prod canary start release-1 Deploy $ kube rollback # undo last deploy $ kube reload # renew replica sets Deploy support
13. ©2019 Wantedly, Inc. $ kube prod deploy master # branch name $ kube prod deploy bebb47 # any commit hash $ kube prod deploy -c # current branch $ kube prod lock # lock deployment $ kube prod canary start release-1 Deploy $ kube rollback # undo last deploy $ kube reload # renew replica sets Deploy support [~/.go/src/github.com/wantedly/visit-recommendation] $ kube sandbox deploy master Wait for all commit status checks have passed with interval 10s Note: status check [codecov/patch codecov/project] ignored. ready to deploy. deploy to (deployment: "visit-recommendation", container: "visit-recommendation") before: quay.io/wantedly/visit-recommendation:b5290905664257de9ebf21a29ac07287c13aea6e after: quay.io/wantedly/visit-recommendation:b5290905664257de9ebf21a29ac07287c13aea6e deploy to (deployment: "visit-recommendation-newrelic", container: "visit-recommendation") before: quay.io/wantedly/visit-recommendation:b5290905664257de9ebf21a29ac07287c13aea6e after: quay.io/wantedly/visit-recommendation:b5290905664257de9ebf21a29ac07287c13aea6e deploy to (deployment: "worker", container: "visit-recommendation") before: quay.io/wantedly/visit-recommendation:b5290905664257de9ebf21a29ac07287c13aea6e after: quay.io/wantedly/visit-recommendation:b5290905664257de9ebf21a29ac07287c13aea6e deployments successfully updated! check rollout status by `kubectl rollout status deployment/DEPLOYMENT --namespace visit-recommendation` Executing "kubectl rollout status deployment/visit-recommendation --namespace visit-recommendation" Executing "kubectl rollout status deployment/worker --namespace visit-recommendation" Executing "kubectl rollout status deployment/visit-recommendation-newrelic --namespace visit-recommendation" 2019/07/23 16:43:09 visit-recommendation-newrelic Waiting for deployment "visit-recommendation-newrelic" rollout to finish: 1 old replicas are pending termination... 2019/07/23 16:43:09 worker Waiting for deployment "worker" rollout to finish: 0 out of 3 new replicas have been updated... 2019/07/23 16:43:09 visit-recommendation Waiting for deployment "visit-recommendation" rollout to finish: 2 out of 3 new replicas have been updated... 2019/07/23 16:43:10 worker Waiting for deployment "worker" rollout to finish: 3 old replicas are pending termination... 2019/07/23 16:43:14 worker Waiting for deployment "worker" rollout to finish: 3 old replicas are pending termination... 2019/07/23 16:43:14 worker Waiting for deployment "worker" rollout to finish: 3 old replicas are pending termination... 2019/07/23 16:43:14 worker Waiting for deployment "worker" rollout to finish: 2 old replicas are pending termination... 2019/07/23 16:43:16 worker Waiting for deployment "worker" rollout to finish: 2 old replicas are pending termination... 2019/07/23 16:43:16 worker Waiting for deployment "worker" rollout to finish: 2 old replicas are pending termination... 2019/07/23 16:43:16 worker Waiting for deployment "worker" rollout to finish: 1 old replicas are pending termination... 2019/07/23 16:43:19 worker Waiting for deployment "worker" rollout to finish: 1 old replicas are pending termination... 2019/07/23 16:43:19 worker deployment "worker" successfully rolled out 2019/07/23 16:43:33 visit-recommendation-newrelic Waiting for deployment "visit-recommendation-newrelic" rollout to finish: 1 old replicas are pending termination... 2019/07/23 16:43:34 visit-recommendation-newrelic deployment "visit-recommendation-newrelic" successfully rolled out 2019/07/23 16:44:05 visit-recommendation Waiting for deployment "visit-recommendation" rollout to finish: 2 out of 3 new replicas have been updated... 2019/07/23 16:44:06 visit-recommendation Waiting for deployment "visit-recommendation" rollout to finish: 2 out of 3 new replicas have been updated... 2019/07/23 16:44:07 visit-recommendation Waiting for deployment "visit-recommendation" rollout to finish: 2 old replicas are pending termination... 2019/07/23 16:44:07 visit-recommendation Waiting for deployment "visit-recommendation" rollout to finish: 2 old replicas are pending termination... 2019/07/23 16:44:08 visit-recommendation Waiting for deployment "visit-recommendation" rollout to finish: 1 old replicas are pending termination... 2019/07/23 16:44:32 visit-recommendation deployment "visit-recommendation" successfully rolled out Manages authentication GitHub Status check Update deployment Rollout status simultaneously Handy Commands to Deploy Resolves docker tag List target deployments Update GitHub Deployment Check deploy lock
14. ©2019 Wantedly, Inc. Other Handy Commands $ kube prod sh master bash $ kube sandbox sh c62bbc --port-forward 3000:3000 rails s $ kube prod tail # run k8stail similar to stern $ kube sandbox dotenv set FOO=bar # Set env vars $ kube prod argo list # Use the same config Run any command inside the cluster $ kube prod bash # run in simplified docker image Use cluster resource to check your commit Other support command
15. ©2019 Wantedly, Inc. CI Tool language: bash env: - // encrypted credentials here install: - bash <(curl -sL https://get.wantedlyapp.com/kube) script: - kube ci-build deploy: - skip_cleanup: true provider: script script: kube prod deploy $TRAVIS_COMMIT on: branch: master
16. ©2019 Wantedly, Inc. CI Tool language: bash env: - // encrypted credentials here install: - bash <(curl -sL https://get.wantedlyapp.com/kube) script: - kube ci-build deploy: - skip_cleanup: true provider: script script: kube prod deploy $TRAVIS_COMMIT on: branch: master Install kube
17. ©2019 Wantedly, Inc. CI Tool language: bash env: - // encrypted credentials here install: - bash <(curl -sL https://get.wantedlyapp.com/kube) script: - kube ci-build deploy: - skip_cleanup: true provider: script script: kube prod deploy $TRAVIS_COMMIT on: branch: master Install kube build and push a docker image
18. ©2019 Wantedly, Inc. CI Tool language: bash env: - // encrypted credentials here install: - bash <(curl -sL https://get.wantedlyapp.com/kube) script: - kube ci-build deploy: - skip_cleanup: true provider: script script: kube prod deploy $TRAVIS_COMMIT on: branch: master deploy the revision Install kube build and push a docker image
19. ©2019 Wantedly, Inc. Architecture ~/.wantedly !"" bin # %"" kube !"" credentials # %"" github-access-token %"" kube %"" tools %"" <tool name> %"" <version> %"" bin # %"" <tool name> %"" lib %"" … 1. Install binaries if not present 2. Compose args for the command 3. Execute the command Installing kube solves everything Execute depended binaries
20. ©2019 Wantedly, Inc. Architecture Execute depended binaries Cluster and namespace configuration Create kubeconfig.yml in tmp dir Execute bin with KUBECONFIG=/path/to/the/dir/kubeconfig.yml Run set-cluster / set-credentials / set-context / use-context Export command exports the config to the default path $ kube prod export Engineers can use any tool they want
21. ©2019 Wantedly, Inc. Takeaways Don’t let them care about infrastructure Let engineers focus on their products Engineers don’t have to know Docker or Kubernetes Add ours to your case studies Many ways to create tools like k tool