Kube - The core tool at Wantedly

Transcript

1. ©2019 Wantedly, Inc.
   Kube
   the core tool at Wantedly
   CloudNative Days Tokyo Meetup w/ Melanie Cebula
   Jul 23, 2019 - Shimpei Otsubo @potsbo
   

   View Slide

2. ©2019 Wantedly, Inc.
   Shimpei Otsubo @potsbo
   Visit Recommendation Squad, Wantedly, Inc.
   2018 New Grads
   Kubernetes
   Dev Tools
   CI / CD
   AuthN / Z
   Productivity Dvorak
   

   View Slide

3. ©2019 Wantedly, Inc.
   Why this talk?
   Provide another case study
   K tool at Airbnb is similar to ours
   But in different ways
   

   View Slide

4. ©2019 Wantedly, Inc.
   Wantedly Scale
   40 Engineers
   6 Infrastructure Engineers
   5 Kubernetes Clusters
   300 namespaces in total
   99% of services are on Kubernetes
   1 cluster for one env - prod / qa / sandbox
   

   View Slide

5. ©2019 Wantedly, Inc.
   Motivation
   To know Kubernetes is NOT their job
   The concept is a little bit too complicated
   Engineers switch around namespaces
   Automatic namespace management is required
   Operation without knowing Kubernetes
   Tool to reduce operation mistakes
   

   View Slide

6. ©2019 Wantedly, Inc.
   implement configure
   This is NOT how we do
   App Engineers
   Products infrastructure
   Infra Engineers
   

   View Slide

7. ©2019 Wantedly, Inc.
   implement configure
   ask / help
   This is NOT how we do
   App Engineers
   Products infrastructure
   Infra Engineers
   

   View Slide

8. ©2019 Wantedly, Inc.
   App Engineers
   Infra Engineers
   implement Access through tools and APIs
   Products infrastructure
   Provide tools
   automate
   This is how we do
   

   View Slide

9. ©2019 Wantedly, Inc.
   Manifest Generator
   Generates namespace, deploy, service, and hpa
   Convention solves almost everything
   [~/src/github.com/wantedly/new-service] $ kube generate autoscale
   generated ~/src/github.com/wantedly/new-service/kubernetes/deployment-with-newrelic.yaml
   generated ~/src/github.com/wantedly/new-service/kubernetes/deployment.yaml
   generated ~/src/github.com/wantedly/new-service/kubernetes/service.yaml
   generated ~/src/github.com/wantedly/new-service/kubernetes/hpa.yaml
   generated ~/src/github.com/wantedly/new-service/kubernetes/namespace.yaml
   No configuration required
   One namespace for one repo
   

   View Slide

10. ©2019 Wantedly, Inc.
    Kubectl Wrapper
    Kube is fully compatible with kubectl
    $ kube $ kubectl
    Automatic Configuration
    completion
    -n
    $ kube get po $ kubectl get po -n some-service
    Credential filled
    Inside a repo called some-service
    

    View Slide

11. ©2019 Wantedly, Inc.
    Kubectl Wrapper
    G
    Token Token
    Teams
    Groups
    RBAC!!
    genmon
    TokenReview
    Results
    Token
    Authentication is automatically managed
    HR team assigns people to correct team
    Infrastructure team manages only RBAC config
    

    View Slide

12. ©2019 Wantedly, Inc.
    Handy Commands to Deploy
    $ kube prod deploy master # branch name
    $ kube prod deploy bebb47 # any commit hash
    $ kube prod deploy -c # current branch
    $ kube prod lock # lock deployment
    $ kube prod canary start release-1
    Deploy
    $ kube rollback # undo last deploy
    $ kube reload # renew replica sets
    Deploy support
    

    View Slide

13. ©2019 Wantedly, Inc.
    $ kube prod deploy master # branch name
    $ kube prod deploy bebb47 # any commit hash
    $ kube prod deploy -c # current branch
    $ kube prod lock # lock deployment
    $ kube prod canary start release-1
    Deploy
    $ kube rollback # undo last deploy
    $ kube reload # renew replica sets
    Deploy support
    [~/.go/src/github.com/wantedly/visit-recommendation] $ kube sandbox deploy master
    Wait for all commit status checks have passed with interval 10s
    Note: status check [codecov/patch codecov/project] ignored.
    ready to deploy.
    deploy to (deployment: "visit-recommendation", container: "visit-recommendation")
    before: quay.io/wantedly/visit-recommendation:b5290905664257de9ebf21a29ac07287c13aea6e
    after: quay.io/wantedly/visit-recommendation:b5290905664257de9ebf21a29ac07287c13aea6e
    deploy to (deployment: "visit-recommendation-newrelic", container: "visit-recommendation")
    before: quay.io/wantedly/visit-recommendation:b5290905664257de9ebf21a29ac07287c13aea6e
    after: quay.io/wantedly/visit-recommendation:b5290905664257de9ebf21a29ac07287c13aea6e
    deploy to (deployment: "worker", container: "visit-recommendation")
    before: quay.io/wantedly/visit-recommendation:b5290905664257de9ebf21a29ac07287c13aea6e
    after: quay.io/wantedly/visit-recommendation:b5290905664257de9ebf21a29ac07287c13aea6e
    deployments successfully updated! check rollout status by `kubectl rollout status deployment/DEPLOYMENT --namespace visit-recommendation`
    Executing "kubectl rollout status deployment/visit-recommendation --namespace visit-recommendation"
    Executing "kubectl rollout status deployment/worker --namespace visit-recommendation"
    Executing "kubectl rollout status deployment/visit-recommendation-newrelic --namespace visit-recommendation"
    2019/07/23 16:43:09 visit-recommendation-newrelic Waiting for deployment "visit-recommendation-newrelic" rollout to finish: 1 old replicas are pending termination...
    2019/07/23 16:43:09 worker Waiting for deployment "worker" rollout to finish: 0 out of 3 new replicas have been updated...
    2019/07/23 16:43:09 visit-recommendation Waiting for deployment "visit-recommendation" rollout to finish: 2 out of 3 new replicas have been updated...
    2019/07/23 16:43:10 worker Waiting for deployment "worker" rollout to finish: 3 old replicas are pending termination...
    2019/07/23 16:43:14 worker Waiting for deployment "worker" rollout to finish: 3 old replicas are pending termination...
    2019/07/23 16:43:14 worker Waiting for deployment "worker" rollout to finish: 3 old replicas are pending termination...
    2019/07/23 16:43:14 worker Waiting for deployment "worker" rollout to finish: 2 old replicas are pending termination...
    2019/07/23 16:43:16 worker Waiting for deployment "worker" rollout to finish: 2 old replicas are pending termination...
    2019/07/23 16:43:16 worker Waiting for deployment "worker" rollout to finish: 2 old replicas are pending termination...
    2019/07/23 16:43:16 worker Waiting for deployment "worker" rollout to finish: 1 old replicas are pending termination...
    2019/07/23 16:43:19 worker Waiting for deployment "worker" rollout to finish: 1 old replicas are pending termination...
    2019/07/23 16:43:19 worker deployment "worker" successfully rolled out
    2019/07/23 16:43:33 visit-recommendation-newrelic Waiting for deployment "visit-recommendation-newrelic" rollout to finish: 1 old replicas are pending termination...
    2019/07/23 16:43:34 visit-recommendation-newrelic deployment "visit-recommendation-newrelic" successfully rolled out
    2019/07/23 16:44:05 visit-recommendation Waiting for deployment "visit-recommendation" rollout to finish: 2 out of 3 new replicas have been updated...
    2019/07/23 16:44:06 visit-recommendation Waiting for deployment "visit-recommendation" rollout to finish: 2 out of 3 new replicas have been updated...
    2019/07/23 16:44:07 visit-recommendation Waiting for deployment "visit-recommendation" rollout to finish: 2 old replicas are pending termination...
    2019/07/23 16:44:07 visit-recommendation Waiting for deployment "visit-recommendation" rollout to finish: 2 old replicas are pending termination...
    2019/07/23 16:44:08 visit-recommendation Waiting for deployment "visit-recommendation" rollout to finish: 1 old replicas are pending termination...
    2019/07/23 16:44:32 visit-recommendation deployment "visit-recommendation" successfully rolled out
    Manages authentication
    GitHub Status check
    Update deployment
    Rollout status simultaneously
    Handy Commands to Deploy
    Resolves docker tag
    List target deployments
    Update GitHub Deployment
    Check deploy lock
    

    View Slide

14. ©2019 Wantedly, Inc.
    Other Handy Commands
    $ kube prod sh master bash
    $ kube sandbox sh c62bbc --port-forward 3000:3000 rails s
    $ kube prod tail # run k8stail similar to stern
    $ kube sandbox dotenv set FOO=bar # Set env vars
    $ kube prod argo list # Use the same config
    Run any command inside the cluster
    $ kube prod bash # run in simplified docker image
    Use cluster resource to check your commit
    Other support command
    

    View Slide

15. ©2019 Wantedly, Inc.
    CI Tool
    language: bash
    env:
    - // encrypted credentials here
    install:
    - bash script:
    - kube ci-build
    deploy:
    - skip_cleanup: true
    provider: script
    script: kube prod deploy $TRAVIS_COMMIT
    on:
    branch: master
    

    View Slide

16. ©2019 Wantedly, Inc.
    CI Tool
    language: bash
    env:
    - // encrypted credentials here
    install:
    - bash script:
    - kube ci-build
    deploy:
    - skip_cleanup: true
    provider: script
    script: kube prod deploy $TRAVIS_COMMIT
    on:
    branch: master
    Install kube
    

    View Slide

17. ©2019 Wantedly, Inc.
    CI Tool
    language: bash
    env:
    - // encrypted credentials here
    install:
    - bash script:
    - kube ci-build
    deploy:
    - skip_cleanup: true
    provider: script
    script: kube prod deploy $TRAVIS_COMMIT
    on:
    branch: master
    Install kube
    build and push a docker image
    

    View Slide

18. ©2019 Wantedly, Inc.
    CI Tool
    language: bash
    env:
    - // encrypted credentials here
    install:
    - bash script:
    - kube ci-build
    deploy:
    - skip_cleanup: true
    provider: script
    script: kube prod deploy $TRAVIS_COMMIT
    on:
    branch: master
    deploy the revision
    Install kube
    build and push a docker image
    

    View Slide

19. ©2019 Wantedly, Inc.
    Architecture
    ~/.wantedly
    !"" bin
    # %"" kube
    !"" credentials
    # %"" github-access-token
    %"" kube
    %"" tools
    %""
    %""
    %"" bin
    # %""
    %"" lib
    %"" …
    1. Install binaries if not present
    2. Compose args for the command
    3. Execute the command
    Installing kube solves everything
    Execute depended binaries
    

    View Slide

20. ©2019 Wantedly, Inc.
    Architecture
    Execute depended binaries
    Cluster and namespace configuration
    Create kubeconfig.yml in tmp dir
    Execute bin with KUBECONFIG=/path/to/the/dir/kubeconfig.yml
    Run set-cluster / set-credentials / set-context / use-context
    Export command
    exports the config to the default path
    $ kube prod export
    Engineers can use any tool they want
    

    View Slide

21. ©2019 Wantedly, Inc.
    Takeaways
    Don’t let them care about infrastructure
    Let engineers focus on their products
    Engineers don’t have to know Docker or Kubernetes
    Add ours to your case studies
    Many ways to create tools like k tool
    

    View Slide