Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Microservices on Kubernetes

Shimpei Otsubo
October 25, 2018
10k

Microservices on Kubernetes

Shimpei Otsubo

October 25, 2018
Tweet

Transcript

  1. ©2018 Wantedly, Inc.
    Microservices on Kubernetes
    Convention, Enforcement, and Migration
    Kubernetes & Cloud Native Meetup
    25.Oct.2018 - Shimpei Otsubo @potsbo

    View full-size slide

  2. ©2018 Wantedly, Inc.
    4IJNQFJ0UTVCP!QPUTCP
    Infrastructure Team, Wantedly, Inc.
    2018 New Grads
    Kubernetes
    Dev Tools
    CI / CD
    AuthN / Z
    Productivity Dvorak

    View full-size slide

  3. ©2018 Wantedly, Inc.
    All Services on
    70 various sizes of microservices
    More than 2 per engineer

    View full-size slide

  4. ©2018 Wantedly, Inc.
    Docker for 4 yrs.
    We have been early adaptors
    Kubernetes for 2.5 yrs.
    We have own best practices
    To manage this # of microservices

    View full-size slide

  5. ©2018 Wantedly, Inc.
    Enforcement
    Conventions
    Migration
    Summary

    View full-size slide

  6. ©2018 Wantedly, Inc.
    Enforcement
    Conventions
    Migration
    Summary

    View full-size slide

  7. ©2018 Wantedly, Inc.
    Convention
    Bootstrap fast, maintain only logic
    one namespace for repo
    company-wide convention
    each env (production, qa, sandbox) has a Kubernetes cluster
    same third-party services across all services
    commit hash as image tag
    automatic deploy after PR merged for most apps
    configuration
    build/deploy
    monitoring
    etc…

    View full-size slide

  8. ©2018 Wantedly, Inc.
    The Latest k8s ops flow Fully managed
    set image
    check
    pull
    deploy
    push push
    hook
    build
    build

    View full-size slide

  9. ©2018 Wantedly, Inc.
    commit hash as docker image tag
    GitHub commit status shows docker image is ready
    Almost any commit is deployable
    push push
    hook
    build
    build
    Fully managed
    The Latest k8s ops flow

    View full-size slide

  10. ©2018 Wantedly, Inc.
    `kube prod deploy master` to deploy
    Wrapper to invoke `kubectl set image`
    Use same image in every purpose prod, qa, and dev
    set image
    check
    pull
    deploy
    Fully managed
    The Latest k8s ops flow
    Automatic deploy to production

    View full-size slide

  11. ©2018 Wantedly, Inc.
    Conventions make development productive
    •commit hash as tag
    •one namespace for one repo

    View full-size slide

  12. ©2018 Wantedly, Inc.
    Enforcement
    Conventions
    Migration
    Summary

    View full-size slide

  13. ©2018 Wantedly, Inc.
    Enforcement
    Library to install everything you need to build an app
    servicex
    • Log format and collection
    • Error collection
    • Performance monitoring
    • Distributed Tracing
    • Client Library for other services
    Following the rule has to be the easiest way
    Available in Golang, Ruby, Python
    Ideally less than an hour to deploy to the production
    a bootstrap document to launch a new app

    View full-size slide

  14. ©2018 Wantedly, Inc.
    Enforcement
    everything with kube command
    Following the rule has to be the easiest way
    In-house build/deploy tool
    Same tools in CI and laptops
    Automation is transparent to developers
    kube build
    kube prod deploy master
    to build and push a docker image
    to deploy to production
    to generate manifest files
    auto generated manifests
    kube generate autoscale

    View full-size slide

  15. ©2018 Wantedly, Inc.
    language: bash
    env:
    - // encrypted credentials here
    install:
    - bash <(curl -sL https://get.wantedlyapp.com/kube)
    script:
    - kube build
    deploy:
    - skip_cleanup: true
    provider: script
    script: kube prod deploy $TRAVIS_COMMIT
    on:
    branch: master
    Every command works in local laptops

    View full-size slide

  16. ©2018 Wantedly, Inc.
    language: bash
    env:
    - // encrypted credentials here
    install:
    - bash <(curl -sL https://get.wantedlyapp.com/kube)
    script:
    - kube build
    deploy:
    - skip_cleanup: true
    provider: script
    script: kube prod deploy $TRAVIS_COMMIT
    on:
    branch: master
    Every command works in local laptops
    Install kube

    View full-size slide

  17. ©2018 Wantedly, Inc.
    language: bash
    env:
    - // encrypted credentials here
    install:
    - bash <(curl -sL https://get.wantedlyapp.com/kube)
    script:
    - kube build
    deploy:
    - skip_cleanup: true
    provider: script
    script: kube prod deploy $TRAVIS_COMMIT
    on:
    branch: master
    Every command works in local laptops
    Install kube
    build and push a docker image

    View full-size slide

  18. ©2018 Wantedly, Inc.
    language: bash
    env:
    - // encrypted credentials here
    install:
    - bash <(curl -sL https://get.wantedlyapp.com/kube)
    script:
    - kube build
    deploy:
    - skip_cleanup: true
    provider: script
    script: kube prod deploy $TRAVIS_COMMIT
    on:
    branch: master
    deploy the revision
    Install kube
    build and push a docker image
    Every command works in local laptops

    View full-size slide

  19. ©2018 Wantedly, Inc.
    Conventions make development productive
    Enforcement to follow the conventions
    •commit hash as tag
    •servicex a library to install everything
    •kube in CI and laptops
    •one namespace for one repo

    View full-size slide

  20. ©2018 Wantedly, Inc.
    Enforcement
    Conventions
    Migration
    Summary

    View full-size slide

  21. ©2018 Wantedly, Inc.
    Migration
    We migrated the biggest legacy app to k8s
    Dockerfile from scratch
    Faster Build
    Hack assets:precompile
    QA/Dev first
    Communication with the backend team
    Drop fast-rollback feature
    Drop automatic deploy
    As test for both Infra team and the backend team

    View full-size slide

  22. ©2018 Wantedly, Inc.
    build
    Legacy
    Build / Deploy Flow Diff
    tag
    Env Var
    Modern
    EC2 Travis
    Env Commit hash
    in images k8s secrets
    Deploy
    blue / green rolling
    Fully AWS Fully Managed
    Infra
    Modern has the best practice
    Strategy
    manual automatic

    View full-size slide

  23. ©2018 Wantedly, Inc.
    %FQMPZTBNFWFSTJPOUPUXPFOWT
    0ME"-#
    /FX"-#
    JOUFSOBM
    DNS

    View full-size slide

  24. ©2018 Wantedly, Inc.
    JOUFSOBM
    DNS
    4PNFPGSFRVFTUTHPUPLT
    ;FSPTFDPOETXJUDIQFSDFOUBHFPGUSBGGJDUIBUHPFTUPLT &BTZUPBCPSU
    0ME"-#
    /FX"-#

    View full-size slide

  25. ©2018 Wantedly, Inc.
    "MMUSBGGJDUPLT
    JOUFSOBM
    DNS
    0ME"-#
    /FX"-#

    View full-size slide

  26. ©2018 Wantedly, Inc.
    "CPSUEVFUPFYQFDUFEFSSPS
    JOUFSOBM
    DNS
    *DBVTFEBOJODJEFOUEVSJOHUIJTPQFSBUJPO CVUHSFBUUPIBWFBSPMMCBDLQBUI
    0ME"-#
    /FX"-#

    View full-size slide

  27. ©2018 Wantedly, Inc.
    5SZBHBJO
    JOUFSOBM
    DNS
    0ME"-#
    /FX"-#

    View full-size slide

  28. ©2018 Wantedly, Inc.
    4XJUDI%/4
    JOUFSOBM
    DNS
    0ME"-#
    /FX"-#

    View full-size slide

  29. ©2018 Wantedly, Inc.
    JOUFSOBM
    DNS
    %POF
    0ME"-#
    /FX"-#

    View full-size slide

  30. ©2018 Wantedly, Inc.
    DNS
    %POF
    /FX"-#

    View full-size slide

  31. ©2018 Wantedly, Inc.
    Conventions make development productive
    Summary
    Migration is worth performing
    Enforcement to follow the conventions
    •same third-party services
    •commit hash as tag
    •automatic deploy
    •servicex a library to install everything
    •kube in CI and laptops
    •one namespace for one repo
    •temporary exception is ok
    •but, have to follow the rule sooner or later

    View full-size slide

  32. ©2018 Wantedly, Inc.
    We’re Hiring
    .JDSPTFSWJDFT.POEBZ
    HJUIVCDPNXBOUFEMZNJDSPTFSWJDFT@NPOEBZ

    View full-size slide