Slide 1

Slide 1 text

©2018 Wantedly, Inc. Microservices on Kubernetes Convention, Enforcement, and Migration Kubernetes & Cloud Native Meetup 25.Oct.2018 - Shimpei Otsubo @potsbo

Slide 2

Slide 2 text

©2018 Wantedly, Inc. 4IJNQFJ0UTVCP!QPUTCP Infrastructure Team, Wantedly, Inc. 2018 New Grads Kubernetes Dev Tools CI / CD AuthN / Z Productivity Dvorak

Slide 3

Slide 3 text

©2018 Wantedly, Inc. All Services on 70 various sizes of microservices More than 2 per engineer

Slide 4

Slide 4 text

©2018 Wantedly, Inc. Docker for 4 yrs. We have been early adaptors Kubernetes for 2.5 yrs. We have own best practices To manage this # of microservices

Slide 5

Slide 5 text

©2018 Wantedly, Inc. Enforcement Conventions Migration Summary

Slide 6

Slide 6 text

©2018 Wantedly, Inc. Enforcement Conventions Migration Summary

Slide 7

Slide 7 text

©2018 Wantedly, Inc. Convention Bootstrap fast, maintain only logic one namespace for repo company-wide convention each env (production, qa, sandbox) has a Kubernetes cluster same third-party services across all services commit hash as image tag automatic deploy after PR merged for most apps configuration build/deploy monitoring etc…

Slide 8

Slide 8 text

©2018 Wantedly, Inc. The Latest k8s ops flow Fully managed set image check pull deploy push push hook build build

Slide 9

Slide 9 text

©2018 Wantedly, Inc. commit hash as docker image tag GitHub commit status shows docker image is ready Almost any commit is deployable push push hook build build Fully managed The Latest k8s ops flow

Slide 10

Slide 10 text

©2018 Wantedly, Inc. `kube prod deploy master` to deploy Wrapper to invoke `kubectl set image` Use same image in every purpose prod, qa, and dev set image check pull deploy Fully managed The Latest k8s ops flow Automatic deploy to production

Slide 11

Slide 11 text

©2018 Wantedly, Inc. Conventions make development productive •commit hash as tag •one namespace for one repo

Slide 12

Slide 12 text

©2018 Wantedly, Inc. Enforcement Conventions Migration Summary

Slide 13

Slide 13 text

©2018 Wantedly, Inc. Enforcement Library to install everything you need to build an app servicex • Log format and collection • Error collection • Performance monitoring • Distributed Tracing • Client Library for other services Following the rule has to be the easiest way Available in Golang, Ruby, Python Ideally less than an hour to deploy to the production a bootstrap document to launch a new app

Slide 14

Slide 14 text

©2018 Wantedly, Inc. Enforcement everything with kube command Following the rule has to be the easiest way In-house build/deploy tool Same tools in CI and laptops Automation is transparent to developers kube build kube prod deploy master to build and push a docker image to deploy to production to generate manifest files auto generated manifests kube generate autoscale

Slide 15

Slide 15 text

©2018 Wantedly, Inc. language: bash env: - // encrypted credentials here install: - bash <(curl -sL https://get.wantedlyapp.com/kube) script: - kube build deploy: - skip_cleanup: true provider: script script: kube prod deploy $TRAVIS_COMMIT on: branch: master Every command works in local laptops

Slide 16

Slide 16 text

©2018 Wantedly, Inc. language: bash env: - // encrypted credentials here install: - bash <(curl -sL https://get.wantedlyapp.com/kube) script: - kube build deploy: - skip_cleanup: true provider: script script: kube prod deploy $TRAVIS_COMMIT on: branch: master Every command works in local laptops Install kube

Slide 17

Slide 17 text

©2018 Wantedly, Inc. language: bash env: - // encrypted credentials here install: - bash <(curl -sL https://get.wantedlyapp.com/kube) script: - kube build deploy: - skip_cleanup: true provider: script script: kube prod deploy $TRAVIS_COMMIT on: branch: master Every command works in local laptops Install kube build and push a docker image

Slide 18

Slide 18 text

©2018 Wantedly, Inc. language: bash env: - // encrypted credentials here install: - bash <(curl -sL https://get.wantedlyapp.com/kube) script: - kube build deploy: - skip_cleanup: true provider: script script: kube prod deploy $TRAVIS_COMMIT on: branch: master deploy the revision Install kube build and push a docker image Every command works in local laptops

Slide 19

Slide 19 text

©2018 Wantedly, Inc. Conventions make development productive Enforcement to follow the conventions •commit hash as tag •servicex a library to install everything •kube in CI and laptops •one namespace for one repo

Slide 20

Slide 20 text

©2018 Wantedly, Inc. Enforcement Conventions Migration Summary

Slide 21

Slide 21 text

©2018 Wantedly, Inc. Migration We migrated the biggest legacy app to k8s Dockerfile from scratch Faster Build Hack assets:precompile QA/Dev first Communication with the backend team Drop fast-rollback feature Drop automatic deploy As test for both Infra team and the backend team

Slide 22

Slide 22 text

©2018 Wantedly, Inc. build Legacy Build / Deploy Flow Diff tag Env Var Modern EC2 Travis Env Commit hash in images k8s secrets Deploy blue / green rolling Fully AWS Fully Managed Infra Modern has the best practice Strategy manual automatic

Slide 23

Slide 23 text

©2018 Wantedly, Inc. %FQMPZTBNFWFSTJPOUPUXPFOWT 0ME"-# /FX"-# JOUFSOBM DNS

Slide 24

Slide 24 text

©2018 Wantedly, Inc. JOUFSOBM DNS 4PNFPGSFRVFTUTHPUPLT ;FSPTFDPOETXJUDIQFSDFOUBHFPGUSBGGJDUIBUHPFTUPLT &BTZUPBCPSU 0ME"-# /FX"-#

Slide 25

Slide 25 text

©2018 Wantedly, Inc. "MMUSBGGJDUPLT JOUFSOBM DNS 0ME"-# /FX"-#

Slide 26

Slide 26 text

©2018 Wantedly, Inc. "CPSUEVFUPFYQFDUFEFSSPS JOUFSOBM DNS *DBVTFEBOJODJEFOUEVSJOHUIJTPQFSBUJPO CVUHSFBUUPIBWFBSPMMCBDLQBUI 0ME"-# /FX"-#

Slide 27

Slide 27 text

©2018 Wantedly, Inc. 5SZBHBJO JOUFSOBM DNS 0ME"-# /FX"-#

Slide 28

Slide 28 text

©2018 Wantedly, Inc. 4XJUDI%/4 JOUFSOBM DNS 0ME"-# /FX"-#

Slide 29

Slide 29 text

©2018 Wantedly, Inc. JOUFSOBM DNS %POF 0ME"-# /FX"-#

Slide 30

Slide 30 text

©2018 Wantedly, Inc. DNS %POF /FX"-#

Slide 31

Slide 31 text

©2018 Wantedly, Inc. Conventions make development productive Summary Migration is worth performing Enforcement to follow the conventions •same third-party services •commit hash as tag •automatic deploy •servicex a library to install everything •kube in CI and laptops •one namespace for one repo •temporary exception is ok •but, have to follow the rule sooner or later

Slide 32

Slide 32 text

©2018 Wantedly, Inc. We’re Hiring .JDSPTFSWJDFT.POEBZ HJUIVCDPNXBOUFEMZNJDSPTFSWJDFT@NPOEBZ