Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes Best Practices

Kubernetes Best Practices

Kubernetes and friends are powerful tools that can really simplify your operations. However, there are many gotchas and common pitfalls that can ruin your experience. I’ll share some best practices around building and deploying your containers that will let you run more stably, efficiently, and securely.

Sandeep Dinesh

July 14, 2017
Tweet

More Decks by Sandeep Dinesh

Other Decks in Technology

Transcript

  1. Sandeep Dinesh
    Developer Advocate
    @sandeepdinesh
    github.com/thesandlord
    Kubernetes Best Practices

    View Slide

  2. Google Cloud Platform 2
    Kubernetes is really flexible

    View Slide

  3. Google Cloud Platform 3
    But you might yourself in the

    View Slide

  4. Building Containers

    View Slide

  5. Google Cloud Platform 5
    Don’t trust arbitrary base images!

    View Slide

  6. 6
    Google Cloud Platform
    Static Analysis of Containers
    https://github.com/banyanops/collector
    https://github.com/coreos/clair

    View Slide

  7. Google Cloud Platform 7
    Use small base images

    View Slide

  8. 8
    Google Cloud Platform
    Overhead
    Node.js App
    Your App → 5MB
    Your App’s Dependencies → 95MB
    Total App Size → 100MB
    Docker Base Images:
    node:8 → 667MB
    node:8-wheezy → 521MB
    node:8-slim → 225MB
    node:8-alpine → 63.7MB
    scratch → ~50MB

    View Slide

  9. 9
    Google Cloud Platform
    Overhead
    Node.js App
    Your App → 5MB
    Your App’s Dependencies → 95MB
    Total App Size → 100MB
    Docker Base Images:
    node:8 → 667MB
    node:8-wheezy → 521MB
    node:8-slim → 225MB
    node:8-alpine → 63.7MB
    scratch → ~50MB
    ← 6.6x App Size!!

    View Slide

  10. 10
    Google Cloud Platform
    Overhead
    Node.js App
    Your App → 5MB
    Your App’s Dependencies → 95MB
    Total App Size → 100MB
    Docker Base Images:
    node:8 → 667MB
    node:8-wheezy → 521MB
    node:8-slim → 225MB
    node:8-alpine → 63.7MB
    scratch → ~50MB
    ← 6.6x App Size!!

    13.3x “min”
    overhead!!

    View Slide

  11. 11
    Google Cloud Platform
    Overhead
    Node.js App
    Your App → 5MB
    Your App’s Dependencies → 95MB
    Total App Size → 100MB
    Docker Base Images:
    node:8 → 667MB
    node:8-wheezy → 521MB
    node:8-slim → 225MB
    node:8-alpine → 63.7MB
    scratch → ~50MB
    ← 6.6x App Size!!

    13.3x “min”
    overhead!!
    Pros:
    Builds are faster
    Need less storage
    Cold starts (image pull) are faster
    Potentially less attack surface
    Cons:
    Less tooling inside container
    “Non-standard” environment

    View Slide

  12. Google Cloud Platform 12
    Use the “builder pattern”

    View Slide

  13. 13
    Google Cloud Platform
    Code
    Build Container
    Compiler
    Dev Deps
    Unit Tests
    etc...
    Build Artifact(s) Runtime Container
    Runtime Env
    Debug/Monitor Tooling
    Binaries
    Static Files
    Bundles
    Transpiled Code

    View Slide

  14. Google Cloud Platform 14
    Docker bringing native support for multi-stage
    builds in Docker CE 17.05

    View Slide

  15. Container Internals

    View Slide

  16. Google Cloud Platform 16
    Use a non-root user inside the container

    View Slide

  17. 17
    Google Cloud Platform
    FROM node:alpine
    RUN apk update && apk add imagemagick
    RUN groupadd -r nodejs
    RUN useradd -m -r -g nodejs nodejs
    USER nodejs
    ADD package.json package.json
    RUN npm install
    ADD index.js index.js
    CMD npm start
    Example Dockerfile

    View Slide

  18. 18
    Google Cloud Platform
    Enforce it!
    apiVersion: v1
    kind: Pod
    metadata:
    name: hello-world
    spec:
    containers:
    # specification of the pod’s containers
    # ...
    securityContext:
    runAsNonRoot: true

    View Slide

  19. Google Cloud Platform 19
    Make the filesystem read-only

    View Slide

  20. 20
    Google Cloud Platform
    apiVersion: v1
    kind: Pod
    metadata:
    name: hello-world
    spec:
    containers:
    # specification of the pod’s containers
    # ...
    securityContext:
    runAsNonRoot: true
    readOnlyRootFilesystem: true
    Enforce it!

    View Slide

  21. Google Cloud Platform 21
    One process per container

    View Slide

  22. Google Cloud Platform 22
    Don’t restart on failure. Crash cleanly instead.

    View Slide

  23. Google Cloud Platform 23
    Log to stdout and stderr

    View Slide

  24. Google Cloud Platform 24
    Add “dumb-init” to prevent zombie processes

    View Slide

  25. 25
    Google Cloud Platform
    FROM node:alpine
    RUN apk update && apk add imagemagick
    RUN groupadd -r nodejs
    RUN useradd -m -r -g nodejs nodejs
    USER nodejs
    ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 \
    /usr/local/bin/dumb-init
    RUN chmod +x /usr/local/bin/dumb-init
    ENTRYPOINT ["/usr/bin/dumb-init", "--"]
    ADD package.json package.json
    RUN npm install
    ADD index.js index.js
    CMD npm start
    Example Dockerfile

    View Slide

  26. Google Cloud Platform 26
    Good News: No need to do this in K8s 1.7

    View Slide

  27. Deployments

    View Slide

  28. Google Cloud Platform 28
    Use the “record” option for easier rollbacks

    View Slide

  29. 29
    Google Cloud Platform
    $ kubectl apply -f deployment.yaml --record

    $ kubectl rollout history deployments my-deployment
    deployments "ghost-recorded"
    REVISION CHANGE-CAUSE
    1 kubectl apply -f deployment.yaml --record
    2 kubectl edit deployments my-deployment
    3 kubectl set image deployment/my-deplyoment my-container=app:2.0

    View Slide

  30. Google Cloud Platform 30
    Use plenty of descriptive labels

    View Slide

  31. 31
    Google Cloud Platform
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: web
    spec:
    replicas: 12
    template:
    metadata:
    labels:
    name: web
    color: blue
    experimental: 'true'
    Labels

    View Slide

  32. 32
    Google Cloud Platform
    App: Nifty
    Phase: Dev
    Role: FE
    App: Nifty
    Phase: Test
    Role: FE
    App: Nifty
    Phase: Dev
    Role: BE
    App: Nifty
    Phase: Test
    Role: BE
    Labels

    View Slide

  33. 33
    Google Cloud Platform
    App: Nifty
    Phase: Dev
    Role: FE
    App: Nifty
    Phase: Test
    Role: FE
    App: Nifty
    Phase: Dev
    Role: BE
    App: Nifty
    Phase: Test
    Role: BE
    Labels
    App = Nifty

    View Slide

  34. 34
    Google Cloud Platform
    App: Nifty
    Phase: Dev
    Role: FE
    App: Nifty
    Phase: Test
    Role: FE
    App: Nifty
    Phase: Dev
    Role: BE
    App: Nifty
    Phase: Test
    Role: BE
    Labels
    Role = BE

    View Slide

  35. 35
    Google Cloud Platform
    App: Nifty
    Phase: Dev
    Role: FE
    App: Nifty
    Phase: Test
    Role: FE
    App: Nifty
    Phase: Dev
    Role: BE
    App: Nifty
    Phase: Test
    Role: BE
    Labels
    Phase = Dev

    View Slide

  36. 36
    Google Cloud Platform
    App: Nifty
    Phase: Dev
    Role: FE
    App: Nifty
    Phase: Test
    Role: FE
    App: Nifty
    Phase: Dev
    Role: BE
    App: Nifty
    Phase: Test
    Role: BE
    Labels
    Phase = Dev
    Role = FE

    View Slide

  37. Google Cloud Platform 37
    Use sidecar containers for proxies, watchers, etc

    View Slide

  38. 38
    Google Cloud Platform
    Examples
    App
    Database
    localhost
    App App
    Proxy Proxy Proxy
    secure
    connection

    View Slide

  39. 39
    Google Cloud Platform
    Auth, Rate
    Limiting, etc.
    Examples
    App
    Incoming Requests
    localhost
    App App
    Proxy Proxy Proxy

    View Slide

  40. Google Cloud Platform 40
    Don’t use sidecars for bootstrapping!

    View Slide

  41. Google Cloud Platform 41
    Use init containers instead!

    View Slide

  42. 42
    Google Cloud Platform
    apiVersion: v1
    kind: Pod
    metadata:
    name: awesomeapp-pod
    labels:
    app: awesomeapp
    annotations:
    pod.beta.kubernetes.io/init-containers
    : '[
    {
    "name": "init-myapp",
    "image": "busybox",
    "command": ["sh", "-c", "until nslookup myapp; do echo waiting for myapp; sleep 2; done;"]
    },
    {
    "name": "init-mydb",
    "image": "busybox",
    "command": ["sh", "-c", "until nslookup mydb; do echo waiting for mydb; sleep 2; done;"]
    }
    ]'
    spec:
    containers:
    - name: awesomeapp-container
    image: busybox
    command: ['sh', '-c', 'echo The app is running! && sleep 3600'
    ]

    View Slide

  43. Google Cloud Platform 43
    Don’t use :latest or no tag

    View Slide

  44. Google Cloud Platform 44
    Readiness and Liveness probes are your friend

    View Slide

  45. 45
    Google Cloud Platform
    Readiness → Is the app ready to start serving traffic?
    ● Won’t be added to a service endpoint until it passes
    ● Required for a “production app” in my opinion
    Liveness → Is the app still running?
    ● Default is “process is running”
    ● Possible that the process can be running but not working correctly
    ● Good to define, might not be 100% necessary
    These can sometimes be the same endpoint, but not always
    Health Checks

    View Slide

  46. Services

    View Slide

  47. Google Cloud Platform 47
    Don’t always use type: LoadBalancer

    View Slide

  48. Google Cloud Platform 48
    Ingress is great

    View Slide

  49. 49
    Google Cloud Platform
    Ingress
    Service 1 Service 2 Service 3
    websocket.mydomain.com mydomain.com
    /foo /bar

    View Slide

  50. Google Cloud Platform 50
    type: NodePort can be “good enough”

    View Slide

  51. Google Cloud Platform 51
    Use Static IPs. They are free*!

    View Slide

  52. 52
    Google Cloud Platform
    apiVersion: v1
    kind: Service
    metadata:
    name: myservice
    spec:
    type: LoadBalancer
    loadBalancerIP: QQQ.ZZZ.YYY.XXX
    ports:
    - port: 80
    targetPort: 3000
    protocol: TCP
    selector:
    name: myapp
    $ gcloud compute addresses create ingress --global

    $ gcloud compute addresses create myservice --region=us-west1
    Created …
    address: QQQ.ZZZ.YYY.XXX

    $
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    name: myingress
    annotations:
    kubernetes.io/ingress.global-static-ip-name: "ingress"
    spec:
    backend:
    serviceName: myservice
    servicePort: 80

    View Slide

  53. Google Cloud Platform 53
    Map external services to internal ones

    View Slide

  54. 54
    Google Cloud Platform
    External Services
    kind: Service
    apiVersion: v1
    metadata:
    name: mydatabase
    namespace: prod
    spec:
    type: ExternalName
    externalName : my.database.example.com
    ports:
    - port: 12345
    kind: Service
    apiVersion: v1
    metadata:
    name: mydatabase
    spec:
    ports:
    - protocol: TCP
    port: 80
    targetPort: 12345
    kind: Endpoints
    apiVersion: v1
    metadata:
    name: mydatabase
    subsets:
    - addresses:
    - ip: 10.128.0.2
    ports:
    - port: 12345
    Hosted Database Database outside cluster but inside network

    View Slide

  55. Application Architecture

    View Slide

  56. Google Cloud Platform 56
    Use Helm Charts

    View Slide

  57. Google Cloud Platform 57
    ALL downstream dependencies are unreliable

    View Slide

  58. Google Cloud Platform 58
    Make sure your microservices aren’t too micro

    View Slide

  59. Google Cloud Platform 59
    Use a “Service Mesh”

    View Slide

  60. 60
    Google Cloud Platform
    https://github.com/istio/istio
    https://github.com/linkerd/linkerd

    View Slide

  61. Google Cloud Platform 61
    Use a PaaS?

    View Slide

  62. 62
    Google Cloud Platform

    View Slide

  63. Cluster Management

    View Slide

  64. Google Cloud Platform 64
    Use Google Container Engine

    View Slide

  65. Google Cloud Platform 65
    Resources, Anti-Affinity, and Scheduling

    View Slide

  66. 66
    Google Cloud Platform
    Node Affinity
    hostname
    zone
    region
    instance-type
    os
    arch
    custom!

    View Slide

  67. 67
    Google Cloud Platform
    Node Taints / Tolerations
    special hardware
    dedicated hosts
    etc

    View Slide

  68. 68
    Google Cloud Platform
    Pod Affinity / Anti-Affinity
    hostname
    zone
    region

    View Slide

  69. Google Cloud Platform 69
    Use Namespaces to split up your cluster

    View Slide

  70. Google Cloud Platform 70
    Role Based Access Control

    View Slide

  71. Google Cloud Platform 71
    Unleash the Chaos Monkey

    View Slide

  72. 72
    Google Cloud Platform
    More Resources
    ● http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html
    ● https://github.com/gravitational/workshop/blob/master/k8sprod.md
    ● https://nodesource.com/blog/8-protips-to-start-killing-it-when-dockerizing-node-js/
    ● https://www.ianlewis.org/en/using-kubernetes-health-checks
    ● https://www.linux.com/learn/rolling-updates-and-rollbacks-using-kubernetes-deployments
    ● https://kubernetes.io/docs/api-reference/v1.6/

    View Slide

  73. Questions?
    What best practices do you have?

    View Slide