$30 off During Our Annual Pro Sale. View Details »

Running_Legacy_Apps_on_K8s.pdf

Joshua Timberman
May 21, 2019
2.1k

 Running_Legacy_Apps_on_K8s.pdf

Joshua Timberman

May 21, 2019
Tweet

Transcript

  1. Legacy Applications on
    Kubernetes
    Joshua Timberman
    @jtimberman

    View Slide

  2. https://commons.wikimedia.org/wiki/File:BalticServers_data_center.jpg
    % whoami

    View Slide

  3. Why are we here?

    View Slide

  4. Running Legacy Applications on
    Kubernetes

    View Slide

  5. Running Legacy Heritage
    Applications on Kubernetes

    View Slide

  6. What is a Legacy Application?
    In computing, a legacy application program of, relating to, or
    being a previous or outdated computer system, yet still in use.
    Often referencing a system as "legacy" means that it paved
    the way for the standards that would follow it. This can
    also imply that the system is out of date or in need of
    replacement.

    View Slide

  7. Chef Ops runs many applications

    View Slide

  8. Applications, sample set
    • Hosted Chef (api.chef.io)
    • Downloads (downloads.chef.io)
    • Omnitruck (omnitruck.chef.io)
    • Learn Chef (learn.chef.io)
    • Chef Community Cookbooks (supermarket.chef.io)
    • Internal and External utilities
    • "Gateway" and "glue" services
    • Kubernetes itself ;)
    • etc...

    View Slide

  9. Workflow Matters

    View Slide

  10. Why Kubernetes?

    View Slide

  11. Cloud Native

    View Slide

  12. Platform Consistency

    View Slide

  13. Cohesion

    View Slide

  14. Cost-savings
    https://en.wikipedia.org/wiki/Denomination_effect#/media/File:Exchange_Money_Conversion_to_Foreign_Currency.jpg

    View Slide

  15. https://commons.wikimedia.org/wiki/File:Frozen_peas.JPG

    View Slide

  16. https://commons.wikimedia.org/wiki/File:Mashed_potatoes_(1).jpg

    View Slide

  17. What about Chef?

    View Slide

  18. As it turns out...
    https://www.flickr.com/photos/tofu_mugwump/22420266811
    https://creativecommons.org/licenses/by/2.0/

    View Slide

  19. We still have an OS.

    View Slide

  20. But we use K8SaaS!
    https://www.jpl.nasa.gov/news/news.php?feature=7113

    View Slide

  21. View Slide

  22. Policyfiles
    https://en.wikipedia.org/wiki/Ledger#/media/File:Ledger.png

    View Slide

  23. View Slide

  24. Test Kitchen (for cookbooks)
    https://commons.wikimedia.org/wiki/File:Conference_House_-_Rustic_Kitchen.jpg

    View Slide

  25. DevSec Linux Baseline
    https://dev-sec.io/baselines/linux/

    View Slide

  26. CIS AWS Benchmark
    https://commons.wikimedia.org/wiki/File:Benchmark_at_former_Abbey_Cinema.jpg

    View Slide

  27. View Slide

  28. Ruby Scaffolding
    https://commons.wikimedia.org/wiki/File:Seattle_-
    _construction_at_SE_corner_of_11th_%26_E_Pine_01.jpg

    View Slide

  29. core-plans/scaffolding-base
    core-plans/scaffolding-chef
    core-plans/scaffolding-go
    core-plans/scaffolding-go17
    core-plans/scaffolding-gradle
    core-plans/scaffolding-inspec
    core-plans/scaffolding-node
    core-plans/scaffolding-ruby
    scaffolding plans

    View Slide

  30. Effortless Config

    View Slide

  31. How did we get here?

    View Slide

  32. View Slide

  33. https://flic.kr/p/bFe9Bs

    View Slide

  34. Kubernetes is a portable,
    extensible open-source platform
    for managing containerized
    workloads and services, that
    facilitates both declarative
    configuration and automation.

    View Slide

  35. Tl;dr: Docker as a Service
    https://libreshot.com/container-ship-freighter/

    View Slide

  36. YAML*
    * The "A" is optional, though.

    View Slide

  37. kops - "Kubernetes Operations"

    View Slide

  38. apiVersion: kops/v1alpha2
    kind: InstanceGroup
    metadata:
    labels:
    kops.k8s.io/cluster: kubernetes.chef-internal
    name: nodes
    spec:
    additionalUserData:
    - content: |
    #!/bin/sh - installs, configures, and runs chef
    name: chefbootstrap.sh
    type: text/x-shellscript
    cloudLabels:
    X-Application: kubernetes
    X-Contact: jtimberman
    X-Dept: Operations
    X-Environment: production
    image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08
    machineType: m4.xlarge
    maxSize: 6
    minSize: 4
    nodeLabels:
    kops.k8s.io/instancegroup: nodes
    role: Node
    subnets:
    - us-west-2a
    Kops nodes and masters configuration

    View Slide

  39. #!/bin/bash
    pkg_origin=$1
    pkg_name=$2
    if [ ! -e "/bin/hab" ]; then
    curl https://raw.githubusercontent.com/habitat-sh/habitat/master/components/hab/install.sh |
    bash
    fi
    [[ `grep "^hab:" /etc/passwd` ]] || useradd hab
    [[ `grep "^hab:" /etc/group `]] || groupadd hab
    hab pkg install $pkg_origin/$pkg_name
    pkg_prefix=$(find /hab/pkgs/$pkg_origin/$pkg_name -maxdepth 2 -mindepth 2 | sort | tail -n 1)
    hab pkg exec $pkg_origin/$pkg_name chef-client -z -c $pkg_prefix/config/bootstrap-config.rb
    chefbootstrap.sh - Effortless Config!

    View Slide

  40. k8s-node Chef Infra Policyfile
    • User management (LDAP, SSH Keys)
    • Centralized Logging
    • Send InSpec profile data to Chef Automate
    • Keeps Chef Infra package updated
    • Ensure Chef Infra is running as a service

    View Slide

  41. CI / CD

    View Slide

  42. == CI SaaS...

    View Slide

  43. Expeditor

    View Slide

  44. Expeditor listens to webhooks and takes actions

    View Slide

  45. pipelines:
    - verify:
    description: Pull Request validation tests
    - habitat/build:
    description: Build Habitat package
    - deploy/acceptance:
    description: Deploy changes to Kubernetes
    definition: .expeditor/deploy.pipeline.yml
    env:
    - ENVIRONMENT: acceptance
    - APP: APPNAME
    - deploy/production:
    description: Deploy changes to Kubernetes
    definition: .expeditor/deploy.pipeline.yml
    env:
    - ENVIRONMENT: production
    - APP: APPNAME
    .expeditor/config.yml: pipelines

    View Slide

  46. YAML INTENSIFIES

    View Slide

  47. View Slide

  48. Build Habitat Packages

    View Slide

  49. jtimberman@localhost% hab studio enter
    ...
    [1][default:/src:0]# build
    ...
    learnchef-backend:
    learnchef-backend: I love it when a plan.sh comes
    together.
    learnchef-backend:
    learnchef-backend: Build time: 2m24s
    Build Habitat Packages

    View Slide

  50. View Slide

  51. Publish Docker Images

    View Slide

  52. View Slide

  53. pipelines:
    - verify:
    description: Pull Request validation tests
    - deploy/acceptance:
    description: Deploy changes to kubernetes
    definition: .expeditor/deploy.pipeline.yml
    env:
    - ENVIRONMENT: acceptance
    - APP: APPNAME
    - deploy/production:
    description: Deploy changes to kubernetes
    definition: .expeditor/deploy.pipeline.yml
    env:
    - ENVIRONMENT: production
    - APP: APPNAME
    Buildkite Deploys to Kubernetes

    View Slide

  54. SECRETS??!!
    https://commons.wikimedia.org/wiki/File:WinonaSavingsBankVault.JPG

    View Slide

  55. Secrets Management
    • SSL certificates
    • API tokens
    • SSH keys
    • Configuration files
    • Repositories
    • AWS Accounts
    • Kubernetes

    View Slide

  56. View Slide

  57. K8s Secrets / ConfigMap
    https://commons.wikimedia.org/wiki/File:World_map_.jpg

    View Slide

  58. Application Migration
    https://www.maxpixel.net/Fly-Flock-Of-Birds-Crane-Bird-Migration-2147788

    View Slide

  59. Habitat Package

    View Slide

  60. pkg_name=APPNAME
    pkg_origin=chefops
    pkg_version=0.1.0
    pkg_maintainer="Chef Operations "
    pkg_license=('Apache-2.0')
    pkg_deps=(
    core/coreutils
    )
    pkg_build_deps=(
    core/make
    core/gcc
    core/tar
    )
    pkg_scaffolding=core/scaffolding-ruby
    scaffolding_ruby_pkg=core/ruby24/$(cat "$SRC_PATH/.ruby-version")
    declare -A scaffolding_env
    scaffolding_env[APPNAME_YAML]="/hab/svc/$pkg_name/config/config.yml"
    habitat/plan.sh

    View Slide

  61. Expeditor & Buildkite

    View Slide

  62. Expeditor & Buildkite Configuration
    • Pipeline definitions
    • Expeditor's config.yml
    • Scripts run by pipelines in Buildkite
    • Kubernetes deployment

    View Slide

  63. Kubernetes Configuration

    View Slide

  64. #!/bin/bash
    set -euo pipefail
    if [[ ! -z ${CI+x} ]]; then
    mkdir -p ~/.kube
    aws --profile chef-cd s3 cp s3://chef-cd-citadel/kubernetes.chef.co.config ~/.kube/config
    else
    echo "WARN: Not running in Buildkite, assuming local manual deployment"
    echo "WARN: This requires that ~/.kube/config exists with the proper content"
    fi
    export ENVIRONMENT=${ENVIRONMENT:-dev}
    export APP=${APP}
    DEBUG=${DEBUG:-false}
    # This block translates the "environment" into the appropriate Habitat
    # channel from which to deploy the packages
    if [ "$ENVIRONMENT" == "acceptance" ]; then
    export CHANNEL=acceptance
    elif [ "$ENVIRONMENT" == "production" ]; then
    export CHANNEL=stable
    elif [ "$ENVIRONMENT" == "dev" ] || [ "$ENVIRONMENT" == "test" ]; then
    export CHANNEL=unstable
    else
    echo "We do not currently support deploying to $ENVIRONMENT"
    exit 1
    fi
    # We need the HAB_AUTH_TOKEN set (via Buildkite pipeline) for private packages
    get_image_tag() {
    results=$(curl --silent -H "Authorization: Bearer $HAB_AUTH_TOKEN" https://willem.habitat.sh/v1/depot/channels/chefops/${CHANNEL}/pkgs/${APP}/latest | jq '.ident')
    pkg_version=$(echo "$results" | jq -r .version)
    pkg_release=$(echo "$results" | jq -r .release)
    echo "${pkg_version}-${pkg_release}"
    }
    # Retrieves the ELB's public DNS name
    get_elb_hostname() {
    kubectl get services ${APP}-${ENVIRONMENT} --namespace=${APP} -o json 2>/dev/null | \
    jq '.status.loadBalancer.ingress[].hostname' -r
    }
    # The ELB isn't ready until the hostname is set, so wait until it's ready
    wait_for_elb() {
    attempts=0
    max_attempts=10
    elb_host=""
    while [[ $attempts -lt $max_attempts ]]; do
    elb_host=$(get_elb_hostname || echo)
    if [[ ! -n $elb_host ]]; then
    echo "Did not find ELB yet... sleeping 5s"
    attempts=$[$attempts + 1]
    sleep 5
    else
    echo "Found ELB: $elb_host"
    break
    fi
    done
    }
    # Used for debugging on a local workstation
    if [[ $DEBUG == "true" ]]; then
    echo "--- DEBUG: Environment"
    echo "Application: ${APP}"
    echo "Channel: ${CHANNEL}"
    echo "Environment: ${ENVIRONMENT}"
    echo "Version: $(get_image_tag)"
    fi
    echo "--- Applying kubernetes configuration for ${ENVIRONMENT} to cluster"
    IMAGE_TAG=$(get_image_tag) erb -T- kubernetes/deployment.yml | kubectl apply -f -
    if [[ `grep -c "^kind: Service$" kubernetes/deployment.yml` -gt 0 ]]; then
    echo "+++ Waiting for Load Balancer..."
    wait_for_elb
    fi
    kubernetes.sh

    View Slide

  65. .expeditor/buildkite/kubernetes.sh
    • Deploy pipeline in Buildkite runs the script
    • Copies credentials used for kubectl
    • Retrieves the Habitat package information from
    Builder
    • Applies the kubernetes/deployment.yml
    manifest
    • Waits until the ELB is available (if there is one)
    • Success!

    View Slide

  66. Habitat package information
    • Version
    • Release
    • Channel(s)
    • `get_image_tag`

    View Slide

  67. export IMAGE_TAG=$(get_image_tag)
    erb -T- kubernetes/deployment.yml | kubectl apply -f -
    kubernetes.sh

    View Slide

  68. Kubernetes Resources
    • Pods
    • StatefulSets & Deployments
    • Services
    • Namespaces
    • Custom Resource Definitions (CRDs)
    • (e.g., Habitat operator, Prometheus operator)

    View Slide

  69. ---
    apiVersion: v1
    kind: Namespace
    metadata:
    name: nginx-demo
    ---
    apiVersion: habitat.sh/v1beta1
    kind: Habitat
    metadata:
    name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    namespace: nginx-demo
    labels:
    app: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    customVersion: v1beta2
    spec:
    v1beta2:
    image: chefops/nginx-demo:<%= ENV['IMAGE_TAG'] %>
    count: 2
    service:
    name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    topology: standalone
    ---
    apiVersion: v1
    kind: Service
    metadata:
    name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    namespace: nginx-demo
    spec:
    selector:
    habitat-name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    ports:
    - name: http
    port: 80
    targetPort: 80
    protocol: TCP
    type: LoadBalancer
    Kubernetes: A Primer via Configuration

    View Slide

  70. ---
    apiVersion: v1
    kind: Namespace
    metadata:
    name: nginx-demo
    kubernetes/deployment.yml - Namespace
    • Namespaces isolate resources in "virtual clusters"
    • Each application gets a namespace

    View Slide

  71. apiVersion: habitat.sh/v1beta1
    kind: Habitat
    metadata:
    name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    namespace: nginx-demo
    labels:
    app: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    customVersion: v1beta2
    spec:
    v1beta2:
    image: chefops/nginx-demo:<%= ENV['IMAGE_TAG'] %>
    count: 2
    service:
    name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    topology: standalone
    kubernetes/deployment.yml - Habitat
    • Applications are deployed using the Habitat Operator
    • This creates a StatefulSet with two pods
    • The pods will use the Docker image specified

    View Slide

  72. apiVersion: v1
    kind: Service
    metadata:
    name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    namespace: nginx-demo
    spec:
    selector:
    habitat-name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    ports:
    - name: http
    port: 80
    targetPort: 80
    protocol: TCP
    type: LoadBalancer
    kubernetes/deployment.yml - Service

    View Slide

  73. % kubectl get pods -n omnitruck
    NAME READY STATUS RESTARTS AGE
    omnitruck-acceptance-0 1/1 Running 0 5d9h
    omnitruck-acceptance-1 1/1 Running 0 5d9h
    omnitruck-poller-acceptance-1558471800-hvh45 0/1 Completed 0 31m
    omnitruck-poller-acceptance-1558472400-d4pps 0/1 Completed 0 21m
    omnitruck-poller-acceptance-1558473000-dsmcn 0/1 Completed 0 11m
    omnitruck-poller-acceptance-1558473600-b72lg 1/1 Running 0 112s
    omnitruck-poller-production-1558471800-rbt7n 0/1 Completed 0 31m
    omnitruck-poller-production-1558472400-92qkv 0/1 Completed 0 21m
    omnitruck-poller-production-1558473000-fb6lh 0/1 Completed 0 11m
    omnitruck-poller-production-1558473600-lw868 1/1 Running 0 111s
    omnitruck-production-0 1/1 Running 0 5d9h
    omnitruck-production-1 1/1 Running 0 5d9h
    omnitruck-production-2 1/1 Running 0 5d9h
    pods

    View Slide

  74. % kubectl get services -n omnitruck
    NAME TYPE CLUSTER-IP
    EXTERNAL-IP PORT(S) AGE
    omnitruck-acceptance LoadBalancer 100.69.37.56
    a260b98a3fcab... 443:32433/TCP 162d
    omnitruck-production LoadBalancer 100.70.244.59
    a92a69a04fcc2... 443:31683/TCP 161d
    services

    View Slide

  75. Stateless Services Only

    View Slide

  76. Monitoring

    View Slide

  77. Monitoring Observability

    View Slide

  78. Prometheus + Grafana

    View Slide

  79. https://github.com/coreos/prometheus-operator

    View Slide

  80. View Slide

  81. % uptime

    View Slide

  82. Workflow Steps

    View Slide

  83. No Live Demo Today!
    https://flic.kr/p/RJ5xEs

    View Slide

  84. Procedure
    1.Make changes to the Repository, open a pull request
    2.Wait for Buildkite verification, peer approval, then
    merge
    3.Wait for Expeditor to complete merge actions
    4.Verify acceptance with stakeholders
    5.Promote to Production

    View Slide

  85. Previous Procedure
    • Let's just say...
    • It was more than 5 steps
    • And many of them were manual
    • And the procedure differed slightly between
    applications...

    View Slide

  86. Make changes to the repository, open a pull
    request
    git checkout -b $USERNAME/my-branch
    # make some changes with your favorite editor
    git add .
    git commit -m 'updating the site with new features or fixes'
    git push origin $USERNAME/my-branch

    View Slide

  87. Wait for Expeditor to complete merge
    actions

    View Slide

  88. Verify Acceptance with stakeholders

    View Slide

  89. Promote to Production

    View Slide

  90. Coworker: "Wait... that's all?"

    View Slide

  91. *Spoiler: That's not all...
    https://commons.wikimedia.org/wiki/File:Bugatti_veyron2.jpg

    View Slide

  92. Conclusion

    View Slide

  93. Automation has shifted

    View Slide

  94. Workflow > Infrastructure

    View Slide

  95. Business benefit

    View Slide

  96. Questions?

    View Slide

  97. Thank you!
    Joshua Timberman
    @jtimberman
    All images used in this presentation are public domain, created by me,
    or licensed under the Creative Commons Attribution-Share Alike 4.0 license:
    https://creativecommons.org/licenses/by-sa/4.0/deed.en
    The only modification made to any images is resizing to fit into the slides

    View Slide