Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Running_Legacy_Apps_on_K8s.pdf

4eb6098fc8de5a5f37199c3668b11590?s=47 Joshua Timberman
May 21, 2019
2k

 Running_Legacy_Apps_on_K8s.pdf

4eb6098fc8de5a5f37199c3668b11590?s=128

Joshua Timberman

May 21, 2019
Tweet

Transcript

  1. Legacy Applications on Kubernetes Joshua Timberman <joshua@chef.io> @jtimberman

  2. https://commons.wikimedia.org/wiki/File:BalticServers_data_center.jpg % whoami

  3. Why are we here?

  4. Running Legacy Applications on Kubernetes

  5. Running Legacy Heritage Applications on Kubernetes

  6. What is a Legacy Application? In computing, a legacy application

    program of, relating to, or being a previous or outdated computer system, yet still in use. Often referencing a system as "legacy" means that it paved the way for the standards that would follow it. This can also imply that the system is out of date or in need of replacement.
  7. Chef Ops runs many applications

  8. Applications, sample set • Hosted Chef (api.chef.io) • Downloads (downloads.chef.io)

    • Omnitruck (omnitruck.chef.io) • Learn Chef (learn.chef.io) • Chef Community Cookbooks (supermarket.chef.io) • Internal and External utilities • "Gateway" and "glue" services • Kubernetes itself ;) • etc...
  9. Workflow Matters

  10. Why Kubernetes?

  11. Cloud Native

  12. Platform Consistency

  13. Cohesion

  14. Cost-savings https://en.wikipedia.org/wiki/Denomination_effect#/media/File:Exchange_Money_Conversion_to_Foreign_Currency.jpg

  15. https://commons.wikimedia.org/wiki/File:Frozen_peas.JPG

  16. https://commons.wikimedia.org/wiki/File:Mashed_potatoes_(1).jpg

  17. What about Chef?

  18. As it turns out... https://www.flickr.com/photos/tofu_mugwump/22420266811 https://creativecommons.org/licenses/by/2.0/

  19. We still have an OS.

  20. But we use K8SaaS! https://www.jpl.nasa.gov/news/news.php?feature=7113

  21. None
  22. Policyfiles https://en.wikipedia.org/wiki/Ledger#/media/File:Ledger.png

  23. None
  24. Test Kitchen (for cookbooks) https://commons.wikimedia.org/wiki/File:Conference_House_-_Rustic_Kitchen.jpg

  25. DevSec Linux Baseline https://dev-sec.io/baselines/linux/

  26. CIS AWS Benchmark https://commons.wikimedia.org/wiki/File:Benchmark_at_former_Abbey_Cinema.jpg

  27. None
  28. Ruby Scaffolding https://commons.wikimedia.org/wiki/File:Seattle_- _construction_at_SE_corner_of_11th_%26_E_Pine_01.jpg

  29. core-plans/scaffolding-base core-plans/scaffolding-chef core-plans/scaffolding-go core-plans/scaffolding-go17 core-plans/scaffolding-gradle core-plans/scaffolding-inspec core-plans/scaffolding-node core-plans/scaffolding-ruby scaffolding plans

  30. Effortless Config

  31. How did we get here?

  32. None
  33. https://flic.kr/p/bFe9Bs

  34. Kubernetes is a portable, extensible open-source platform for managing containerized

    workloads and services, that facilitates both declarative configuration and automation.
  35. Tl;dr: Docker as a Service https://libreshot.com/container-ship-freighter/

  36. YAML* * The "A" is optional, though.

  37. kops - "Kubernetes Operations"

  38. apiVersion: kops/v1alpha2 kind: InstanceGroup metadata: labels: kops.k8s.io/cluster: kubernetes.chef-internal name: nodes

    spec: additionalUserData: - content: | #!/bin/sh - installs, configures, and runs chef name: chefbootstrap.sh type: text/x-shellscript cloudLabels: X-Application: kubernetes X-Contact: jtimberman X-Dept: Operations X-Environment: production image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08 machineType: m4.xlarge maxSize: 6 minSize: 4 nodeLabels: kops.k8s.io/instancegroup: nodes role: Node subnets: - us-west-2a Kops nodes and masters configuration
  39. #!/bin/bash pkg_origin=$1 pkg_name=$2 if [ ! -e "/bin/hab" ]; then

    curl https://raw.githubusercontent.com/habitat-sh/habitat/master/components/hab/install.sh | bash fi [[ `grep "^hab:" /etc/passwd` ]] || useradd hab [[ `grep "^hab:" /etc/group `]] || groupadd hab hab pkg install $pkg_origin/$pkg_name pkg_prefix=$(find /hab/pkgs/$pkg_origin/$pkg_name -maxdepth 2 -mindepth 2 | sort | tail -n 1) hab pkg exec $pkg_origin/$pkg_name chef-client -z -c $pkg_prefix/config/bootstrap-config.rb chefbootstrap.sh - Effortless Config!
  40. k8s-node Chef Infra Policyfile • User management (LDAP, SSH Keys)

    • Centralized Logging • Send InSpec profile data to Chef Automate • Keeps Chef Infra package updated • Ensure Chef Infra is running as a service
  41. CI / CD

  42. == CI SaaS...

  43. Expeditor

  44. Expeditor listens to webhooks and takes actions

  45. pipelines: - verify: description: Pull Request validation tests - habitat/build:

    description: Build Habitat package - deploy/acceptance: description: Deploy changes to Kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: acceptance - APP: APPNAME - deploy/production: description: Deploy changes to Kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: production - APP: APPNAME .expeditor/config.yml: pipelines
  46. YAML INTENSIFIES

  47. None
  48. Build Habitat Packages

  49. jtimberman@localhost% hab studio enter ... <output truncated> [1][default:/src:0]# build ...

    <output truncated> learnchef-backend: learnchef-backend: I love it when a plan.sh comes together. learnchef-backend: learnchef-backend: Build time: 2m24s Build Habitat Packages
  50. None
  51. Publish Docker Images

  52. None
  53. pipelines: - verify: description: Pull Request validation tests - deploy/acceptance:

    description: Deploy changes to kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: acceptance - APP: APPNAME - deploy/production: description: Deploy changes to kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: production - APP: APPNAME Buildkite Deploys to Kubernetes
  54. SECRETS??!! https://commons.wikimedia.org/wiki/File:WinonaSavingsBankVault.JPG

  55. Secrets Management • SSL certificates • API tokens • SSH

    keys • Configuration files • Repositories • AWS Accounts • Kubernetes
  56. None
  57. K8s Secrets / ConfigMap https://commons.wikimedia.org/wiki/File:World_map_.jpg

  58. Application Migration https://www.maxpixel.net/Fly-Flock-Of-Birds-Crane-Bird-Migration-2147788

  59. Habitat Package

  60. pkg_name=APPNAME pkg_origin=chefops pkg_version=0.1.0 pkg_maintainer="Chef Operations <ops@chef.io>" pkg_license=('Apache-2.0') pkg_deps=( core/coreutils )

    pkg_build_deps=( core/make core/gcc core/tar ) pkg_scaffolding=core/scaffolding-ruby scaffolding_ruby_pkg=core/ruby24/$(cat "$SRC_PATH/.ruby-version") declare -A scaffolding_env scaffolding_env[APPNAME_YAML]="/hab/svc/$pkg_name/config/config.yml" habitat/plan.sh
  61. Expeditor & Buildkite

  62. Expeditor & Buildkite Configuration • Pipeline definitions • Expeditor's config.yml

    • Scripts run by pipelines in Buildkite • Kubernetes deployment
  63. Kubernetes Configuration

  64. #!/bin/bash set -euo pipefail if [[ ! -z ${CI+x} ]];

    then mkdir -p ~/.kube aws --profile chef-cd s3 cp s3://chef-cd-citadel/kubernetes.chef.co.config ~/.kube/config else echo "WARN: Not running in Buildkite, assuming local manual deployment" echo "WARN: This requires that ~/.kube/config exists with the proper content" fi export ENVIRONMENT=${ENVIRONMENT:-dev} export APP=${APP} DEBUG=${DEBUG:-false} # This block translates the "environment" into the appropriate Habitat # channel from which to deploy the packages if [ "$ENVIRONMENT" == "acceptance" ]; then export CHANNEL=acceptance elif [ "$ENVIRONMENT" == "production" ]; then export CHANNEL=stable elif [ "$ENVIRONMENT" == "dev" ] || [ "$ENVIRONMENT" == "test" ]; then export CHANNEL=unstable else echo "We do not currently support deploying to $ENVIRONMENT" exit 1 fi # We need the HAB_AUTH_TOKEN set (via Buildkite pipeline) for private packages get_image_tag() { results=$(curl --silent -H "Authorization: Bearer $HAB_AUTH_TOKEN" https://willem.habitat.sh/v1/depot/channels/chefops/${CHANNEL}/pkgs/${APP}/latest | jq '.ident') pkg_version=$(echo "$results" | jq -r .version) pkg_release=$(echo "$results" | jq -r .release) echo "${pkg_version}-${pkg_release}" } # Retrieves the ELB's public DNS name get_elb_hostname() { kubectl get services ${APP}-${ENVIRONMENT} --namespace=${APP} -o json 2>/dev/null | \ jq '.status.loadBalancer.ingress[].hostname' -r } # The ELB isn't ready until the hostname is set, so wait until it's ready wait_for_elb() { attempts=0 max_attempts=10 elb_host="" while [[ $attempts -lt $max_attempts ]]; do elb_host=$(get_elb_hostname || echo) if [[ ! -n $elb_host ]]; then echo "Did not find ELB yet... sleeping 5s" attempts=$[$attempts + 1] sleep 5 else echo "Found ELB: $elb_host" break fi done } # Used for debugging on a local workstation if [[ $DEBUG == "true" ]]; then echo "--- DEBUG: Environment" echo "Application: ${APP}" echo "Channel: ${CHANNEL}" echo "Environment: ${ENVIRONMENT}" echo "Version: $(get_image_tag)" fi echo "--- Applying kubernetes configuration for ${ENVIRONMENT} to cluster" IMAGE_TAG=$(get_image_tag) erb -T- kubernetes/deployment.yml | kubectl apply -f - if [[ `grep -c "^kind: Service$" kubernetes/deployment.yml` -gt 0 ]]; then echo "+++ Waiting for Load Balancer..." wait_for_elb fi kubernetes.sh
  65. .expeditor/buildkite/kubernetes.sh • Deploy pipeline in Buildkite runs the script •

    Copies credentials used for kubectl • Retrieves the Habitat package information from Builder • Applies the kubernetes/deployment.yml manifest • Waits until the ELB is available (if there is one) • Success!
  66. Habitat package information • Version • Release • Channel(s) •

    `get_image_tag`
  67. export IMAGE_TAG=$(get_image_tag) erb -T- kubernetes/deployment.yml | kubectl apply -f -

    kubernetes.sh
  68. Kubernetes Resources • Pods • StatefulSets & Deployments • Services

    • Namespaces • Custom Resource Definitions (CRDs) • (e.g., Habitat operator, Prometheus operator)
  69. --- apiVersion: v1 kind: Namespace metadata: name: nginx-demo --- apiVersion:

    habitat.sh/v1beta1 kind: Habitat metadata: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> namespace: nginx-demo labels: app: nginx-demo-<%= ENV['ENVIRONMENT'] %> customVersion: v1beta2 spec: v1beta2: image: chefops/nginx-demo:<%= ENV['IMAGE_TAG'] %> count: 2 service: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> topology: standalone --- apiVersion: v1 kind: Service metadata: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> namespace: nginx-demo spec: selector: habitat-name: nginx-demo-<%= ENV['ENVIRONMENT'] %> ports: - name: http port: 80 targetPort: 80 protocol: TCP type: LoadBalancer Kubernetes: A Primer via Configuration
  70. --- apiVersion: v1 kind: Namespace metadata: name: nginx-demo kubernetes/deployment.yml -

    Namespace • Namespaces isolate resources in "virtual clusters" • Each application gets a namespace
  71. apiVersion: habitat.sh/v1beta1 kind: Habitat metadata: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> namespace:

    nginx-demo labels: app: nginx-demo-<%= ENV['ENVIRONMENT'] %> customVersion: v1beta2 spec: v1beta2: image: chefops/nginx-demo:<%= ENV['IMAGE_TAG'] %> count: 2 service: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> topology: standalone kubernetes/deployment.yml - Habitat • Applications are deployed using the Habitat Operator • This creates a StatefulSet with two pods • The pods will use the Docker image specified
  72. apiVersion: v1 kind: Service metadata: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> namespace:

    nginx-demo spec: selector: habitat-name: nginx-demo-<%= ENV['ENVIRONMENT'] %> ports: - name: http port: 80 targetPort: 80 protocol: TCP type: LoadBalancer kubernetes/deployment.yml - Service
  73. % kubectl get pods -n omnitruck NAME READY STATUS RESTARTS

    AGE omnitruck-acceptance-0 1/1 Running 0 5d9h omnitruck-acceptance-1 1/1 Running 0 5d9h omnitruck-poller-acceptance-1558471800-hvh45 0/1 Completed 0 31m omnitruck-poller-acceptance-1558472400-d4pps 0/1 Completed 0 21m omnitruck-poller-acceptance-1558473000-dsmcn 0/1 Completed 0 11m omnitruck-poller-acceptance-1558473600-b72lg 1/1 Running 0 112s omnitruck-poller-production-1558471800-rbt7n 0/1 Completed 0 31m omnitruck-poller-production-1558472400-92qkv 0/1 Completed 0 21m omnitruck-poller-production-1558473000-fb6lh 0/1 Completed 0 11m omnitruck-poller-production-1558473600-lw868 1/1 Running 0 111s omnitruck-production-0 1/1 Running 0 5d9h omnitruck-production-1 1/1 Running 0 5d9h omnitruck-production-2 1/1 Running 0 5d9h pods
  74. % kubectl get services -n omnitruck NAME TYPE CLUSTER-IP EXTERNAL-IP

    PORT(S) AGE omnitruck-acceptance LoadBalancer 100.69.37.56 a260b98a3fcab... 443:32433/TCP 162d omnitruck-production LoadBalancer 100.70.244.59 a92a69a04fcc2... 443:31683/TCP 161d services
  75. Stateless Services Only

  76. Monitoring

  77. Monitoring Observability

  78. Prometheus + Grafana

  79. https://github.com/coreos/prometheus-operator

  80. None
  81. % uptime

  82. Workflow Steps

  83. No Live Demo Today! https://flic.kr/p/RJ5xEs

  84. Procedure 1.Make changes to the Repository, open a pull request

    2.Wait for Buildkite verification, peer approval, then merge 3.Wait for Expeditor to complete merge actions 4.Verify acceptance with stakeholders 5.Promote to Production
  85. Previous Procedure • Let's just say... • It was more

    than 5 steps • And many of them were manual • And the procedure differed slightly between applications...
  86. Make changes to the repository, open a pull request git

    checkout -b $USERNAME/my-branch # make some changes with your favorite editor git add . git commit -m 'updating the site with new features or fixes' git push origin $USERNAME/my-branch
  87. Wait for Expeditor to complete merge actions

  88. Verify Acceptance with stakeholders

  89. Promote to Production

  90. Coworker: "Wait... that's all?"

  91. *Spoiler: That's not all... https://commons.wikimedia.org/wiki/File:Bugatti_veyron2.jpg

  92. Conclusion

  93. Automation has shifted

  94. Workflow > Infrastructure

  95. Business benefit

  96. Questions?

  97. Thank you! Joshua Timberman <joshua@chef.io> @jtimberman All images used in

    this presentation are public domain, created by me, or licensed under the Creative Commons Attribution-Share Alike 4.0 license: https://creativecommons.org/licenses/by-sa/4.0/deed.en The only modification made to any images is resizing to fit into the slides