Running_Legacy_Apps_on_K8s.pdf

Transcript

1. Legacy Applications on Kubernetes Joshua Timberman <joshua@chef.io> @jtimberman
2. https://commons.wikimedia.org/wiki/File:BalticServers_data_center.jpg % whoami
3. Why are we here?
4. Running Legacy Applications on Kubernetes
5. Running Legacy Heritage Applications on Kubernetes
6. What is a Legacy Application? In computing, a legacy application program of, relating to, or being a previous or outdated computer system, yet still in use. Often referencing a system as "legacy" means that it paved the way for the standards that would follow it. This can also imply that the system is out of date or in need of replacement.
7. Chef Ops runs many applications
8. Applications, sample set • Hosted Chef (api.chef.io) • Downloads (downloads.chef.io) • Omnitruck (omnitruck.chef.io) • Learn Chef (learn.chef.io) • Chef Community Cookbooks (supermarket.chef.io) • Internal and External utilities • "Gateway" and "glue" services • Kubernetes itself ;) • etc...
9. Workflow Matters
10. Why Kubernetes?
11. Cloud Native
12. Platform Consistency
13. Cohesion
14. Cost-savings https://en.wikipedia.org/wiki/Denomination_effect#/media/File:Exchange_Money_Conversion_to_Foreign_Currency.jpg
15. https://commons.wikimedia.org/wiki/File:Frozen_peas.JPG
16. https://commons.wikimedia.org/wiki/File:Mashed_potatoes_(1).jpg
17. What about Chef?
18. As it turns out... https://www.flickr.com/photos/tofu_mugwump/22420266811 https://creativecommons.org/licenses/by/2.0/
19. We still have an OS.
20. But we use K8SaaS! https://www.jpl.nasa.gov/news/news.php?feature=7113
22. Policyfiles https://en.wikipedia.org/wiki/Ledger#/media/File:Ledger.png
24. Test Kitchen (for cookbooks) https://commons.wikimedia.org/wiki/File:Conference_House_-_Rustic_Kitchen.jpg
25. DevSec Linux Baseline https://dev-sec.io/baselines/linux/
26. CIS AWS Benchmark https://commons.wikimedia.org/wiki/File:Benchmark_at_former_Abbey_Cinema.jpg
28. Ruby Scaffolding https://commons.wikimedia.org/wiki/File:Seattle_- _construction_at_SE_corner_of_11th_%26_E_Pine_01.jpg
29. core-plans/scaffolding-base core-plans/scaffolding-chef core-plans/scaffolding-go core-plans/scaffolding-go17 core-plans/scaffolding-gradle core-plans/scaffolding-inspec core-plans/scaffolding-node core-plans/scaffolding-ruby scaffolding plans
30. Effortless Config
31. How did we get here?
33. https://flic.kr/p/bFe9Bs
34. Kubernetes is a portable, extensible open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation.
35. Tl;dr: Docker as a Service https://libreshot.com/container-ship-freighter/
36. YAML* * The "A" is optional, though.
37. kops - "Kubernetes Operations"
38. apiVersion: kops/v1alpha2 kind: InstanceGroup metadata: labels: kops.k8s.io/cluster: kubernetes.chef-internal name: nodes spec: additionalUserData: - content: | #!/bin/sh - installs, configures, and runs chef name: chefbootstrap.sh type: text/x-shellscript cloudLabels: X-Application: kubernetes X-Contact: jtimberman X-Dept: Operations X-Environment: production image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08 machineType: m4.xlarge maxSize: 6 minSize: 4 nodeLabels: kops.k8s.io/instancegroup: nodes role: Node subnets: - us-west-2a Kops nodes and masters configuration
39. #!/bin/bash pkg_origin=$1 pkg_name=$2 if [ ! -e "/bin/hab" ]; then curl https://raw.githubusercontent.com/habitat-sh/habitat/master/components/hab/install.sh | bash fi [[ `grep "^hab:" /etc/passwd` ]] || useradd hab [[ `grep "^hab:" /etc/group `]] || groupadd hab hab pkg install $pkg_origin/$pkg_name pkg_prefix=$(find /hab/pkgs/$pkg_origin/$pkg_name -maxdepth 2 -mindepth 2 | sort | tail -n 1) hab pkg exec $pkg_origin/$pkg_name chef-client -z -c $pkg_prefix/config/bootstrap-config.rb chefbootstrap.sh - Effortless Config!
40. k8s-node Chef Infra Policyfile • User management (LDAP, SSH Keys) • Centralized Logging • Send InSpec profile data to Chef Automate • Keeps Chef Infra package updated • Ensure Chef Infra is running as a service
41. CI / CD
42. == CI SaaS...
43. Expeditor
44. Expeditor listens to webhooks and takes actions
45. pipelines: - verify: description: Pull Request validation tests - habitat/build: description: Build Habitat package - deploy/acceptance: description: Deploy changes to Kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: acceptance - APP: APPNAME - deploy/production: description: Deploy changes to Kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: production - APP: APPNAME .expeditor/config.yml: pipelines
46. YAML INTENSIFIES
48. Build Habitat Packages
49. jtimberman@localhost% hab studio enter ... <output truncated> [1][default:/src:0]# build ... <output truncated> learnchef-backend: learnchef-backend: I love it when a plan.sh comes together. learnchef-backend: learnchef-backend: Build time: 2m24s Build Habitat Packages
51. Publish Docker Images
53. pipelines: - verify: description: Pull Request validation tests - deploy/acceptance: description: Deploy changes to kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: acceptance - APP: APPNAME - deploy/production: description: Deploy changes to kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: production - APP: APPNAME Buildkite Deploys to Kubernetes
54. SECRETS??!! https://commons.wikimedia.org/wiki/File:WinonaSavingsBankVault.JPG
55. Secrets Management • SSL certificates • API tokens • SSH keys • Configuration files • Repositories • AWS Accounts • Kubernetes
57. K8s Secrets / ConfigMap https://commons.wikimedia.org/wiki/File:World_map_.jpg
58. Application Migration https://www.maxpixel.net/Fly-Flock-Of-Birds-Crane-Bird-Migration-2147788
59. Habitat Package
60. pkg_name=APPNAME pkg_origin=chefops pkg_version=0.1.0 pkg_maintainer="Chef Operations <ops@chef.io>" pkg_license=('Apache-2.0') pkg_deps=( core/coreutils ) pkg_build_deps=( core/make core/gcc core/tar ) pkg_scaffolding=core/scaffolding-ruby scaffolding_ruby_pkg=core/ruby24/$(cat "$SRC_PATH/.ruby-version") declare -A scaffolding_env scaffolding_env[APPNAME_YAML]="/hab/svc/$pkg_name/config/config.yml" habitat/plan.sh
61. Expeditor & Buildkite
62. Expeditor & Buildkite Configuration • Pipeline definitions • Expeditor's config.yml • Scripts run by pipelines in Buildkite • Kubernetes deployment
63. Kubernetes Configuration
64. #!/bin/bash set -euo pipefail if [[ ! -z ${CI+x} ]]; then mkdir -p ~/.kube aws --profile chef-cd s3 cp s3://chef-cd-citadel/kubernetes.chef.co.config ~/.kube/config else echo "WARN: Not running in Buildkite, assuming local manual deployment" echo "WARN: This requires that ~/.kube/config exists with the proper content" fi export ENVIRONMENT=${ENVIRONMENT:-dev} export APP=${APP} DEBUG=${DEBUG:-false} # This block translates the "environment" into the appropriate Habitat # channel from which to deploy the packages if [ "$ENVIRONMENT" == "acceptance" ]; then export CHANNEL=acceptance elif [ "$ENVIRONMENT" == "production" ]; then export CHANNEL=stable elif [ "$ENVIRONMENT" == "dev" ] || [ "$ENVIRONMENT" == "test" ]; then export CHANNEL=unstable else echo "We do not currently support deploying to $ENVIRONMENT" exit 1 fi # We need the HAB_AUTH_TOKEN set (via Buildkite pipeline) for private packages get_image_tag() { results=$(curl --silent -H "Authorization: Bearer $HAB_AUTH_TOKEN" https://willem.habitat.sh/v1/depot/channels/chefops/${CHANNEL}/pkgs/${APP}/latest | jq '.ident') pkg_version=$(echo "$results" | jq -r .version) pkg_release=$(echo "$results" | jq -r .release) echo "${pkg_version}-${pkg_release}" } # Retrieves the ELB's public DNS name get_elb_hostname() { kubectl get services ${APP}-${ENVIRONMENT} --namespace=${APP} -o json 2>/dev/null | \ jq '.status.loadBalancer.ingress[].hostname' -r } # The ELB isn't ready until the hostname is set, so wait until it's ready wait_for_elb() { attempts=0 max_attempts=10 elb_host="" while [[ $attempts -lt $max_attempts ]]; do elb_host=$(get_elb_hostname || echo) if [[ ! -n $elb_host ]]; then echo "Did not find ELB yet... sleeping 5s" attempts=$[$attempts + 1] sleep 5 else echo "Found ELB: $elb_host" break fi done } # Used for debugging on a local workstation if [[ $DEBUG == "true" ]]; then echo "--- DEBUG: Environment" echo "Application: ${APP}" echo "Channel: ${CHANNEL}" echo "Environment: ${ENVIRONMENT}" echo "Version: $(get_image_tag)" fi echo "--- Applying kubernetes configuration for ${ENVIRONMENT} to cluster" IMAGE_TAG=$(get_image_tag) erb -T- kubernetes/deployment.yml | kubectl apply -f - if [[ `grep -c "^kind: Service$" kubernetes/deployment.yml` -gt 0 ]]; then echo "+++ Waiting for Load Balancer..." wait_for_elb fi kubernetes.sh
65. .expeditor/buildkite/kubernetes.sh • Deploy pipeline in Buildkite runs the script • Copies credentials used for kubectl • Retrieves the Habitat package information from Builder • Applies the kubernetes/deployment.yml manifest • Waits until the ELB is available (if there is one) • Success!
66. Habitat package information • Version • Release • Channel(s) • `get_image_tag`
67. export IMAGE_TAG=$(get_image_tag) erb -T- kubernetes/deployment.yml | kubectl apply -f - kubernetes.sh
68. Kubernetes Resources • Pods • StatefulSets & Deployments • Services • Namespaces • Custom Resource Definitions (CRDs) • (e.g., Habitat operator, Prometheus operator)
69. --- apiVersion: v1 kind: Namespace metadata: name: nginx-demo --- apiVersion: habitat.sh/v1beta1 kind: Habitat metadata: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> namespace: nginx-demo labels: app: nginx-demo-<%= ENV['ENVIRONMENT'] %> customVersion: v1beta2 spec: v1beta2: image: chefops/nginx-demo:<%= ENV['IMAGE_TAG'] %> count: 2 service: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> topology: standalone --- apiVersion: v1 kind: Service metadata: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> namespace: nginx-demo spec: selector: habitat-name: nginx-demo-<%= ENV['ENVIRONMENT'] %> ports: - name: http port: 80 targetPort: 80 protocol: TCP type: LoadBalancer Kubernetes: A Primer via Configuration
70. --- apiVersion: v1 kind: Namespace metadata: name: nginx-demo kubernetes/deployment.yml - Namespace • Namespaces isolate resources in "virtual clusters" • Each application gets a namespace
71. apiVersion: habitat.sh/v1beta1 kind: Habitat metadata: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> namespace: nginx-demo labels: app: nginx-demo-<%= ENV['ENVIRONMENT'] %> customVersion: v1beta2 spec: v1beta2: image: chefops/nginx-demo:<%= ENV['IMAGE_TAG'] %> count: 2 service: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> topology: standalone kubernetes/deployment.yml - Habitat • Applications are deployed using the Habitat Operator • This creates a StatefulSet with two pods • The pods will use the Docker image specified
72. apiVersion: v1 kind: Service metadata: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> namespace: nginx-demo spec: selector: habitat-name: nginx-demo-<%= ENV['ENVIRONMENT'] %> ports: - name: http port: 80 targetPort: 80 protocol: TCP type: LoadBalancer kubernetes/deployment.yml - Service
73. % kubectl get pods -n omnitruck NAME READY STATUS RESTARTS AGE omnitruck-acceptance-0 1/1 Running 0 5d9h omnitruck-acceptance-1 1/1 Running 0 5d9h omnitruck-poller-acceptance-1558471800-hvh45 0/1 Completed 0 31m omnitruck-poller-acceptance-1558472400-d4pps 0/1 Completed 0 21m omnitruck-poller-acceptance-1558473000-dsmcn 0/1 Completed 0 11m omnitruck-poller-acceptance-1558473600-b72lg 1/1 Running 0 112s omnitruck-poller-production-1558471800-rbt7n 0/1 Completed 0 31m omnitruck-poller-production-1558472400-92qkv 0/1 Completed 0 21m omnitruck-poller-production-1558473000-fb6lh 0/1 Completed 0 11m omnitruck-poller-production-1558473600-lw868 1/1 Running 0 111s omnitruck-production-0 1/1 Running 0 5d9h omnitruck-production-1 1/1 Running 0 5d9h omnitruck-production-2 1/1 Running 0 5d9h pods
74. % kubectl get services -n omnitruck NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE omnitruck-acceptance LoadBalancer 100.69.37.56 a260b98a3fcab... 443:32433/TCP 162d omnitruck-production LoadBalancer 100.70.244.59 a92a69a04fcc2... 443:31683/TCP 161d services
75. Stateless Services Only
76. Monitoring
77. Monitoring Observability
78. Prometheus + Grafana
79. https://github.com/coreos/prometheus-operator
81. % uptime
82. Workflow Steps
83. No Live Demo Today! https://flic.kr/p/RJ5xEs
84. Procedure 1.Make changes to the Repository, open a pull request 2.Wait for Buildkite verification, peer approval, then merge 3.Wait for Expeditor to complete merge actions 4.Verify acceptance with stakeholders 5.Promote to Production
85. Previous Procedure • Let's just say... • It was more than 5 steps • And many of them were manual • And the procedure differed slightly between applications...
86. Make changes to the repository, open a pull request git checkout -b $USERNAME/my-branch # make some changes with your favorite editor git add . git commit -m 'updating the site with new features or fixes' git push origin $USERNAME/my-branch
87. Wait for Expeditor to complete merge actions
88. Verify Acceptance with stakeholders
89. Promote to Production
90. Coworker: "Wait... that's all?"
91. *Spoiler: That's not all... https://commons.wikimedia.org/wiki/File:Bugatti_veyron2.jpg
92. Conclusion
93. Automation has shifted
94. Workflow > Infrastructure
95. Business benefit
96. Questions?
97. Thank you! Joshua Timberman <joshua@chef.io> @jtimberman All images used in this presentation are public domain, created by me, or licensed under the Creative Commons Attribution-Share Alike 4.0 license: https://creativecommons.org/licenses/by-sa/4.0/deed.en The only modification made to any images is resizing to fit into the slides