Slide 1

Slide 1 text

Chef's Operations Workflow Joshua Timberman @jtimberman

Slide 2

Slide 2 text

https://commons.wikimedia.org/wiki/File:BalticServers_data_center.jpg % whoami

Slide 3

Slide 3 text

Why are we here? https://www.flickr.com/photos/10785432@N03/6570725517

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

Cloud Native

Slide 6

Slide 6 text

Many applications

Slide 7

Slide 7 text

Applications, sample set • Hosted Chef (api.chef.io) • Downloads (downloads.chef.io) • Omnitruck (omnitruck.chef.io) • Learn Chef (learn.chef.io) • Chef Community Cookbooks (supermarket.chef.io) • Internal and External utilities • "Gateway" and "glue" services • Kubernetes ;) • etc...

Slide 8

Slide 8 text

Platform Consistency

Slide 9

Slide 9 text

Cohesion

Slide 10

Slide 10 text

Industry awareness

Slide 11

Slide 11 text

Components

Slide 12

Slide 12 text

No Live Demo Today! https://flic.kr/p/RJ5xEs

Slide 13

Slide 13 text

Code Repository

Slide 14

Slide 14 text

GitHub

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

CI / CD

Slide 20

Slide 20 text

Buildkite + Chat Bot

Slide 21

Slide 21 text

== CI SaaS...

Slide 22

Slide 22 text

Expeditor, our chatbot ==

Slide 23

Slide 23 text

YAML

Slide 24

Slide 24 text

slack: notify_channel: ops-notify github: delete_branch_on_merge: true # These are our Buildkite pipelines where deploys take place pipelines: - verify: description: Pull Request validation tests - deploy/acceptance: description: Deploy changes to kubernetes.chef.co pods definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: acceptance - APP: chef-web-util - deploy/production: description: Deploy changes to kubernetes.chef.co pods definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: production - APP: chef-web-util # These actions are taken when `/expeditor promote` is run from Slack promote: action: - bash:.expeditor/promote.sh - trigger_pipeline:deploy/production: only_if_conditions: - value_one: "{{target_channel}}" operator: equals value_two: stable - bash:.expeditor/purge-cdn.sh: post_commit: true channels: - acceptance - stable merge_actions: - built_in:trigger_habitat_package_build: post_commit: true ignore_labels: - "Habitat: Skip Build" - "Expeditor: Skip All" habitat_packages: - chef-web-util: origin: chefops export: - docker subscriptions: - workload: docker_image_published:chefops/chef-web-util:acceptance actions: .expeditor/config.yml

Slide 25

Slide 25 text

Expeditor listens to webhooks and takes actions

Slide 26

Slide 26 text

Expeditor Integrations • Buildkite • Docker/Docker Hub • GitHub • Habitat/Habitat Builder • Slack

Slide 27

Slide 27 text

Slack notification

Slide 28

Slide 28 text

Delete merged branches

Slide 29

Slide 29 text

Artifact promotion

Slide 30

Slide 30 text

Trigger Habitat package builds

Slide 31

Slide 31 text

Publish Docker Images

Slide 32

Slide 32 text

pipelines: - verify: description: Pull Request validation tests - deploy/acceptance: description: Deploy changes to kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: acceptance - APP: APPNAME - deploy/production: description: Deploy changes to kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: production - APP: APPNAME .expeditor/config.yml: pipelines

Slide 33

Slide 33 text

pipelines: - verify: description: Pull Request validation tests - deploy/acceptance: description: Deploy changes to kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: acceptance - APP: APPNAME - deploy/production: description: Deploy changes to kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: production - APP: APPNAME .expeditor/config.yml: pipelines .expeditor/verify.pipeline.yml

Slide 34

Slide 34 text

pipelines: - verify: description: Pull Request validation tests - deploy/acceptance: description: Deploy changes to kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: acceptance - APP: APPNAME - deploy/production: description: Deploy changes to kubernetes definition: .expeditor/deploy.pipeline.yml env: - ENVIRONMENT: production - APP: APPNAME .expeditor/config.yml: pipelines

Slide 35

Slide 35 text

steps: - label: static command: | static syntax and lint checking commands (shellcheck, rubocop, etc) plugins: docker#v1.1.1: image: "chefes/buildkite" - label: unit #... unit testing commands, rspec, etc - label: audit #... for auditing bundles in ruby apps for example .expeditor/verify.pipeline.yml

Slide 36

Slide 36 text

New PRs will run Verify pipeline

Slide 37

Slide 37 text

Verify Pipeline - Success!

Slide 38

Slide 38 text

steps: - command: .expeditor/buildkite/kubernetes.sh label: "Kubernetes" concurrency: 1 concurrency_group: chef-nginx-demo-master/deploy/$ENVIRONMENT plugins: docker#v1.1.1: always-pull: true image: "chefes/buildkite" environment: - CHEF_CD_AWS_ACCESS_KEY_ID - CHEF_CD_AWS_SECRET_ACCESS_KEY - AWS_SSL_ARN - ENVIRONMENT - CI - APP - HAB_AUTH_TOKEN deploy.pipeline.yml

Slide 39

Slide 39 text

View the new pipelines

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

Applications!

Slide 42

Slide 42 text

Habitat https:/ /www.habitat.sh

Slide 43

Slide 43 text

Build, Deploy, Manage

Slide 44

Slide 44 text

Service Lifecycle Management

Slide 45

Slide 45 text

Packaging

Slide 46

Slide 46 text

pkg_name="chef-web-util" pkg_origin="chefops" pkg_version="1.0.0" pkg_maintainer="Chef Operations " pkg_license=("Proprietary") pkg_scaffolding="core/scaffolding-ruby" scaffolding_ruby_pkg="core/ruby/$(cat ${PLAN_CONTEXT}/../.ruby-version)" pkg_deps=(core/curl) plan.sh

Slide 47

Slide 47 text

chef-web-util: Source Path: /src chef-web-util: Installed Path: /hab/pkgs/chefops/chef-web-util/ 1.0.0/20181010192533 chef-web-util: Artifact: /src/results/chefops-chef-web- util-1.0.0-20181010192533-x86_64-linux.hart chef-web-util: Build Report: /src/results/last_build.env chef-web-util: SHA256 Checksum: a11540fb7908d4d1e67c56165b9a4388a0c10858aa729b029cc4aa9bf9770feb chef-web-util: Blake2b Checksum: 7a987da5536703ac1301d3c5ed17fe510dabf04d84b02ee052df1a1da7d6c467 chef-web-util: chef-web-util: I love it when a plan.sh comes together. chef-web-util: chef-web-util: Build time: 1m36s hab studio enter > build

Slide 48

Slide 48 text

Habitat packages: • Are timestamp-based artifacts • Declare all their dependencies • Are "self contained" • https://habitat.sh

Slide 49

Slide 49 text

Build Service

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

hab pkg export docker

Slide 53

Slide 53 text

Push Image to Docker Hub

Slide 54

Slide 54 text

Orchestration / Choreography

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

https://flic.kr/p/bFe9Bs

Slide 57

Slide 57 text

Kubernetes is a portable, extensible open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation.

Slide 58

Slide 58 text

Tl;dr: Docker as a Service https://libreshot.com/container-ship-freighter/

Slide 59

Slide 59 text

https://flic.kr/p/4wWPEB

Slide 60

Slide 60 text

steps: - command: .expeditor/buildkite/kubernetes.sh label: "Kubernetes" concurrency: 1 concurrency_group: chef-nginx-demo-master/deploy/$ENVIRONMENT plugins: docker#v1.1.1: always-pull: true image: "chefes/buildkite" environment: - CHEF_CD_AWS_ACCESS_KEY_ID - CHEF_CD_AWS_SECRET_ACCESS_KEY - AWS_SSL_ARN - ENVIRONMENT - CI - APP - HAB_AUTH_TOKEN deploy.pipeline.yml

Slide 61

Slide 61 text

#!/bin/bash set -euo pipefail if [[ ! -z ${CI+x} ]]; then aws-configure chef-cd mkdir -p ~/.kube aws --profile chef-cd s3 cp s3://chef-cd-citadel/kubernetes.chef.co.config ~/.kube/config else echo "WARN: Not running in Buildkite, assuming local manual deployment" echo "WARN: This requires that ~/.kube/config exists with the proper content" fi export ENVIRONMENT=${ENVIRONMENT:-dev} export APP=${APP} DEBUG=${DEBUG:-false} # This block translates the "environment" into the appropriate Habitat # channel from which to deploy the packages if [ "$ENVIRONMENT" == "acceptance" ]; then export CHANNEL=acceptance elif [ "$ENVIRONMENT" == "production" ]; then export CHANNEL=stable elif [ "$ENVIRONMENT" == "dev" ] || [ "$ENVIRONMENT" == "test" ]; then export CHANNEL=unstable else echo "We do not currently support deploying to $ENVIRONMENT" exit 1 fi # We need the HAB_AUTH_TOKEN set (via Buildkite pipeline) for private packages get_image_tag() { results=$(curl --silent -H "Authorization: Bearer $HAB_AUTH_TOKEN" https://willem.habitat.sh/v1/depot/channels/chefops/${CHANNEL}/pkgs/${APP}/latest | jq '.ident') pkg_version=$(echo "$results" | jq -r .version) pkg_release=$(echo "$results" | jq -r .release) echo "${pkg_version}-${pkg_release}" } # Retrieves the ELB's public DNS name get_elb_hostname() { kubectl get services ${APP}-${ENVIRONMENT} --namespace=${APP} -o json 2>/dev/null | \ jq '.status.loadBalancer.ingress[].hostname' -r } # The ELB isn't ready until the hostname is set, so wait until it's ready wait_for_elb() { attempts=0 max_attempts=10 elb_host="" while [[ $attempts -lt $max_attempts ]]; do elb_host=$(get_elb_hostname || echo) if [[ ! -n $elb_host ]]; then echo "Did not find ELB yet... sleeping 5s" attempts=$[$attempts + 1] sleep 5 else echo "Found ELB: $elb_host" break fi done } # Used for debugging on a local workstation if [[ $DEBUG == "true" ]]; then echo "--- DEBUG: Environment" echo "Application: ${APP}" echo "Channel: ${CHANNEL}" echo "Environment: ${ENVIRONMENT}" fi echo "--- Applying kubernetes configuration for ${ENVIRONMENT} to cluster" IMAGE_TAG=$(get_image_tag) erb -T- kubernetes/deployment.yml | kubectl apply -f - if [[ `grep -c "^kind: Service$" kubernetes/deployment.yml` -gt 0 ]]; then echo "+++ Waiting for Load Balancer..." wait_for_elb fi kubernetes.sh

Slide 62

Slide 62 text

.expeditor/buildkite/kubernetes.sh • Deploy pipeline in Buildkite runs the script • Copies credentials used for kubectl • Retrieves the Habitat package information from Builder • Applies the kubernetes/deployment.yml manifest • Waits until the ELB is available (if there is one) • Success!

Slide 63

Slide 63 text

Habitat package data • Version • Release • Channel(s)

Slide 64

Slide 64 text

export IMAGE_TAG=$(get_image_tag) erb -T- kubernetes/deployment.yml | kubectl apply -f - kubernetes.sh

Slide 65

Slide 65 text

Kubernetes Resources • Pods • StatefulSets & Deployments • Services • Namespaces • Custom Resource Definitions (CRDs)

Slide 66

Slide 66 text

--- apiVersion: v1 kind: Namespace metadata: name: nginx-demo --- apiVersion: habitat.sh/v1beta1 kind: Habitat metadata: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> namespace: nginx-demo labels: app: nginx-demo-<%= ENV['ENVIRONMENT'] %> customVersion: v1beta2 spec: v1beta2: image: chefops/nginx-demo:<%= ENV['IMAGE_TAG'] %> count: 2 service: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> topology: standalone --- apiVersion: v1 kind: Service metadata: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> namespace: nginx-demo spec: selector: habitat-name: nginx-demo-<%= ENV['ENVIRONMENT'] %> ports: - name: http port: 80 targetPort: 80 protocol: TCP type: LoadBalancer Kubernetes: A Primer via Configuration

Slide 67

Slide 67 text

--- apiVersion: v1 kind: Namespace metadata: name: nginx-demo kubernetes/deployment.yml - Namespace • Namespaces isolate resources in "virtual clusters" • Each application gets a namespace

Slide 68

Slide 68 text

apiVersion: habitat.sh/v1beta1 kind: Habitat metadata: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> namespace: nginx-demo labels: app: nginx-demo-<%= ENV['ENVIRONMENT'] %> customVersion: v1beta2 spec: v1beta2: image: chefops/nginx-demo:<%= ENV['IMAGE_TAG'] %> count: 2 service: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> topology: standalone kubernetes/deployment.yml - Habitat • Applications are deployed using the Habitat Operator • This creates a StatefulSet with two pods • The pods will use the Docker image specified

Slide 69

Slide 69 text

Habitat Operator https:/ /www.habitat.sh/get-started/kubernetes/

Slide 70

Slide 70 text

apiVersion: v1 kind: Service metadata: name: nginx-demo-<%= ENV['ENVIRONMENT'] %> namespace: nginx-demo spec: selector: habitat-name: nginx-demo-<%= ENV['ENVIRONMENT'] %> ports: - name: http port: 80 targetPort: 80 protocol: TCP type: LoadBalancer kubernetes/deployment.yml - Service

Slide 71

Slide 71 text

% kubectl get all -n nginx-demo NAME READY STATUS RESTARTS AGE pod/nginx-demo-acceptance-0 1/1 Running 0 4m pod/nginx-demo-acceptance-1 1/1 Running 0 4m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-demo-acceptance LoadBalancer 100.64.186.238 ac1532ae98ed5... 80:31027/TCP 4m NAME DESIRED CURRENT AGE statefulset.apps/nginx-demo-acceptance 2 2 4m Kubernetes resources are created

Slide 72

Slide 72 text

Infrastructure Automation

Slide 73

Slide 73 text

Kops - "Kubernetes Operations"

Slide 74

Slide 74 text

apiVersion: kops/v1alpha2 kind: InstanceGroup metadata: labels: kops.k8s.io/cluster: kubernetes.chef-internal name: nodes spec: additionalUserData: - content: | #!/bin/sh - installs, configures, and runs chef name: chefbootstrap.sh type: text/x-shellscript cloudLabels: X-Application: kubernetes X-Contact: jtimberman X-Dept: Operations X-Environment: production image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08 machineType: m4.xlarge maxSize: 6 minSize: 4 nodeLabels: kops.k8s.io/instancegroup: nodes role: Node subnets: - us-west-2a Kops nodes and masters configuration

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

additionalUserData: - content: | #!/bin/sh apt-get -y update apt-get -y install python-pip pip install awscli mkdir -p /etc/chef /usr/local/bin/aws --region us-east-1 s3 cp s3://citadel-bucket/validation.pem /etc/chef/validation.pem INSTANCE_ID=$(curl -s 169.254.169.254/1.0/meta-data/instance-id) cat >/etc/chef/client.rb <

Slide 77

Slide 77 text

cat >/etc/chef/client.rb <

Slide 78

Slide 78 text

k8s-node Chef Policyfile recipes • User management (LDAP, SSH Keys) • Centralized Logging • Send Inspec profile data to Chef Automate • Keep Chef package updated • Ensure Chef is running on a schedule

Slide 79

Slide 79 text

Verification and Monitoring

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

Acceptance and Stakeholders https://en.wikipedia.org/wiki/Acceptance_testing#/media/File:James_Webb_Primary_Mirror.jpg

Slide 82

Slide 82 text

Monitoring

Slide 83

Slide 83 text

Monitoring Observability

Slide 84

Slide 84 text

Prometheus + Grafana

Slide 85

Slide 85 text

https://github.com/coreos/prometheus-operator

Slide 86

Slide 86 text

No content

Slide 87

Slide 87 text

SECRETS??!!

Slide 88

Slide 88 text

Secrets Management • SSL certificates • API tokens • SSH keys • Configuration files • Repositories • AWS Accounts • Kubernetes

Slide 89

Slide 89 text

No content

Slide 90

Slide 90 text

Walkthrough

Slide 91

Slide 91 text

Procedure 1.Make changes to the Repository, open a pull request 2.Wait for Buildkite verification, peer approval, then merge 3.Wait for Expeditor to complete merge actions 4.Verify acceptance with stakeholders 5.Promote to Production

Slide 92

Slide 92 text

Previous Procedure • Let's just say... • It was more than 5 steps • And most of them were manual • Unfortunate, really • For an Automation Company

Slide 93

Slide 93 text

Make changes to the repository, open a pull request git checkout -b $USERNAME/my-branch # make some changes with your favorite editor git add . git commit -m 'updating the site with new features or fixes' git push origin $USERNAME/my-branch

Slide 94

Slide 94 text

Wait for Expeditor to complete merge actions

Slide 95

Slide 95 text

Verify Acceptance with stakeholders

Slide 96

Slide 96 text

Promote to Production

Slide 97

Slide 97 text

Coworker: "Wait... that's all?"

Slide 98

Slide 98 text

Conclusion

Slide 99

Slide 99 text

Procedure 1.Make changes to the Repository, open a pull request 2.Wait for Buildkite verification, peer approval, then merge 3.Wait for Expeditor to complete merge actions 4.Verify acceptance with stakeholders 5.Promote to Production

Slide 100

Slide 100 text

Automation has shifted

Slide 101

Slide 101 text

Workflow > Infrastructure

Slide 102

Slide 102 text

Business benefit

Slide 103

Slide 103 text

Questions? Joshua Timberman @jtimberman Resources and Links https://www.habitat.sh https://www.lita.io/ https://buildkite.com https://github.com/habitat-sh/habitat-operator https://github.com/coreos/prometheus-operator