Running_Legacy_Apps_on_K8s.pdf

Transcript

1. Legacy Applications on
   Kubernetes
   Joshua Timberman
   @jtimberman
   

   View Slide

2. https://commons.wikimedia.org/wiki/File:BalticServers_data_center.jpg
   % whoami
   

   View Slide

3. Why are we here?
   

   View Slide

4. Running Legacy Applications on
   Kubernetes
   

   View Slide

5. Running Legacy Heritage
   Applications on Kubernetes
   

   View Slide

6. What is a Legacy Application?
   In computing, a legacy application program of, relating to, or
   being a previous or outdated computer system, yet still in use.
   Often referencing a system as "legacy" means that it paved
   the way for the standards that would follow it. This can
   also imply that the system is out of date or in need of
   replacement.
   

   View Slide

7. Chef Ops runs many applications
   

   View Slide

8. Applications, sample set
   • Hosted Chef (api.chef.io)
   • Downloads (downloads.chef.io)
   • Omnitruck (omnitruck.chef.io)
   • Learn Chef (learn.chef.io)
   • Chef Community Cookbooks (supermarket.chef.io)
   • Internal and External utilities
   • "Gateway" and "glue" services
   • Kubernetes itself ;)
   • etc...
   

   View Slide

9. Workflow Matters
   

   View Slide

10. Why Kubernetes?
    

    View Slide

11. Cloud Native
    

    View Slide

12. Platform Consistency
    

    View Slide

13. Cohesion
    

    View Slide

14. Cost-savings
    https://en.wikipedia.org/wiki/Denomination_effect#/media/File:Exchange_Money_Conversion_to_Foreign_Currency.jpg
    

    View Slide

15. https://commons.wikimedia.org/wiki/File:Frozen_peas.JPG
    

    View Slide

16. https://commons.wikimedia.org/wiki/File:Mashed_potatoes_(1).jpg
    

    View Slide

17. What about Chef?
    

    View Slide

18. As it turns out...
    https://www.flickr.com/photos/tofu_mugwump/22420266811
    https://creativecommons.org/licenses/by/2.0/
    

    View Slide

19. We still have an OS.
    

    View Slide

20. But we use K8SaaS!
    https://www.jpl.nasa.gov/news/news.php?feature=7113
    

    View Slide

21. View Slide

22. Policyfiles
    https://en.wikipedia.org/wiki/Ledger#/media/File:Ledger.png
    

    View Slide

23. View Slide

24. Test Kitchen (for cookbooks)
    https://commons.wikimedia.org/wiki/File:Conference_House_-_Rustic_Kitchen.jpg
    

    View Slide

25. DevSec Linux Baseline
    https://dev-sec.io/baselines/linux/
    

    View Slide

26. CIS AWS Benchmark
    https://commons.wikimedia.org/wiki/File:Benchmark_at_former_Abbey_Cinema.jpg
    

    View Slide

27. View Slide

28. Ruby Scaffolding
    https://commons.wikimedia.org/wiki/File:Seattle_-
    _construction_at_SE_corner_of_11th_%26_E_Pine_01.jpg
    

    View Slide

29. core-plans/scaffolding-base
    core-plans/scaffolding-chef
    core-plans/scaffolding-go
    core-plans/scaffolding-go17
    core-plans/scaffolding-gradle
    core-plans/scaffolding-inspec
    core-plans/scaffolding-node
    core-plans/scaffolding-ruby
    scaffolding plans
    

    View Slide

30. Effortless Config
    

    View Slide

31. How did we get here?
    

    View Slide

32. View Slide

33. https://flic.kr/p/bFe9Bs
    

    View Slide

34. Kubernetes is a portable,
    extensible open-source platform
    for managing containerized
    workloads and services, that
    facilitates both declarative
    configuration and automation.
    

    View Slide

35. Tl;dr: Docker as a Service
    https://libreshot.com/container-ship-freighter/
    

    View Slide

36. YAML*
    * The "A" is optional, though.
    

    View Slide

37. kops - "Kubernetes Operations"
    

    View Slide

38. apiVersion: kops/v1alpha2
    kind: InstanceGroup
    metadata:
    labels:
    kops.k8s.io/cluster: kubernetes.chef-internal
    name: nodes
    spec:
    additionalUserData:
    - content: |
    #!/bin/sh - installs, configures, and runs chef
    name: chefbootstrap.sh
    type: text/x-shellscript
    cloudLabels:
    X-Application: kubernetes
    X-Contact: jtimberman
    X-Dept: Operations
    X-Environment: production
    image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08
    machineType: m4.xlarge
    maxSize: 6
    minSize: 4
    nodeLabels:
    kops.k8s.io/instancegroup: nodes
    role: Node
    subnets:
    - us-west-2a
    Kops nodes and masters configuration
    

    View Slide

39. #!/bin/bash
    pkg_origin=$1
    pkg_name=$2
    if [ ! -e "/bin/hab" ]; then
    curl https://raw.githubusercontent.com/habitat-sh/habitat/master/components/hab/install.sh |
    bash
    fi
    [[ `grep "^hab:" /etc/passwd` ]] || useradd hab
    [[ `grep "^hab:" /etc/group `]] || groupadd hab
    hab pkg install $pkg_origin/$pkg_name
    pkg_prefix=$(find /hab/pkgs/$pkg_origin/$pkg_name -maxdepth 2 -mindepth 2 | sort | tail -n 1)
    hab pkg exec $pkg_origin/$pkg_name chef-client -z -c $pkg_prefix/config/bootstrap-config.rb
    chefbootstrap.sh - Effortless Config!
    

    View Slide

40. k8s-node Chef Infra Policyfile
    • User management (LDAP, SSH Keys)
    • Centralized Logging
    • Send InSpec profile data to Chef Automate
    • Keeps Chef Infra package updated
    • Ensure Chef Infra is running as a service
    

    View Slide

41. CI / CD
    

    View Slide

42. == CI SaaS...
    

    View Slide

43. Expeditor
    

    View Slide

44. Expeditor listens to webhooks and takes actions
    

    View Slide

45. pipelines:
    - verify:
    description: Pull Request validation tests
    - habitat/build:
    description: Build Habitat package
    - deploy/acceptance:
    description: Deploy changes to Kubernetes
    definition: .expeditor/deploy.pipeline.yml
    env:
    - ENVIRONMENT: acceptance
    - APP: APPNAME
    - deploy/production:
    description: Deploy changes to Kubernetes
    definition: .expeditor/deploy.pipeline.yml
    env:
    - ENVIRONMENT: production
    - APP: APPNAME
    .expeditor/config.yml: pipelines
    

    View Slide

46. YAML INTENSIFIES
    

    View Slide

47. View Slide

48. Build Habitat Packages
    

    View Slide

49. jtimberman@localhost% hab studio enter
    ...
    [1][default:/src:0]# build
    ...
    learnchef-backend:
    learnchef-backend: I love it when a plan.sh comes
    together.
    learnchef-backend:
    learnchef-backend: Build time: 2m24s
    Build Habitat Packages
    

    View Slide

50. View Slide

51. Publish Docker Images
    

    View Slide

52. View Slide

53. pipelines:
    - verify:
    description: Pull Request validation tests
    - deploy/acceptance:
    description: Deploy changes to kubernetes
    definition: .expeditor/deploy.pipeline.yml
    env:
    - ENVIRONMENT: acceptance
    - APP: APPNAME
    - deploy/production:
    description: Deploy changes to kubernetes
    definition: .expeditor/deploy.pipeline.yml
    env:
    - ENVIRONMENT: production
    - APP: APPNAME
    Buildkite Deploys to Kubernetes
    

    View Slide

54. SECRETS??!!
    https://commons.wikimedia.org/wiki/File:WinonaSavingsBankVault.JPG
    

    View Slide

55. Secrets Management
    • SSL certificates
    • API tokens
    • SSH keys
    • Configuration files
    • Repositories
    • AWS Accounts
    • Kubernetes
    

    View Slide

56. View Slide

57. K8s Secrets / ConfigMap
    https://commons.wikimedia.org/wiki/File:World_map_.jpg
    

    View Slide

58. Application Migration
    https://www.maxpixel.net/Fly-Flock-Of-Birds-Crane-Bird-Migration-2147788
    

    View Slide

59. Habitat Package
    

    View Slide

60. pkg_name=APPNAME
    pkg_origin=chefops
    pkg_version=0.1.0
    pkg_maintainer="Chef Operations "
    pkg_license=('Apache-2.0')
    pkg_deps=(
    core/coreutils
    )
    pkg_build_deps=(
    core/make
    core/gcc
    core/tar
    )
    pkg_scaffolding=core/scaffolding-ruby
    scaffolding_ruby_pkg=core/ruby24/$(cat "$SRC_PATH/.ruby-version")
    declare -A scaffolding_env
    scaffolding_env[APPNAME_YAML]="/hab/svc/$pkg_name/config/config.yml"
    habitat/plan.sh
    

    View Slide

61. Expeditor & Buildkite
    

    View Slide

62. Expeditor & Buildkite Configuration
    • Pipeline definitions
    • Expeditor's config.yml
    • Scripts run by pipelines in Buildkite
    • Kubernetes deployment
    

    View Slide

63. Kubernetes Configuration
    

    View Slide

64. #!/bin/bash
    set -euo pipefail
    if [[ ! -z ${CI+x} ]]; then
    mkdir -p ~/.kube
    aws --profile chef-cd s3 cp s3://chef-cd-citadel/kubernetes.chef.co.config ~/.kube/config
    else
    echo "WARN: Not running in Buildkite, assuming local manual deployment"
    echo "WARN: This requires that ~/.kube/config exists with the proper content"
    fi
    export ENVIRONMENT=${ENVIRONMENT:-dev}
    export APP=${APP}
    DEBUG=${DEBUG:-false}
    # This block translates the "environment" into the appropriate Habitat
    # channel from which to deploy the packages
    if [ "$ENVIRONMENT" == "acceptance" ]; then
    export CHANNEL=acceptance
    elif [ "$ENVIRONMENT" == "production" ]; then
    export CHANNEL=stable
    elif [ "$ENVIRONMENT" == "dev" ] || [ "$ENVIRONMENT" == "test" ]; then
    export CHANNEL=unstable
    else
    echo "We do not currently support deploying to $ENVIRONMENT"
    exit 1
    fi
    # We need the HAB_AUTH_TOKEN set (via Buildkite pipeline) for private packages
    get_image_tag() {
    results=$(curl --silent -H "Authorization: Bearer $HAB_AUTH_TOKEN" https://willem.habitat.sh/v1/depot/channels/chefops/${CHANNEL}/pkgs/${APP}/latest | jq '.ident')
    pkg_version=$(echo "$results" | jq -r .version)
    pkg_release=$(echo "$results" | jq -r .release)
    echo "${pkg_version}-${pkg_release}"
    }
    # Retrieves the ELB's public DNS name
    get_elb_hostname() {
    kubectl get services ${APP}-${ENVIRONMENT} --namespace=${APP} -o json 2>/dev/null | \
    jq '.status.loadBalancer.ingress[].hostname' -r
    }
    # The ELB isn't ready until the hostname is set, so wait until it's ready
    wait_for_elb() {
    attempts=0
    max_attempts=10
    elb_host=""
    while [[ $attempts -lt $max_attempts ]]; do
    elb_host=$(get_elb_hostname || echo)
    if [[ ! -n $elb_host ]]; then
    echo "Did not find ELB yet... sleeping 5s"
    attempts=$[$attempts + 1]
    sleep 5
    else
    echo "Found ELB: $elb_host"
    break
    fi
    done
    }
    # Used for debugging on a local workstation
    if [[ $DEBUG == "true" ]]; then
    echo "--- DEBUG: Environment"
    echo "Application: ${APP}"
    echo "Channel: ${CHANNEL}"
    echo "Environment: ${ENVIRONMENT}"
    echo "Version: $(get_image_tag)"
    fi
    echo "--- Applying kubernetes configuration for ${ENVIRONMENT} to cluster"
    IMAGE_TAG=$(get_image_tag) erb -T- kubernetes/deployment.yml | kubectl apply -f -
    if [[ `grep -c "^kind: Service$" kubernetes/deployment.yml` -gt 0 ]]; then
    echo "+++ Waiting for Load Balancer..."
    wait_for_elb
    fi
    kubernetes.sh
    

    View Slide

65. .expeditor/buildkite/kubernetes.sh
    • Deploy pipeline in Buildkite runs the script
    • Copies credentials used for kubectl
    • Retrieves the Habitat package information from
    Builder
    • Applies the kubernetes/deployment.yml
    manifest
    • Waits until the ELB is available (if there is one)
    • Success!
    

    View Slide

66. Habitat package information
    • Version
    • Release
    • Channel(s)
    • `get_image_tag`
    

    View Slide

67. export IMAGE_TAG=$(get_image_tag)
    erb -T- kubernetes/deployment.yml | kubectl apply -f -
    kubernetes.sh
    

    View Slide

68. Kubernetes Resources
    • Pods
    • StatefulSets & Deployments
    • Services
    • Namespaces
    • Custom Resource Definitions (CRDs)
    • (e.g., Habitat operator, Prometheus operator)
    

    View Slide

69. ---
    apiVersion: v1
    kind: Namespace
    metadata:
    name: nginx-demo
    ---
    apiVersion: habitat.sh/v1beta1
    kind: Habitat
    metadata:
    name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    namespace: nginx-demo
    labels:
    app: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    customVersion: v1beta2
    spec:
    v1beta2:
    image: chefops/nginx-demo:<%= ENV['IMAGE_TAG'] %>
    count: 2
    service:
    name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    topology: standalone
    ---
    apiVersion: v1
    kind: Service
    metadata:
    name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    namespace: nginx-demo
    spec:
    selector:
    habitat-name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    ports:
    - name: http
    port: 80
    targetPort: 80
    protocol: TCP
    type: LoadBalancer
    Kubernetes: A Primer via Configuration
    

    View Slide

70. ---
    apiVersion: v1
    kind: Namespace
    metadata:
    name: nginx-demo
    kubernetes/deployment.yml - Namespace
    • Namespaces isolate resources in "virtual clusters"
    • Each application gets a namespace
    

    View Slide

71. apiVersion: habitat.sh/v1beta1
    kind: Habitat
    metadata:
    name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    namespace: nginx-demo
    labels:
    app: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    customVersion: v1beta2
    spec:
    v1beta2:
    image: chefops/nginx-demo:<%= ENV['IMAGE_TAG'] %>
    count: 2
    service:
    name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    topology: standalone
    kubernetes/deployment.yml - Habitat
    • Applications are deployed using the Habitat Operator
    • This creates a StatefulSet with two pods
    • The pods will use the Docker image specified
    

    View Slide

72. apiVersion: v1
    kind: Service
    metadata:
    name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    namespace: nginx-demo
    spec:
    selector:
    habitat-name: nginx-demo-<%= ENV['ENVIRONMENT'] %>
    ports:
    - name: http
    port: 80
    targetPort: 80
    protocol: TCP
    type: LoadBalancer
    kubernetes/deployment.yml - Service
    

    View Slide

73. % kubectl get pods -n omnitruck
    NAME READY STATUS RESTARTS AGE
    omnitruck-acceptance-0 1/1 Running 0 5d9h
    omnitruck-acceptance-1 1/1 Running 0 5d9h
    omnitruck-poller-acceptance-1558471800-hvh45 0/1 Completed 0 31m
    omnitruck-poller-acceptance-1558472400-d4pps 0/1 Completed 0 21m
    omnitruck-poller-acceptance-1558473000-dsmcn 0/1 Completed 0 11m
    omnitruck-poller-acceptance-1558473600-b72lg 1/1 Running 0 112s
    omnitruck-poller-production-1558471800-rbt7n 0/1 Completed 0 31m
    omnitruck-poller-production-1558472400-92qkv 0/1 Completed 0 21m
    omnitruck-poller-production-1558473000-fb6lh 0/1 Completed 0 11m
    omnitruck-poller-production-1558473600-lw868 1/1 Running 0 111s
    omnitruck-production-0 1/1 Running 0 5d9h
    omnitruck-production-1 1/1 Running 0 5d9h
    omnitruck-production-2 1/1 Running 0 5d9h
    pods
    

    View Slide

74. % kubectl get services -n omnitruck
    NAME TYPE CLUSTER-IP
    EXTERNAL-IP PORT(S) AGE
    omnitruck-acceptance LoadBalancer 100.69.37.56
    a260b98a3fcab... 443:32433/TCP 162d
    omnitruck-production LoadBalancer 100.70.244.59
    a92a69a04fcc2... 443:31683/TCP 161d
    services
    

    View Slide

75. Stateless Services Only
    

    View Slide

76. Monitoring
    

    View Slide

77. Monitoring Observability
    

    View Slide

78. Prometheus + Grafana
    

    View Slide

79. https://github.com/coreos/prometheus-operator
    

    View Slide

80. View Slide

81. % uptime
    

    View Slide

82. Workflow Steps
    

    View Slide

83. No Live Demo Today!
    https://flic.kr/p/RJ5xEs
    

    View Slide

84. Procedure
    1.Make changes to the Repository, open a pull request
    2.Wait for Buildkite verification, peer approval, then
    merge
    3.Wait for Expeditor to complete merge actions
    4.Verify acceptance with stakeholders
    5.Promote to Production
    

    View Slide

85. Previous Procedure
    • Let's just say...
    • It was more than 5 steps
    • And many of them were manual
    • And the procedure differed slightly between
    applications...
    

    View Slide

86. Make changes to the repository, open a pull
    request
    git checkout -b $USERNAME/my-branch
    # make some changes with your favorite editor
    git add .
    git commit -m 'updating the site with new features or fixes'
    git push origin $USERNAME/my-branch
    

    View Slide

87. Wait for Expeditor to complete merge
    actions
    

    View Slide

88. Verify Acceptance with stakeholders
    

    View Slide

89. Promote to Production
    

    View Slide

90. Coworker: "Wait... that's all?"
    

    View Slide

91. *Spoiler: That's not all...
    https://commons.wikimedia.org/wiki/File:Bugatti_veyron2.jpg
    

    View Slide

92. Conclusion
    

    View Slide

93. Automation has shifted
    

    View Slide

94. Workflow > Infrastructure
    

    View Slide

95. Business benefit
    

    View Slide

96. Questions?
    

    View Slide

97. Thank you!
    Joshua Timberman
    @jtimberman
    All images used in this presentation are public domain, created by me,
    or licensed under the Creative Commons Attribution-Share Alike 4.0 license:
    https://creativecommons.org/licenses/by-sa/4.0/deed.en
    The only modification made to any images is resizing to fit into the slides
    

    View Slide