Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling Jenkins with Kubernetes on Google Container Engine

Scaling Jenkins with Kubernetes on Google Container Engine

Guillaume Laforge

July 11, 2017
Tweet

More Decks by Guillaume Laforge

Other Decks in Technology

Transcript

  1. Scale Jenkins
    with Kubernetes
    on Google Container Engine
    Guillaume Laforge
    Developer Advocate
    Google Cloud Platform
    @glaforge

    View Slide

  2. Agenda
    What are all
    those cool tools?
    Why would we want to
    use these for our CI/CD?
    How are we going to
    implement this?

    View Slide

  3. What?

    View Slide

  4. View Slide

  5. Source: https://www.cloudbees.com/jenkins/about

    View Slide

  6. View Slide

  7. Source: https://kubernetes.io/

    View Slide

  8. @glaforge
    Kubernetes concepts
    Pod
    Volume
    Service
    Deployment
    Cluster
    Label
    Selector
    Secret
    Ingress
    Liveness Probe
    Readiness Probe
    Job
    Liveness Probe
    Readiness Probe
    ReplicaSet
    DaemonSet
    StatefulSet
    ConfigMap

    View Slide

  9. View Slide

  10. Source: https://cloud.google.com/container-engine/

    View Slide

  11. Why?

    View Slide

  12. @glaforge
    One developer

    View Slide

  13. @glaforge
    A few developers

    View Slide

  14. @glaforge
    Tons of developers!

    View Slide

  15. @glaforge
    One project

    View Slide

  16. @glaforge
    Tons of projects

    View Slide

  17. @glaforge
    Many projects & developers, tons of commits...

    View Slide

  18. With more slaves,
    you’d build a pyramid!

    View Slide

  19. @glaforge
    Our solution!
    + +

    View Slide

  20. How?
    How?

    View Slide

  21. DEMO / LAB
    https://github.com/GoogleCloudPlatform
    /continuous-deployment-on-kubernetes

    View Slide

  22. @glaforge
    Creating a Kubernetes cluster for Jenkins
    // set a compute zone to run our environment
    $ gcloud config set compute/zone us-east1-d
    // create a cluster with 3 minimum nodes
    $ gcloud container clusters create jenkins-cd --num-nodes 3
    --scopes "https://www.googleapis.com/auth/projecthosting,storage-rw"
    // retrieve the credentials for our cluster
    $ gcloud container clusters get-credentials jenkins-cd

    View Slide

  23. @glaforge
    Creating a Kubernetes cluster for Jenkins
    // create a dedicated namespace
    $ kubectl create ns jenkins
    // create the Jenkins home volume
    $ gcloud compute images create jenkins-home-image --source-uri
    https://storage.googleapis.com/solutions-public-assets/jenkins-cd/jenkins-home-v3.tar.gz
    $ gcloud compute disks create jenkins-home --image jenkins-home-image

    View Slide

  24. @glaforge
    Creating a Kubernetes cluster for Jenkins
    ● Jenkins Kubernetes Plugin
    Start Jenkins build executor containers in the Kubernetes cluster when
    builds are requested, terminate those containers when builds complete,
    freeing resources up for the rest of the cluster
    ● Jenkins Pipelines
    Define our build pipeline declaratively and keep it checked into source
    code management alongside our application code
    ● Google OAuth Plugin
    Allows you to add your google OAuth credentials to Jenkins

    View Slide

  25. @glaforge
    Creating a Kubernetes cluster for Jenkins
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: jenkins
    namespace: jenkins
    spec:
    replicas: 1
    template:
    metadata:
    labels:
    app: master
    spec:
    containers:
    - name: master
    image: jenkins:2.7.2
    ports:
    - containerPort: 8080
    - containerPort: 50000
    volumeMounts:
    - mountPath: /var/jenkins_home
    name: jenkins-home
    resources:
    limits:
    cpu: 500m
    memory: 1500Mi
    requests:
    cpu: 500m
    memory: 1500Mi
    volumes:
    - name: jenkins-home
    gcePersistentDisk:
    pdName: jenkins-home
    fsType: ext4
    partition: 1
    readinessProbe:
    httpGet:
    path: /login
    port: 8080
    periodSeconds: 10
    timeoutSeconds: 5
    successThreshold: 2
    failureThreshold: 5
    env:
    - name: JENKINS_OPTS
    valueFrom:
    secretKeyRef:
    name: jenkins
    key: options
    jenkins.yaml

    View Slide

  26. @glaforge
    Creating a Kubernetes cluster for Jenkins
    service_jenkins.yaml
    # jenkins_service_ui
    ---
    kind: Service
    apiVersion: v1
    metadata:
    name: jenkins-ui
    namespace: jenkins
    spec:
    type: NodePort
    selector:
    app: master
    ports:
    - protocol: TCP
    port: 8080
    targetPort: 8080
    name: ui
    # jenkins_service_discovery
    ---
    kind: Service
    apiVersion: v1
    metadata:
    name: jenkins-discovery
    namespace: jenkins
    spec:
    selector:
    app: master
    ports:
    - protocol: TCP
    port: 50000
    targetPort: 50000
    name: slaves

    View Slide

  27. @glaforge
    The app to build and deploy

    View Slide

  28. @glaforge
    // create a new cluster for our app
    $ kubectl create ns production
    // setup the different environments
    $ kubectl --namespace=production apply -f k8s/production
    $ kubectl --namespace=production apply -f k8s/canary
    $ kubectl --namespace=production apply -f k8s/services
    // scale the production service
    $ kubectl --namespace=production scale deployment
    gceme-frontend-production --replicas=4
    Building and deploying

    View Slide

  29. @glaforge
    Building and deploying
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
    name: gceme-frontend-production
    spec:
    replicas:
    template:
    metadata:
    name: frontend
    labels:
    app: gceme
    role: frontend
    env: production
    spec:
    containers:
    - name: frontend
    image: gcr.io/cloud-solutions-images/gceme:1.0.0
    resources:
    limits:
    memory: "500Mi"
    cpu: "100m"
    imagePullPolicy: Always
    readinessProbe:
    httpGet:
    path: /healthz
    port: 80
    command: ["sh", "-c",
    "app -frontend=true -backend-service=http://gceme-backend:8080 -
    ports:
    - name: frontend
    containerPort: 80
    frontend_production.yaml

    View Slide

  30. @glaforge
    Building and deploying
    FROM jenkinsci/jnlp-slave
    ENV CLOUDSDK_CORE_DISABLE_PROMPTS 1
    ENV PATH /opt/google-cloud-sdk/bin:$PATH
    USER root
    RUN apt-get update -y
    RUN apt-get install -y jq
    RUN curl https://sdk.cloud.google.com |
    bash && mv google-cloud-sdk /opt
    RUN gcloud components install kubectl
    Dockerfile

    View Slide

  31. @glaforge
    Building and deploying
    node {
    def project = 'jenkins-gke-scaling'
    def appName = 'gceme'
    def feSvcName = "${appName}-frontend"
    def imageTag = "gcr.io/${project}/${appName}:${env.BRANCH_NAME}.${env.BUILD_NUMBER}"
    checkout scm
    stage 'Build image'
    sh("docker build -t ${imageTag} .")
    stage 'Run Go tests'
    sh("docker run ${imageTag} go test")
    stage 'Push image to registry'
    sh("gcloud docker -- push ${imageTag}")
    Jenkinsfile

    View Slide

  32. @glaforge
    Building and deploying
    stage "Deploy Application"
    switch (env.BRANCH_NAME) {
    // Roll out to canary environment
    case "canary":
    // Change deployed image in canary to the one we just built
    sh("sed -i.bak 's#gcr.io/cloud-solutions-images/gceme:1.0.0#${imageTag}#' ./k8s/canary/*.yaml")
    sh("kubectl --namespace=production apply -f k8s/services/")
    sh("kubectl --namespace=production apply -f k8s/canary/")
    sh("echo http://`kubectl --namespace=production get service/${feSvcName} --output=json | " +
    "jq -r '.status.loadBalancer.ingress[0].ip'` > ${feSvcName}")
    break
    Jenkinsfile

    View Slide

  33. @glaforge
    Building and deploying
    // Roll out to production
    case "master":
    // Change deployed image in canary to the one we just built
    sh("sed -i.bak 's#gcr.io/cloud-solutions-images/gceme:1.0.0#${imageTag}#' ./k8s/production/*.yaml")
    sh("kubectl --namespace=production apply -f k8s/services/")
    sh("kubectl --namespace=production apply -f k8s/production/")
    sh("echo http://`kubectl --namespace=production get service/${feSvcName} --output=json | " +
    "jq -r '.status.loadBalancer.ingress[0].ip'` > ${feSvcName}")
    break
    Jenkinsfile

    View Slide

  34. @glaforge
    Building and deploying
    // Roll out a dev environment
    default:
    // Create namespace if it doesn't exist
    sh("kubectl get ns ${env.BRANCH_NAME} || kubectl create ns ${env.BRANCH_NAME}")
    // Don't use public load balancing for development branches
    sh("sed -i.bak 's#LoadBalancer#ClusterIP#' ./k8s/services/frontend.yaml")
    sh("sed -i.bak 's#gcr.io/cloud-solutions-images/gceme:1.0.0#${imageTag}#' ./k8s/dev/*.yaml")
    sh("kubectl --namespace=${env.BRANCH_NAME} apply -f k8s/services/")
    sh("kubectl --namespace=${env.BRANCH_NAME} apply -f k8s/dev/")
    echo 'To access your environment run `kubectl proxy`'
    echo "Then access your service via " +
    "http://localhost:8001/api/v1/proxy/namespaces/${env.BRANCH_NAME}/services/${feSvcName}:80/"
    }
    }
    Jenkinsfile

    View Slide

  35. Summary &
    resources

    View Slide

  36. @glaforge
    Summary

    View Slide

  37. @glaforge
    Summary

    View Slide

  38. @glaforge
    Resources
    Overview of Jenkins on Container Engine
    https://cloud.google.com/solutions/jenkins-on-container-engine
    Setting up Jenkins on Container Engine
    https://cloud.google.com/solutions/jenkins-on-container-engine-tutorial
    Configuring Jenkins for Container Engine
    https://cloud.google.com/solutions/configuring-jenkins-container-engine
    Continuous Deployment to Container Engine using Jenkins
    https://cloud.google.com/solutions/continuous-delivery-jenkins-container-engine
    Lab: Build a Continuous Deployment Pipeline with Jenkins and Kubernetes
    https://github.com/GoogleCloudPlatform/continuous-deployment-on-kubernetes

    View Slide

  39. Questions?

    View Slide

  40. KTHXBAI
    And come to
    our booth &
    say hi!

    View Slide