Scaling Jenkins with Kubernetes on Google Container Engine

Transcript

1. Scale Jenkins
   with Kubernetes
   on Google Container Engine
   Guillaume Laforge
   Developer Advocate
   Google Cloud Platform
   @glaforge
   

   View Slide

2. Agenda
   What are all
   those cool tools?
   Why would we want to
   use these for our CI/CD?
   How are we going to
   implement this?
   

   View Slide

3. What?
   

   View Slide

4. View Slide

5. Source: https://www.cloudbees.com/jenkins/about
   

   View Slide

6. View Slide

7. Source: https://kubernetes.io/
   

   View Slide

8. @glaforge
   Kubernetes concepts
   Pod
   Volume
   Service
   Deployment
   Cluster
   Label
   Selector
   Secret
   Ingress
   Liveness Probe
   Readiness Probe
   Job
   Liveness Probe
   Readiness Probe
   ReplicaSet
   DaemonSet
   StatefulSet
   ConfigMap
   

   View Slide

9. View Slide

10. Source: https://cloud.google.com/container-engine/
    

    View Slide

11. Why?
    

    View Slide

12. @glaforge
    One developer
    

    View Slide

13. @glaforge
    A few developers
    

    View Slide

14. @glaforge
    Tons of developers!
    

    View Slide

15. @glaforge
    One project
    

    View Slide

16. @glaforge
    Tons of projects
    

    View Slide

17. @glaforge
    Many projects & developers, tons of commits...
    

    View Slide

18. With more slaves,
    you’d build a pyramid!
    

    View Slide

19. @glaforge
    Our solution!
    + +
    

    View Slide

20. How?
    How?
    

    View Slide

21. DEMO / LAB
    https://github.com/GoogleCloudPlatform
    /continuous-deployment-on-kubernetes
    

    View Slide

22. @glaforge
    Creating a Kubernetes cluster for Jenkins
    // set a compute zone to run our environment
    $ gcloud config set compute/zone us-east1-d
    // create a cluster with 3 minimum nodes
    $ gcloud container clusters create jenkins-cd --num-nodes 3
    --scopes "https://www.googleapis.com/auth/projecthosting,storage-rw"
    // retrieve the credentials for our cluster
    $ gcloud container clusters get-credentials jenkins-cd
    

    View Slide

23. @glaforge
    Creating a Kubernetes cluster for Jenkins
    // create a dedicated namespace
    $ kubectl create ns jenkins
    // create the Jenkins home volume
    $ gcloud compute images create jenkins-home-image --source-uri
    https://storage.googleapis.com/solutions-public-assets/jenkins-cd/jenkins-home-v3.tar.gz
    $ gcloud compute disks create jenkins-home --image jenkins-home-image
    

    View Slide

24. @glaforge
    Creating a Kubernetes cluster for Jenkins
    ● Jenkins Kubernetes Plugin
    Start Jenkins build executor containers in the Kubernetes cluster when
    builds are requested, terminate those containers when builds complete,
    freeing resources up for the rest of the cluster
    ● Jenkins Pipelines
    Define our build pipeline declaratively and keep it checked into source
    code management alongside our application code
    ● Google OAuth Plugin
    Allows you to add your google OAuth credentials to Jenkins
    

    View Slide

25. @glaforge
    Creating a Kubernetes cluster for Jenkins
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: jenkins
    namespace: jenkins
    spec:
    replicas: 1
    template:
    metadata:
    labels:
    app: master
    spec:
    containers:
    - name: master
    image: jenkins:2.7.2
    ports:
    - containerPort: 8080
    - containerPort: 50000
    volumeMounts:
    - mountPath: /var/jenkins_home
    name: jenkins-home
    resources:
    limits:
    cpu: 500m
    memory: 1500Mi
    requests:
    cpu: 500m
    memory: 1500Mi
    volumes:
    - name: jenkins-home
    gcePersistentDisk:
    pdName: jenkins-home
    fsType: ext4
    partition: 1
    readinessProbe:
    httpGet:
    path: /login
    port: 8080
    periodSeconds: 10
    timeoutSeconds: 5
    successThreshold: 2
    failureThreshold: 5
    env:
    - name: JENKINS_OPTS
    valueFrom:
    secretKeyRef:
    name: jenkins
    key: options
    jenkins.yaml
    

    View Slide

26. @glaforge
    Creating a Kubernetes cluster for Jenkins
    service_jenkins.yaml
    # jenkins_service_ui
    ---
    kind: Service
    apiVersion: v1
    metadata:
    name: jenkins-ui
    namespace: jenkins
    spec:
    type: NodePort
    selector:
    app: master
    ports:
    - protocol: TCP
    port: 8080
    targetPort: 8080
    name: ui
    # jenkins_service_discovery
    ---
    kind: Service
    apiVersion: v1
    metadata:
    name: jenkins-discovery
    namespace: jenkins
    spec:
    selector:
    app: master
    ports:
    - protocol: TCP
    port: 50000
    targetPort: 50000
    name: slaves
    

    View Slide

27. @glaforge
    The app to build and deploy
    

    View Slide

28. @glaforge
    // create a new cluster for our app
    $ kubectl create ns production
    // setup the different environments
    $ kubectl --namespace=production apply -f k8s/production
    $ kubectl --namespace=production apply -f k8s/canary
    $ kubectl --namespace=production apply -f k8s/services
    // scale the production service
    $ kubectl --namespace=production scale deployment
    gceme-frontend-production --replicas=4
    Building and deploying
    

    View Slide

29. @glaforge
    Building and deploying
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
    name: gceme-frontend-production
    spec:
    replicas:
    template:
    metadata:
    name: frontend
    labels:
    app: gceme
    role: frontend
    env: production
    spec:
    containers:
    - name: frontend
    image: gcr.io/cloud-solutions-images/gceme:1.0.0
    resources:
    limits:
    memory: "500Mi"
    cpu: "100m"
    imagePullPolicy: Always
    readinessProbe:
    httpGet:
    path: /healthz
    port: 80
    command: ["sh", "-c",
    "app -frontend=true -backend-service=http://gceme-backend:8080 -
    ports:
    - name: frontend
    containerPort: 80
    frontend_production.yaml
    

    View Slide

30. @glaforge
    Building and deploying
    FROM jenkinsci/jnlp-slave
    ENV CLOUDSDK_CORE_DISABLE_PROMPTS 1
    ENV PATH /opt/google-cloud-sdk/bin:$PATH
    USER root
    RUN apt-get update -y
    RUN apt-get install -y jq
    RUN curl https://sdk.cloud.google.com |
    bash && mv google-cloud-sdk /opt
    RUN gcloud components install kubectl
    Dockerfile
    

    View Slide

31. @glaforge
    Building and deploying
    node {
    def project = 'jenkins-gke-scaling'
    def appName = 'gceme'
    def feSvcName = "${appName}-frontend"
    def imageTag = "gcr.io/${project}/${appName}:${env.BRANCH_NAME}.${env.BUILD_NUMBER}"
    checkout scm
    stage 'Build image'
    sh("docker build -t ${imageTag} .")
    stage 'Run Go tests'
    sh("docker run ${imageTag} go test")
    stage 'Push image to registry'
    sh("gcloud docker -- push ${imageTag}")
    Jenkinsfile
    

    View Slide

32. @glaforge
    Building and deploying
    stage "Deploy Application"
    switch (env.BRANCH_NAME) {
    // Roll out to canary environment
    case "canary":
    // Change deployed image in canary to the one we just built
    sh("sed -i.bak 's#gcr.io/cloud-solutions-images/gceme:1.0.0#${imageTag}#' ./k8s/canary/*.yaml")
    sh("kubectl --namespace=production apply -f k8s/services/")
    sh("kubectl --namespace=production apply -f k8s/canary/")
    sh("echo http://`kubectl --namespace=production get service/${feSvcName} --output=json | " +
    "jq -r '.status.loadBalancer.ingress[0].ip'` > ${feSvcName}")
    break
    Jenkinsfile
    

    View Slide

33. @glaforge
    Building and deploying
    // Roll out to production
    case "master":
    // Change deployed image in canary to the one we just built
    sh("sed -i.bak 's#gcr.io/cloud-solutions-images/gceme:1.0.0#${imageTag}#' ./k8s/production/*.yaml")
    sh("kubectl --namespace=production apply -f k8s/services/")
    sh("kubectl --namespace=production apply -f k8s/production/")
    sh("echo http://`kubectl --namespace=production get service/${feSvcName} --output=json | " +
    "jq -r '.status.loadBalancer.ingress[0].ip'` > ${feSvcName}")
    break
    Jenkinsfile
    

    View Slide

34. @glaforge
    Building and deploying
    // Roll out a dev environment
    default:
    // Create namespace if it doesn't exist
    sh("kubectl get ns ${env.BRANCH_NAME} || kubectl create ns ${env.BRANCH_NAME}")
    // Don't use public load balancing for development branches
    sh("sed -i.bak 's#LoadBalancer#ClusterIP#' ./k8s/services/frontend.yaml")
    sh("sed -i.bak 's#gcr.io/cloud-solutions-images/gceme:1.0.0#${imageTag}#' ./k8s/dev/*.yaml")
    sh("kubectl --namespace=${env.BRANCH_NAME} apply -f k8s/services/")
    sh("kubectl --namespace=${env.BRANCH_NAME} apply -f k8s/dev/")
    echo 'To access your environment run `kubectl proxy`'
    echo "Then access your service via " +
    "http://localhost:8001/api/v1/proxy/namespaces/${env.BRANCH_NAME}/services/${feSvcName}:80/"
    }
    }
    Jenkinsfile
    

    View Slide

35. Summary &
    resources
    

    View Slide

36. @glaforge
    Summary
    

    View Slide

37. @glaforge
    Summary
    

    View Slide

38. @glaforge
    Resources
    Overview of Jenkins on Container Engine
    https://cloud.google.com/solutions/jenkins-on-container-engine
    Setting up Jenkins on Container Engine
    https://cloud.google.com/solutions/jenkins-on-container-engine-tutorial
    Configuring Jenkins for Container Engine
    https://cloud.google.com/solutions/configuring-jenkins-container-engine
    Continuous Deployment to Container Engine using Jenkins
    https://cloud.google.com/solutions/continuous-delivery-jenkins-container-engine
    Lab: Build a Continuous Deployment Pipeline with Jenkins and Kubernetes
    https://github.com/GoogleCloudPlatform/continuous-deployment-on-kubernetes
    

    View Slide

39. Questions?
    

    View Slide

40. KTHXBAI
    And come to
    our booth &
    say hi!
    

    View Slide