$30 off During Our Annual Pro Sale. View Details »

Taming Jenkins

Taming Jenkins

Continuous Integration is often the weak link in the move from infrastructure as a collection of pets to a well-organized cattle ranch. Jenkins, while extremely useful as an in-house CI factory, is more often than not a very opaque piece of legacy infrastructure, holding key company information hostage in a maze of XML files.

In this talk, we will look at recent changes in Jenkins that make it more compatible with the infrastructure as code movement, building a repeatable, scalable, and secure build infrastructure with simple configuration.

Pierre-Yves Ritschard

November 17, 2017
Tweet

More Decks by Pierre-Yves Ritschard

Other Decks in Programming

Transcript

  1. TAMING JENKINS
    TAMING JENKINS
    BUILDING A CLOUD-NATIVE CI ENVIRONMENT
    BUILDING A CLOUD-NATIVE CI ENVIRONMENT
    1

    View Slide

  2. 2 . 1

    View Slide

  3. HALLO
    HALLO
    : Three-line Bio
    CTO & Co-founder at
    Distributed systems and monitoring enthusiast
    Open-Source developer
    Clojure Libraries, OpenBSD, Riemann, Collectd, and more.
    @pyr
    Exoscale
    3 . 1

    View Slide

  4. TAMING JENKINS
    TAMING JENKINS
    Building a Cloud-Native CI environment
    4 . 1

    View Slide

  5. EXOSCALE
    EXOSCALE
    Infrastructure as a service
    Part of A1 Digital
    Zones in Frankfurt, Vienna, Zürich, Geneva
    5 . 1

    View Slide

  6. EXOSCALE
    EXOSCALE
    6 . 1

    View Slide

  7. EXOSCALE
    EXOSCALE
    provider "exoscale" {
    api_key = "${var.exoscale_api_key}"
    secret_key = "${var.exoscale_secret_key}"
    }
    resource "exoscale_instance" "web" {
    template = "Ubuntu 17.04"
    disk_size = "50g"
    profile = "medium"
    ssh_key = "production"
    }
    7 . 1

    View Slide

  8. I THOUGHT THIS WAS A DEV
    I THOUGHT THIS WAS A DEV
    CONFERENCE!
    CONFERENCE!
    8 . 1

    View Slide

  9. WHAT'S IN A CLOUD PROVIDER
    WHAT'S IN A CLOUD PROVIDER
    Datacenter operations
    So ware development
    9 . 1

    View Slide

  10. SOFTWARE AT EXOSCALE
    SOFTWARE AT EXOSCALE
    Object storage controller
    Network controller
    Internal SDN
    Customer management
    Metering system
    Billing
    Web portal
    10 . 1

    View Slide

  11. LANGUAGES AT EXOSCALE
    LANGUAGES AT EXOSCALE
    C & Go
    Clojure
    Python
    ClojureScript & JS
    11 . 1

    View Slide

  12. THE BUILD FACTORY
    THE BUILD FACTORY
    From code to artifact
    From artifact to deploy
    12 . 1

    View Slide

  13. BUILD FACTORY
    BUILD FACTORY
    13 . 1

    View Slide

  14. MAIN DESIGN IDEA
    MAIN DESIGN IDEA
    Security
    Reproducibility
    Observability
    Checkpoints
    Cloud-Native
    14 . 1

    View Slide

  15. SECURITY
    SECURITY
    No code download on production hosts
    Signed packages
    15 . 1

    View Slide

  16. REPRODUCIBILITY
    REPRODUCIBILITY
    Building once (and in chroots) ensures clean packages
    Reproducible builds make wide changes easier
    We need staging deploys and production deploys to be identical
    16 . 1

    View Slide

  17. OBSERVABILITY
    OBSERVABILITY
    When did we last build this?
    What did the output look like?
    What commit did it correspond to?
    17 . 1

    View Slide

  18. CHECKPOINTS
    CHECKPOINTS
    CD is great for test and staging
    We are wary of unattended production deploys
    There should be a clear (but simple) trigger
    18 . 1

    View Slide

  19. CLOUD-NATIVE
    CLOUD-NATIVE
    Configuration as code
    Fast infrastructure set-up and teardown
    No decay over time
    It should be easy to grow or shrink the factory's throughput
    19 . 1

    View Slide

  20. BUILD FACTORY
    BUILD FACTORY
    20 . 1

    View Slide

  21. BUILD FACTORY
    BUILD FACTORY
    21 . 1

    View Slide

  22. BUILD FACTORY
    BUILD FACTORY
    22 . 1

    View Slide

  23. BUILD FACTORY
    BUILD FACTORY
    23 . 1

    View Slide

  24. BUILD FACTORY
    BUILD FACTORY
    24 . 1

    View Slide

  25. BUILD FACTORY
    BUILD FACTORY
    25 . 1

    View Slide

  26. BUILD FACTORY
    BUILD FACTORY
    26 . 1

    View Slide

  27. BUILD FACTORY
    BUILD FACTORY
    27 . 1

    View Slide

  28. BUILD FACTORY
    BUILD FACTORY
    28 . 1

    View Slide

  29. ARE WE THERE YET?
    ARE WE THERE YET?
    29 . 1

    View Slide

  30. MOST OF IT IS FINE
    MOST OF IT IS FINE
    But legacy lurks in every environment.
    30 . 1

    View Slide

  31. THE CHECKLIST
    THE CHECKLIST
    Security ✔
    Reproducibility ✔
    Observability ✔
    Checkpoints ✔
    Cloud-Native ❌
    31 . 1

    View Slide

  32. OUR JENKINS
    OUR JENKINS
    32 . 1

    View Slide

  33. OUR JENKINS
    OUR JENKINS
    A single machine, builds close to configuration
    Jenkins user has sudo access
    Which plugins are installed?
    How is config restored?
    33 . 1

    View Slide

  34. HOW DID WE GET HERE?
    HOW DID WE GET HERE?
    Jenkins makes automation hard
    Password in logs
    No clear configuration file standard
    34 . 1

    View Slide

  35. HOW DID WE GET HERE?
    HOW DID WE GET HERE?



    week in review
    false


    https://github.com/exoscale/wir/

    View Slide

  36. XML-SERIALIZED OBJECTS
    XML-SERIALIZED OBJECTS

    View Slide

  37. XML-SERIALIZED OBJECTS
    XML-SERIALIZED OBJECTS
    Lengthy descriptions
    No standard text to config
    Too many combinations to template
    37 . 1

    View Slide

  38. NO STANDARD HOOK MECHANISM
    NO STANDARD HOOK MECHANISM
    All job configuration is in-situ
    No way to externally say: run IRC notifications for all these jobs
    Even fewer avenues for templating
    38 . 1

    View Slide

  39. FEW MITIGATION STRATEGIES
    FEW MITIGATION STRATEGIES
    Jobs are edited through the web
    Backups to git as primary recovery mechanism
    39 . 1

    View Slide

  40. DIRECT CONSEQUENCE
    DIRECT CONSEQUENCE
    Jenkins down means no deploy
    Time to recovery: ???
    40 . 1

    View Slide

  41. BE GENTLE
    BE GENTLE
    This was all started in 2012
    41 . 1

    View Slide

  42. JENKINS AUTOMATION, CIRCA 2012
    JENKINS AUTOMATION, CIRCA 2012
    42 . 1

    View Slide

  43. JENKINS IN 2017: A NEW HOPE
    JENKINS IN 2017: A NEW HOPE
    Pipeline Jobs
    Job DSL
    Init via groovy
    43 . 1

    View Slide

  44. OUR IMPLEMENTATION
    OUR IMPLEMENTATION
    The road to a cloud-native CI environment
    44 . 1

    View Slide

  45. PIPELINE JOBS
    PIPELINE JOBS
    node {
    stage('build') {
    checkout scm
    sh 'lein compile :all'
    sh 'lein uberjar'
    archiveArtifacts artifacts: '**/target/*.jar', fingerprint: true
    }
    stage('test') {
    sh 'lein test'
    }
    }
    45 . 1

    View Slide

  46. PIPELINE JOBS
    PIPELINE JOBS
    .
    ├── Dockerfile
    ├── Jenkinsfile
    ├── project.clj
    ├── src
    │ └── foo
    │ └── core.clj
    └── test
    └── foo
    └── core_test.clj
    4 directories, 5 files
    46 . 1

    View Slide

  47. PIPELINE JOBS
    PIPELINE JOBS
    Keeps build information close to code
    No impedance mismatch between job and code
    Easier to work in branches
    47 . 1

    View Slide

  48. JOB DSL
    JOB DSL
    def githubPipelineJob(org, name, private_repo=true) {
    pipelineJob name {
    definition {
    cpsScm {
    scriptPath 'JenkinsFile'
    git {
    remote {
    github "$org/$name"
    if (private_repo) { credential 'github-token' }
    }
    }
    }
    }
    }
    }
    githubPipelineJob('exoscale', 'portal')
    githubPipelineJob('exoscale', 'api')
    ...
    48 . 1

    View Slide

  49. JOB DSL
    JOB DSL
    A single job generates all jobs
    Updates / Removals when job list changes
    All Jenkins jobs can be expressed
    49 . 1

    View Slide

  50. REGISTERING BUILD HOSTS
    REGISTERING BUILD HOSTS
    Most plugins require some form of manual intervention
    Most are operated from the master
    50 . 1

    View Slide

  51. REGISTERING BUILD HOSTS
    REGISTERING BUILD HOSTS
    Reverse engineered API
    Private SSH key deployed on the master
    As a named credential
    Public SSH key deployed on host
    51 . 1

    View Slide

  52. THE LAST MISSING PIECE
    THE LAST MISSING PIECE
    Still need to bootstrap Jenkins
    Still need to provision additional configuration
    Plugins
    Perms
    Credentials
    52 . 1

    View Slide

  53. GROOVY INITIALIZATION SUPPORT
    GROOVY INITIALIZATION SUPPORT
    Code in $JENKINS_HOME/init.groovy.d/ gets picked up
    All Jenkins functionality can be accessed
    53 . 1

    View Slide

  54. LIVING THE DREAM
    LIVING THE DREAM
    executors: 0
    url: "http://jenkins/"
    seedjob:
    git: "https://github.com/exoscale/jobs.git"
    name: "seed-job"
    credentials: "github-token"
    credentials:
    ssh-key:
    type: PRIVATE_KEY
    username: ubuntu
    description: "Private key for host SSH access"
    key: |
    ...
    github-token:
    type: USER_PASSWORD
    username: build-bot
    password: "..."
    description: "Github autogenerated token credentials for builds"
    perms:
    admin:
    password: '...'
    allowed: [ 'JENKINS_ADMINISTER' ]
    builder:
    password: '...'
    allowed:
    - 'JENKINS_READ'

    View Slide

  55. - 'ITEM_BUILD'
    - 'ITEM_READ'
    - ...
    worker:
    password: '...'
    allowed:
    - 'JENKINS_READ'
    - 'COMPUTER_EXTENDED_READ'
    - 'COMPUTER_CONNECT'
    - ...
    54 . 1

    View Slide

  56. LIVING THE DREAM
    LIVING THE DREAM
    ant
    build-timeout
    email-ext
    extended-read-permission
    github-branch-source
    gradle
    job-dsl
    workflow-aggregator
    pipeline-github-lib
    ssh-slaves
    timestamper
    ws-cleanup
    blueocean
    matrix-auth
    authorize-project
    55 . 1

    View Slide

  57. GROOVY INITIAL SETUP
    GROOVY INITIAL SETUP
    def realm = new HudsonPrivateSecurityRealm(false)
    def strategy = new FullControlOnceLoggedInAuthorizationStrategy()
    realm.createAccount('admin', 'TemporaryLongPassword')
    jenkins.setSecurityRealm(realm)
    jenkins.setAuthorizationStrategy(strategy)
    jenkins.getDescriptor("jenkins.CLI").get().setEnabled(false)
    jenkins.save()
    56 . 1

    View Slide

  58. GROOVY PLUGIN PROVISIONING
    GROOVY PLUGIN PROVISIONING
    def path = System.getenv("JENKINS_PLUGIN_CONFIG") ?: "/etc/jenkins/plugins.list"
    plugins = new File(path).readLines()
    def pm = jenkins.getPluginManager()
    def uc = jenkins.getUpdateCenter()
    def install_list = plugins.findAll{ !pm.getPlugin(it)}
    install_list.each {
    def plugin = uc.getPlugin(it)
    if (plugin) {
    def ftr = plugin.deploy()
    ftr.get(1, TimeUnit.MINUTES)
    }
    }
    jenkins.save()
    if (install_list) { jenkins.restart() }
    57 . 1

    View Slide

  59. GROOVY YAML LOAD
    GROOVY YAML LOAD
    @Grab('org.yaml:snakeyaml:1.18')
    def path = System.getenv("JENKINS_YAML_CONFIG") ?: "/etc/jenkins/jenkins.yaml"
    config = new Yaml().load(new File(path).getText())
    58 . 1

    View Slide

  60. GROOVY JOB DSL BOOTSTRAP
    GROOVY JOB DSL BOOTSTRAP
    dslScript = new ExecuteDslScripts(null)
    dslScript.setTargets("jobs.groovy")
    dslScript.setLookupStrategy(LookupStrategy.JENKINS_ROOT)
    remotes = GitSCM.createRepoList(config.seedjob.git, config.seedjob.credentials)
    branches = [new BranchSpec("*/master")]
    browser = new GithubWeb(config.seedjob.git)
    dslScm = new GitSCM(remotes, branches, false, [], browser, "git", [])
    job = new hudson.model.FreeStyleProject(jenkins, config.seedjob.name)
    job.scm = dslScm
    job.getPublishersList().add(dslScript);
    jenkins.add(job, config.seedjob.name)
    59 . 1

    View Slide

  61. ARE WE THERE YET?
    ARE WE THERE YET?
    60 . 1

    View Slide

  62. MUCH CLOSER
    MUCH CLOSER
    61 . 1

    View Slide

  63. LOOKING FORWARD
    LOOKING FORWARD
    Learning a bit more Groovy :-)
    Less dependencies on build hosts
    More artifact repositories
    Easier interaction with hosts
    62 . 1

    View Slide

  64. THANKS!
    THANKS!
    Questions?
    We are hiring!
    Try Exoscale out!
    on twitter and github
    @pyr
    63 . 1

    View Slide

  65. 64 . 1

    View Slide