Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Taming Jenkins

Taming Jenkins

Continuous Integration is often the weak link in the move from infrastructure as a collection of pets to a well-organized cattle ranch. Jenkins, while extremely useful as an in-house CI factory, is more often than not a very opaque piece of legacy infrastructure, holding key company information hostage in a maze of XML files.

In this talk, we will look at recent changes in Jenkins that make it more compatible with the infrastructure as code movement, building a repeatable, scalable, and secure build infrastructure with simple configuration.

Pierre-Yves Ritschard

November 17, 2017
Tweet

More Decks by Pierre-Yves Ritschard

Other Decks in Programming

Transcript

  1. HALLO HALLO : Three-line Bio CTO & Co-founder at Distributed

    systems and monitoring enthusiast Open-Source developer Clojure Libraries, OpenBSD, Riemann, Collectd, and more. @pyr Exoscale 3 . 1
  2. EXOSCALE EXOSCALE Infrastructure as a service Part of A1 Digital

    Zones in Frankfurt, Vienna, Zürich, Geneva 5 . 1
  3. EXOSCALE EXOSCALE provider "exoscale" { api_key = "${var.exoscale_api_key}" secret_key =

    "${var.exoscale_secret_key}" } resource "exoscale_instance" "web" { template = "Ubuntu 17.04" disk_size = "50g" profile = "medium" ssh_key = "production" } 7 . 1
  4. I THOUGHT THIS WAS A DEV I THOUGHT THIS WAS

    A DEV CONFERENCE! CONFERENCE! 8 . 1
  5. WHAT'S IN A CLOUD PROVIDER WHAT'S IN A CLOUD PROVIDER

    Datacenter operations So ware development 9 . 1
  6. SOFTWARE AT EXOSCALE SOFTWARE AT EXOSCALE Object storage controller Network

    controller Internal SDN Customer management Metering system Billing Web portal 10 . 1
  7. REPRODUCIBILITY REPRODUCIBILITY Building once (and in chroots) ensures clean packages

    Reproducible builds make wide changes easier We need staging deploys and production deploys to be identical 16 . 1
  8. OBSERVABILITY OBSERVABILITY When did we last build this? What did

    the output look like? What commit did it correspond to? 17 . 1
  9. CHECKPOINTS CHECKPOINTS CD is great for test and staging We

    are wary of unattended production deploys There should be a clear (but simple) trigger 18 . 1
  10. CLOUD-NATIVE CLOUD-NATIVE Configuration as code Fast infrastructure set-up and teardown

    No decay over time It should be easy to grow or shrink the factory's throughput 19 . 1
  11. MOST OF IT IS FINE MOST OF IT IS FINE

    But legacy lurks in every environment. 30 . 1
  12. OUR JENKINS OUR JENKINS A single machine, builds close to

    configuration Jenkins user has sudo access Which plugins are installed? How is config restored? 33 . 1
  13. HOW DID WE GET HERE? HOW DID WE GET HERE?

    Jenkins makes automation hard Password in logs No clear configuration file standard 34 . 1
  14. HOW DID WE GET HERE? HOW DID WE GET HERE?

    <?xml version='1.0' encoding='UTF-8'?> <project> <actions/> <description>week in review</description> <keepDependencies>false</keepDependencies> <properties> <com.coravy.hudson.plugins.github.GithubProjectProperty plugin="[email protected]"> <projectUrl>https://github.com/exoscale/wir/</projectUrl> </com.coravy.hudson.plugins.github.GithubProjectProperty> <com.tikal.hudson.plugins.notification.HudsonNotificationProperty plugin="notifi <endpoints> <com.tikal.hudson.plugins.notification.Endpoint> <format>JSON</format> <url>http://notifier:8080/hubot/jenkins-notify</url> <event>all</event> <timeout>30000</timeout> <loglines>0</loglines> </com.tikal.hudson.plugins.notification.Endpoint> </endpoints> </com.tikal.hudson.plugins.notification.HudsonNotificationProperty> <hudson.model.ParametersDefinitionProperty> <parameterDefinitions> <hudson.model.StringParameterDefinition> <name>branch</name> ... 35 . 1
  15. NO STANDARD HOOK MECHANISM NO STANDARD HOOK MECHANISM All job

    configuration is in-situ No way to externally say: run IRC notifications for all these jobs Even fewer avenues for templating 38 . 1
  16. FEW MITIGATION STRATEGIES FEW MITIGATION STRATEGIES Jobs are edited through

    the web Backups to git as primary recovery mechanism 39 . 1
  17. JENKINS IN 2017: A NEW HOPE JENKINS IN 2017: A

    NEW HOPE Pipeline Jobs Job DSL Init via groovy 43 . 1
  18. PIPELINE JOBS PIPELINE JOBS node { stage('build') { checkout scm

    sh 'lein compile :all' sh 'lein uberjar' archiveArtifacts artifacts: '**/target/*.jar', fingerprint: true } stage('test') { sh 'lein test' } } 45 . 1
  19. PIPELINE JOBS PIPELINE JOBS . ├── Dockerfile ├── Jenkinsfile ├──

    project.clj ├── src │ └── foo │ └── core.clj └── test └── foo └── core_test.clj 4 directories, 5 files 46 . 1
  20. PIPELINE JOBS PIPELINE JOBS Keeps build information close to code

    No impedance mismatch between job and code Easier to work in branches 47 . 1
  21. JOB DSL JOB DSL def githubPipelineJob(org, name, private_repo=true) { pipelineJob

    name { definition { cpsScm { scriptPath 'JenkinsFile' git { remote { github "$org/$name" if (private_repo) { credential 'github-token' } } } } } } } githubPipelineJob('exoscale', 'portal') githubPipelineJob('exoscale', 'api') ... 48 . 1
  22. JOB DSL JOB DSL A single job generates all jobs

    Updates / Removals when job list changes All Jenkins jobs can be expressed 49 . 1
  23. REGISTERING BUILD HOSTS REGISTERING BUILD HOSTS Most plugins require some

    form of manual intervention Most are operated from the master 50 . 1
  24. REGISTERING BUILD HOSTS REGISTERING BUILD HOSTS Reverse engineered API Private

    SSH key deployed on the master As a named credential Public SSH key deployed on host 51 . 1
  25. THE LAST MISSING PIECE THE LAST MISSING PIECE Still need

    to bootstrap Jenkins Still need to provision additional configuration Plugins Perms Credentials 52 . 1
  26. LIVING THE DREAM LIVING THE DREAM executors: 0 url: "http://jenkins/"

    seedjob: git: "https://github.com/exoscale/jobs.git" name: "seed-job" credentials: "github-token" credentials: ssh-key: type: PRIVATE_KEY username: ubuntu description: "Private key for host SSH access" key: | ... github-token: type: USER_PASSWORD username: build-bot password: "..." description: "Github autogenerated token credentials for builds" perms: admin: password: '...' allowed: [ 'JENKINS_ADMINISTER' ] builder: password: '...' allowed: - 'JENKINS_READ'
  27. - 'ITEM_BUILD' - 'ITEM_READ' - ... worker: password: '...' allowed:

    - 'JENKINS_READ' - 'COMPUTER_EXTENDED_READ' - 'COMPUTER_CONNECT' - ... 54 . 1
  28. LIVING THE DREAM LIVING THE DREAM ant build-timeout email-ext extended-read-permission

    github-branch-source gradle job-dsl workflow-aggregator pipeline-github-lib ssh-slaves timestamper ws-cleanup blueocean matrix-auth authorize-project 55 . 1
  29. GROOVY INITIAL SETUP GROOVY INITIAL SETUP def realm = new

    HudsonPrivateSecurityRealm(false) def strategy = new FullControlOnceLoggedInAuthorizationStrategy() realm.createAccount('admin', 'TemporaryLongPassword') jenkins.setSecurityRealm(realm) jenkins.setAuthorizationStrategy(strategy) jenkins.getDescriptor("jenkins.CLI").get().setEnabled(false) jenkins.save() 56 . 1
  30. GROOVY PLUGIN PROVISIONING GROOVY PLUGIN PROVISIONING def path = System.getenv("JENKINS_PLUGIN_CONFIG")

    ?: "/etc/jenkins/plugins.list" plugins = new File(path).readLines() def pm = jenkins.getPluginManager() def uc = jenkins.getUpdateCenter() def install_list = plugins.findAll{ !pm.getPlugin(it)} install_list.each { def plugin = uc.getPlugin(it) if (plugin) { def ftr = plugin.deploy() ftr.get(1, TimeUnit.MINUTES) } } jenkins.save() if (install_list) { jenkins.restart() } 57 . 1
  31. GROOVY YAML LOAD GROOVY YAML LOAD @Grab('org.yaml:snakeyaml:1.18') def path =

    System.getenv("JENKINS_YAML_CONFIG") ?: "/etc/jenkins/jenkins.yaml" config = new Yaml().load(new File(path).getText()) 58 . 1
  32. GROOVY JOB DSL BOOTSTRAP GROOVY JOB DSL BOOTSTRAP dslScript =

    new ExecuteDslScripts(null) dslScript.setTargets("jobs.groovy") dslScript.setLookupStrategy(LookupStrategy.JENKINS_ROOT) remotes = GitSCM.createRepoList(config.seedjob.git, config.seedjob.credentials) branches = [new BranchSpec("*/master")] browser = new GithubWeb(config.seedjob.git) dslScm = new GitSCM(remotes, branches, false, [], browser, "git", []) job = new hudson.model.FreeStyleProject(jenkins, config.seedjob.name) job.scm = dslScm job.getPublishersList().add(dslScript); jenkins.add(job, config.seedjob.name) 59 . 1
  33. LOOKING FORWARD LOOKING FORWARD Learning a bit more Groovy :-)

    Less dependencies on build hosts More artifact repositories Easier interaction with hosts 62 . 1