Taming Jenkins

Taming Jenkins

Continuous Integration is often the weak link in the move from infrastructure as a collection of pets to a well-organized cattle ranch. Jenkins, while extremely useful as an in-house CI factory, is more often than not a very opaque piece of legacy infrastructure, holding key company information hostage in a maze of XML files.

In this talk, we will look at recent changes in Jenkins that make it more compatible with the infrastructure as code movement, building a repeatable, scalable, and secure build infrastructure with simple configuration.

2fcc875f98607b3007909fe4be99160d?s=128

Pierre-Yves Ritschard

November 17, 2017
Tweet

Transcript

  1. TAMING JENKINS TAMING JENKINS BUILDING A CLOUD-NATIVE CI ENVIRONMENT BUILDING

    A CLOUD-NATIVE CI ENVIRONMENT 1
  2. 2 . 1

  3. HALLO HALLO : Three-line Bio CTO & Co-founder at Distributed

    systems and monitoring enthusiast Open-Source developer Clojure Libraries, OpenBSD, Riemann, Collectd, and more. @pyr Exoscale 3 . 1
  4. TAMING JENKINS TAMING JENKINS Building a Cloud-Native CI environment 4

    . 1
  5. EXOSCALE EXOSCALE Infrastructure as a service Part of A1 Digital

    Zones in Frankfurt, Vienna, Zürich, Geneva 5 . 1
  6. EXOSCALE EXOSCALE 6 . 1

  7. EXOSCALE EXOSCALE provider "exoscale" { api_key = "${var.exoscale_api_key}" secret_key =

    "${var.exoscale_secret_key}" } resource "exoscale_instance" "web" { template = "Ubuntu 17.04" disk_size = "50g" profile = "medium" ssh_key = "production" } 7 . 1
  8. I THOUGHT THIS WAS A DEV I THOUGHT THIS WAS

    A DEV CONFERENCE! CONFERENCE! 8 . 1
  9. WHAT'S IN A CLOUD PROVIDER WHAT'S IN A CLOUD PROVIDER

    Datacenter operations So ware development 9 . 1
  10. SOFTWARE AT EXOSCALE SOFTWARE AT EXOSCALE Object storage controller Network

    controller Internal SDN Customer management Metering system Billing Web portal 10 . 1
  11. LANGUAGES AT EXOSCALE LANGUAGES AT EXOSCALE C & Go Clojure

    Python ClojureScript & JS 11 . 1
  12. THE BUILD FACTORY THE BUILD FACTORY From code to artifact

    From artifact to deploy 12 . 1
  13. BUILD FACTORY BUILD FACTORY 13 . 1

  14. MAIN DESIGN IDEA MAIN DESIGN IDEA Security Reproducibility Observability Checkpoints

    Cloud-Native 14 . 1
  15. SECURITY SECURITY No code download on production hosts Signed packages

    15 . 1
  16. REPRODUCIBILITY REPRODUCIBILITY Building once (and in chroots) ensures clean packages

    Reproducible builds make wide changes easier We need staging deploys and production deploys to be identical 16 . 1
  17. OBSERVABILITY OBSERVABILITY When did we last build this? What did

    the output look like? What commit did it correspond to? 17 . 1
  18. CHECKPOINTS CHECKPOINTS CD is great for test and staging We

    are wary of unattended production deploys There should be a clear (but simple) trigger 18 . 1
  19. CLOUD-NATIVE CLOUD-NATIVE Configuration as code Fast infrastructure set-up and teardown

    No decay over time It should be easy to grow or shrink the factory's throughput 19 . 1
  20. BUILD FACTORY BUILD FACTORY 20 . 1

  21. BUILD FACTORY BUILD FACTORY 21 . 1

  22. BUILD FACTORY BUILD FACTORY 22 . 1

  23. BUILD FACTORY BUILD FACTORY 23 . 1

  24. BUILD FACTORY BUILD FACTORY 24 . 1

  25. BUILD FACTORY BUILD FACTORY 25 . 1

  26. BUILD FACTORY BUILD FACTORY 26 . 1

  27. BUILD FACTORY BUILD FACTORY 27 . 1

  28. BUILD FACTORY BUILD FACTORY 28 . 1

  29. ARE WE THERE YET? ARE WE THERE YET? 29 .

    1
  30. MOST OF IT IS FINE MOST OF IT IS FINE

    But legacy lurks in every environment. 30 . 1
  31. THE CHECKLIST THE CHECKLIST Security ✔ Reproducibility ✔ Observability ✔

    Checkpoints ✔ Cloud-Native ❌ 31 . 1
  32. OUR JENKINS OUR JENKINS 32 . 1

  33. OUR JENKINS OUR JENKINS A single machine, builds close to

    configuration Jenkins user has sudo access Which plugins are installed? How is config restored? 33 . 1
  34. HOW DID WE GET HERE? HOW DID WE GET HERE?

    Jenkins makes automation hard Password in logs No clear configuration file standard 34 . 1
  35. HOW DID WE GET HERE? HOW DID WE GET HERE?

    <?xml version='1.0' encoding='UTF-8'?> <project> <actions/> <description>week in review</description> <keepDependencies>false</keepDependencies> <properties> <com.coravy.hudson.plugins.github.GithubProjectProperty plugin="github@1.17.0"> <projectUrl>https://github.com/exoscale/wir/</projectUrl> </com.coravy.hudson.plugins.github.GithubProjectProperty> <com.tikal.hudson.plugins.notification.HudsonNotificationProperty plugin="notifi <endpoints> <com.tikal.hudson.plugins.notification.Endpoint> <format>JSON</format> <url>http://notifier:8080/hubot/jenkins-notify</url> <event>all</event> <timeout>30000</timeout> <loglines>0</loglines> </com.tikal.hudson.plugins.notification.Endpoint> </endpoints> </com.tikal.hudson.plugins.notification.HudsonNotificationProperty> <hudson.model.ParametersDefinitionProperty> <parameterDefinitions> <hudson.model.StringParameterDefinition> <name>branch</name> ... 35 . 1
  36. XML-SERIALIZED OBJECTS XML-SERIALIZED OBJECTS <com.tikal.hudson.plugins.notification.HudsonNotificationProperty plugin="notificati <endpoints> <com.tikal.hudson.plugins.notification.Endpoint> ... </endpoints>

    </com.tikal.hudson.plugins.notification.HudsonNotificationProperty> 36 . 1
  37. XML-SERIALIZED OBJECTS XML-SERIALIZED OBJECTS Lengthy descriptions No standard text to

    config Too many combinations to template 37 . 1
  38. NO STANDARD HOOK MECHANISM NO STANDARD HOOK MECHANISM All job

    configuration is in-situ No way to externally say: run IRC notifications for all these jobs Even fewer avenues for templating 38 . 1
  39. FEW MITIGATION STRATEGIES FEW MITIGATION STRATEGIES Jobs are edited through

    the web Backups to git as primary recovery mechanism 39 . 1
  40. DIRECT CONSEQUENCE DIRECT CONSEQUENCE Jenkins down means no deploy Time

    to recovery: ??? 40 . 1
  41. BE GENTLE BE GENTLE This was all started in 2012

    41 . 1
  42. JENKINS AUTOMATION, CIRCA 2012 JENKINS AUTOMATION, CIRCA 2012 42 .

    1
  43. JENKINS IN 2017: A NEW HOPE JENKINS IN 2017: A

    NEW HOPE Pipeline Jobs Job DSL Init via groovy 43 . 1
  44. OUR IMPLEMENTATION OUR IMPLEMENTATION The road to a cloud-native CI

    environment 44 . 1
  45. PIPELINE JOBS PIPELINE JOBS node { stage('build') { checkout scm

    sh 'lein compile :all' sh 'lein uberjar' archiveArtifacts artifacts: '**/target/*.jar', fingerprint: true } stage('test') { sh 'lein test' } } 45 . 1
  46. PIPELINE JOBS PIPELINE JOBS . ├── Dockerfile ├── Jenkinsfile ├──

    project.clj ├── src │ └── foo │ └── core.clj └── test └── foo └── core_test.clj 4 directories, 5 files 46 . 1
  47. PIPELINE JOBS PIPELINE JOBS Keeps build information close to code

    No impedance mismatch between job and code Easier to work in branches 47 . 1
  48. JOB DSL JOB DSL def githubPipelineJob(org, name, private_repo=true) { pipelineJob

    name { definition { cpsScm { scriptPath 'JenkinsFile' git { remote { github "$org/$name" if (private_repo) { credential 'github-token' } } } } } } } githubPipelineJob('exoscale', 'portal') githubPipelineJob('exoscale', 'api') ... 48 . 1
  49. JOB DSL JOB DSL A single job generates all jobs

    Updates / Removals when job list changes All Jenkins jobs can be expressed 49 . 1
  50. REGISTERING BUILD HOSTS REGISTERING BUILD HOSTS Most plugins require some

    form of manual intervention Most are operated from the master 50 . 1
  51. REGISTERING BUILD HOSTS REGISTERING BUILD HOSTS Reverse engineered API Private

    SSH key deployed on the master As a named credential Public SSH key deployed on host 51 . 1
  52. THE LAST MISSING PIECE THE LAST MISSING PIECE Still need

    to bootstrap Jenkins Still need to provision additional configuration Plugins Perms Credentials 52 . 1
  53. GROOVY INITIALIZATION SUPPORT GROOVY INITIALIZATION SUPPORT Code in $JENKINS_HOME/init.groovy.d/ gets

    picked up All Jenkins functionality can be accessed 53 . 1
  54. LIVING THE DREAM LIVING THE DREAM executors: 0 url: "http://jenkins/"

    seedjob: git: "https://github.com/exoscale/jobs.git" name: "seed-job" credentials: "github-token" credentials: ssh-key: type: PRIVATE_KEY username: ubuntu description: "Private key for host SSH access" key: | ... github-token: type: USER_PASSWORD username: build-bot password: "..." description: "Github autogenerated token credentials for builds" perms: admin: password: '...' allowed: [ 'JENKINS_ADMINISTER' ] builder: password: '...' allowed: - 'JENKINS_READ'
  55. - 'ITEM_BUILD' - 'ITEM_READ' - ... worker: password: '...' allowed:

    - 'JENKINS_READ' - 'COMPUTER_EXTENDED_READ' - 'COMPUTER_CONNECT' - ... 54 . 1
  56. LIVING THE DREAM LIVING THE DREAM ant build-timeout email-ext extended-read-permission

    github-branch-source gradle job-dsl workflow-aggregator pipeline-github-lib ssh-slaves timestamper ws-cleanup blueocean matrix-auth authorize-project 55 . 1
  57. GROOVY INITIAL SETUP GROOVY INITIAL SETUP def realm = new

    HudsonPrivateSecurityRealm(false) def strategy = new FullControlOnceLoggedInAuthorizationStrategy() realm.createAccount('admin', 'TemporaryLongPassword') jenkins.setSecurityRealm(realm) jenkins.setAuthorizationStrategy(strategy) jenkins.getDescriptor("jenkins.CLI").get().setEnabled(false) jenkins.save() 56 . 1
  58. GROOVY PLUGIN PROVISIONING GROOVY PLUGIN PROVISIONING def path = System.getenv("JENKINS_PLUGIN_CONFIG")

    ?: "/etc/jenkins/plugins.list" plugins = new File(path).readLines() def pm = jenkins.getPluginManager() def uc = jenkins.getUpdateCenter() def install_list = plugins.findAll{ !pm.getPlugin(it)} install_list.each { def plugin = uc.getPlugin(it) if (plugin) { def ftr = plugin.deploy() ftr.get(1, TimeUnit.MINUTES) } } jenkins.save() if (install_list) { jenkins.restart() } 57 . 1
  59. GROOVY YAML LOAD GROOVY YAML LOAD @Grab('org.yaml:snakeyaml:1.18') def path =

    System.getenv("JENKINS_YAML_CONFIG") ?: "/etc/jenkins/jenkins.yaml" config = new Yaml().load(new File(path).getText()) 58 . 1
  60. GROOVY JOB DSL BOOTSTRAP GROOVY JOB DSL BOOTSTRAP dslScript =

    new ExecuteDslScripts(null) dslScript.setTargets("jobs.groovy") dslScript.setLookupStrategy(LookupStrategy.JENKINS_ROOT) remotes = GitSCM.createRepoList(config.seedjob.git, config.seedjob.credentials) branches = [new BranchSpec("*/master")] browser = new GithubWeb(config.seedjob.git) dslScm = new GitSCM(remotes, branches, false, [], browser, "git", []) job = new hudson.model.FreeStyleProject(jenkins, config.seedjob.name) job.scm = dslScm job.getPublishersList().add(dslScript); jenkins.add(job, config.seedjob.name) 59 . 1
  61. ARE WE THERE YET? ARE WE THERE YET? 60 .

    1
  62. MUCH CLOSER MUCH CLOSER 61 . 1

  63. LOOKING FORWARD LOOKING FORWARD Learning a bit more Groovy :-)

    Less dependencies on build hosts More artifact repositories Easier interaction with hosts 62 . 1
  64. THANKS! THANKS! Questions? We are hiring! Try Exoscale out! on

    twitter and github @pyr 63 . 1
  65. 64 . 1