Taming Jenkins

Continuous Integration is often the weak link in the move from infrastructure as a collection of pets to a well-organized cattle ranch. Jenkins, while extremely useful as an in-house CI factory, is more often than not a very opaque piece of legacy infrastructure, holding key company information hostage in a maze of XML files.

In this talk, we will look at recent changes in Jenkins that make it more compatible with the infrastructure as code movement, building a repeatable, scalable, and secure build infrastructure with simple configuration.

Transcript

1. TAMING JENKINS
   TAMING JENKINS
   BUILDING A CLOUD-NATIVE CI ENVIRONMENT
   BUILDING A CLOUD-NATIVE CI ENVIRONMENT
   1
   

   View Slide

2. 2 . 1
   

   View Slide

3. HALLO
   HALLO
   : Three-line Bio
   CTO & Co-founder at
   Distributed systems and monitoring enthusiast
   Open-Source developer
   Clojure Libraries, OpenBSD, Riemann, Collectd, and more.
   @pyr
   Exoscale
   3 . 1
   

   View Slide

4. TAMING JENKINS
   TAMING JENKINS
   Building a Cloud-Native CI environment
   4 . 1
   

   View Slide

5. EXOSCALE
   EXOSCALE
   Infrastructure as a service
   Part of A1 Digital
   Zones in Frankfurt, Vienna, Zürich, Geneva
   5 . 1
   

   View Slide

6. EXOSCALE
   EXOSCALE
   6 . 1
   

   View Slide

7. EXOSCALE
   EXOSCALE
   provider "exoscale" {
   api_key = "${var.exoscale_api_key}"
   secret_key = "${var.exoscale_secret_key}"
   }
   resource "exoscale_instance" "web" {
   template = "Ubuntu 17.04"
   disk_size = "50g"
   profile = "medium"
   ssh_key = "production"
   }
   7 . 1
   

   View Slide

8. I THOUGHT THIS WAS A DEV
   I THOUGHT THIS WAS A DEV
   CONFERENCE!
   CONFERENCE!
   8 . 1
   

   View Slide

9. WHAT'S IN A CLOUD PROVIDER
   WHAT'S IN A CLOUD PROVIDER
   Datacenter operations
   So ware development
   9 . 1
   

   View Slide

10. SOFTWARE AT EXOSCALE
    SOFTWARE AT EXOSCALE
    Object storage controller
    Network controller
    Internal SDN
    Customer management
    Metering system
    Billing
    Web portal
    10 . 1
    

    View Slide

11. LANGUAGES AT EXOSCALE
    LANGUAGES AT EXOSCALE
    C & Go
    Clojure
    Python
    ClojureScript & JS
    11 . 1
    

    View Slide

12. THE BUILD FACTORY
    THE BUILD FACTORY
    From code to artifact
    From artifact to deploy
    12 . 1
    

    View Slide

13. BUILD FACTORY
    BUILD FACTORY
    13 . 1
    

    View Slide

14. MAIN DESIGN IDEA
    MAIN DESIGN IDEA
    Security
    Reproducibility
    Observability
    Checkpoints
    Cloud-Native
    14 . 1
    

    View Slide

15. SECURITY
    SECURITY
    No code download on production hosts
    Signed packages
    15 . 1
    

    View Slide

16. REPRODUCIBILITY
    REPRODUCIBILITY
    Building once (and in chroots) ensures clean packages
    Reproducible builds make wide changes easier
    We need staging deploys and production deploys to be identical
    16 . 1
    

    View Slide

17. OBSERVABILITY
    OBSERVABILITY
    When did we last build this?
    What did the output look like?
    What commit did it correspond to?
    17 . 1
    

    View Slide

18. CHECKPOINTS
    CHECKPOINTS
    CD is great for test and staging
    We are wary of unattended production deploys
    There should be a clear (but simple) trigger
    18 . 1
    

    View Slide

19. CLOUD-NATIVE
    CLOUD-NATIVE
    Configuration as code
    Fast infrastructure set-up and teardown
    No decay over time
    It should be easy to grow or shrink the factory's throughput
    19 . 1
    

    View Slide

20. BUILD FACTORY
    BUILD FACTORY
    20 . 1
    

    View Slide

21. BUILD FACTORY
    BUILD FACTORY
    21 . 1
    

    View Slide

22. BUILD FACTORY
    BUILD FACTORY
    22 . 1
    

    View Slide

23. BUILD FACTORY
    BUILD FACTORY
    23 . 1
    

    View Slide

24. BUILD FACTORY
    BUILD FACTORY
    24 . 1
    

    View Slide

25. BUILD FACTORY
    BUILD FACTORY
    25 . 1
    

    View Slide

26. BUILD FACTORY
    BUILD FACTORY
    26 . 1
    

    View Slide

27. BUILD FACTORY
    BUILD FACTORY
    27 . 1
    

    View Slide

28. BUILD FACTORY
    BUILD FACTORY
    28 . 1
    

    View Slide

29. ARE WE THERE YET?
    ARE WE THERE YET?
    29 . 1
    

    View Slide

30. MOST OF IT IS FINE
    MOST OF IT IS FINE
    But legacy lurks in every environment.
    30 . 1
    

    View Slide

31. THE CHECKLIST
    THE CHECKLIST
    Security ✔
    Reproducibility ✔
    Observability ✔
    Checkpoints ✔
    Cloud-Native ❌
    31 . 1
    

    View Slide

32. OUR JENKINS
    OUR JENKINS
    32 . 1
    

    View Slide

33. OUR JENKINS
    OUR JENKINS
    A single machine, builds close to configuration
    Jenkins user has sudo access
    Which plugins are installed?
    How is config restored?
    33 . 1
    

    View Slide

34. HOW DID WE GET HERE?
    HOW DID WE GET HERE?
    Jenkins makes automation hard
    Password in logs
    No clear configuration file standard
    34 . 1
    

    View Slide

35. HOW DID WE GET HERE?
    HOW DID WE GET HERE?
    
    
    
    week in review
    false
    
    
    https://github.com/exoscale/wir/
    
    

    View Slide

36. XML-SERIALIZED OBJECTS
    XML-SERIALIZED OBJECTS
    

    View Slide

37. XML-SERIALIZED OBJECTS
    XML-SERIALIZED OBJECTS
    Lengthy descriptions
    No standard text to config
    Too many combinations to template
    37 . 1
    

    View Slide

38. NO STANDARD HOOK MECHANISM
    NO STANDARD HOOK MECHANISM
    All job configuration is in-situ
    No way to externally say: run IRC notifications for all these jobs
    Even fewer avenues for templating
    38 . 1
    

    View Slide

39. FEW MITIGATION STRATEGIES
    FEW MITIGATION STRATEGIES
    Jobs are edited through the web
    Backups to git as primary recovery mechanism
    39 . 1
    

    View Slide

40. DIRECT CONSEQUENCE
    DIRECT CONSEQUENCE
    Jenkins down means no deploy
    Time to recovery: ???
    40 . 1
    

    View Slide

41. BE GENTLE
    BE GENTLE
    This was all started in 2012
    41 . 1
    

    View Slide

42. JENKINS AUTOMATION, CIRCA 2012
    JENKINS AUTOMATION, CIRCA 2012
    42 . 1
    

    View Slide

43. JENKINS IN 2017: A NEW HOPE
    JENKINS IN 2017: A NEW HOPE
    Pipeline Jobs
    Job DSL
    Init via groovy
    43 . 1
    

    View Slide

44. OUR IMPLEMENTATION
    OUR IMPLEMENTATION
    The road to a cloud-native CI environment
    44 . 1
    

    View Slide

45. PIPELINE JOBS
    PIPELINE JOBS
    node {
    stage('build') {
    checkout scm
    sh 'lein compile :all'
    sh 'lein uberjar'
    archiveArtifacts artifacts: '**/target/*.jar', fingerprint: true
    }
    stage('test') {
    sh 'lein test'
    }
    }
    45 . 1
    

    View Slide

46. PIPELINE JOBS
    PIPELINE JOBS
    .
    ├── Dockerfile
    ├── Jenkinsfile
    ├── project.clj
    ├── src
    │ └── foo
    │ └── core.clj
    └── test
    └── foo
    └── core_test.clj
    4 directories, 5 files
    46 . 1
    

    View Slide

47. PIPELINE JOBS
    PIPELINE JOBS
    Keeps build information close to code
    No impedance mismatch between job and code
    Easier to work in branches
    47 . 1
    

    View Slide

48. JOB DSL
    JOB DSL
    def githubPipelineJob(org, name, private_repo=true) {
    pipelineJob name {
    definition {
    cpsScm {
    scriptPath 'JenkinsFile'
    git {
    remote {
    github "$org/$name"
    if (private_repo) { credential 'github-token' }
    }
    }
    }
    }
    }
    }
    githubPipelineJob('exoscale', 'portal')
    githubPipelineJob('exoscale', 'api')
    ...
    48 . 1
    

    View Slide

49. JOB DSL
    JOB DSL
    A single job generates all jobs
    Updates / Removals when job list changes
    All Jenkins jobs can be expressed
    49 . 1
    

    View Slide

50. REGISTERING BUILD HOSTS
    REGISTERING BUILD HOSTS
    Most plugins require some form of manual intervention
    Most are operated from the master
    50 . 1
    

    View Slide

51. REGISTERING BUILD HOSTS
    REGISTERING BUILD HOSTS
    Reverse engineered API
    Private SSH key deployed on the master
    As a named credential
    Public SSH key deployed on host
    51 . 1
    

    View Slide

52. THE LAST MISSING PIECE
    THE LAST MISSING PIECE
    Still need to bootstrap Jenkins
    Still need to provision additional configuration
    Plugins
    Perms
    Credentials
    52 . 1
    

    View Slide

53. GROOVY INITIALIZATION SUPPORT
    GROOVY INITIALIZATION SUPPORT
    Code in $JENKINS_HOME/init.groovy.d/ gets picked up
    All Jenkins functionality can be accessed
    53 . 1
    

    View Slide

54. LIVING THE DREAM
    LIVING THE DREAM
    executors: 0
    url: "http://jenkins/"
    seedjob:
    git: "https://github.com/exoscale/jobs.git"
    name: "seed-job"
    credentials: "github-token"
    credentials:
    ssh-key:
    type: PRIVATE_KEY
    username: ubuntu
    description: "Private key for host SSH access"
    key: |
    ...
    github-token:
    type: USER_PASSWORD
    username: build-bot
    password: "..."
    description: "Github autogenerated token credentials for builds"
    perms:
    admin:
    password: '...'
    allowed: [ 'JENKINS_ADMINISTER' ]
    builder:
    password: '...'
    allowed:
    - 'JENKINS_READ'
    

    View Slide

55. - 'ITEM_BUILD'
    - 'ITEM_READ'
    - ...
    worker:
    password: '...'
    allowed:
    - 'JENKINS_READ'
    - 'COMPUTER_EXTENDED_READ'
    - 'COMPUTER_CONNECT'
    - ...
    54 . 1
    

    View Slide

56. LIVING THE DREAM
    LIVING THE DREAM
    ant
    build-timeout
    email-ext
    extended-read-permission
    github-branch-source
    gradle
    job-dsl
    workflow-aggregator
    pipeline-github-lib
    ssh-slaves
    timestamper
    ws-cleanup
    blueocean
    matrix-auth
    authorize-project
    55 . 1
    

    View Slide

57. GROOVY INITIAL SETUP
    GROOVY INITIAL SETUP
    def realm = new HudsonPrivateSecurityRealm(false)
    def strategy = new FullControlOnceLoggedInAuthorizationStrategy()
    realm.createAccount('admin', 'TemporaryLongPassword')
    jenkins.setSecurityRealm(realm)
    jenkins.setAuthorizationStrategy(strategy)
    jenkins.getDescriptor("jenkins.CLI").get().setEnabled(false)
    jenkins.save()
    56 . 1
    

    View Slide

58. GROOVY PLUGIN PROVISIONING
    GROOVY PLUGIN PROVISIONING
    def path = System.getenv("JENKINS_PLUGIN_CONFIG") ?: "/etc/jenkins/plugins.list"
    plugins = new File(path).readLines()
    def pm = jenkins.getPluginManager()
    def uc = jenkins.getUpdateCenter()
    def install_list = plugins.findAll{ !pm.getPlugin(it)}
    install_list.each {
    def plugin = uc.getPlugin(it)
    if (plugin) {
    def ftr = plugin.deploy()
    ftr.get(1, TimeUnit.MINUTES)
    }
    }
    jenkins.save()
    if (install_list) { jenkins.restart() }
    57 . 1
    

    View Slide

59. GROOVY YAML LOAD
    GROOVY YAML LOAD
    @Grab('org.yaml:snakeyaml:1.18')
    def path = System.getenv("JENKINS_YAML_CONFIG") ?: "/etc/jenkins/jenkins.yaml"
    config = new Yaml().load(new File(path).getText())
    58 . 1
    

    View Slide

60. GROOVY JOB DSL BOOTSTRAP
    GROOVY JOB DSL BOOTSTRAP
    dslScript = new ExecuteDslScripts(null)
    dslScript.setTargets("jobs.groovy")
    dslScript.setLookupStrategy(LookupStrategy.JENKINS_ROOT)
    remotes = GitSCM.createRepoList(config.seedjob.git, config.seedjob.credentials)
    branches = [new BranchSpec("*/master")]
    browser = new GithubWeb(config.seedjob.git)
    dslScm = new GitSCM(remotes, branches, false, [], browser, "git", [])
    job = new hudson.model.FreeStyleProject(jenkins, config.seedjob.name)
    job.scm = dslScm
    job.getPublishersList().add(dslScript);
    jenkins.add(job, config.seedjob.name)
    59 . 1
    

    View Slide

61. ARE WE THERE YET?
    ARE WE THERE YET?
    60 . 1
    

    View Slide

62. MUCH CLOSER
    MUCH CLOSER
    61 . 1
    

    View Slide

63. LOOKING FORWARD
    LOOKING FORWARD
    Learning a bit more Groovy :-)
    Less dependencies on build hosts
    More artifact repositories
    Easier interaction with hosts
    62 . 1
    

    View Slide

64. THANKS!
    THANKS!
    Questions?
    We are hiring!
    Try Exoscale out!
    on twitter and github
    @pyr
    63 . 1
    

    View Slide

65. 64 . 1
    

    View Slide