Taming Jenkins - Speaker Deck

Slide 1

Slide 1 text

TAMING JENKINS TAMING JENKINS BUILDING A CLOUD-NATIVE CI ENVIRONMENT BUILDING A CLOUD-NATIVE CI ENVIRONMENT 1

Slide 2

Slide 2 text

2 . 1

Slide 3

Slide 3 text

HALLO HALLO : Three-line Bio CTO & Co-founder at Distributed systems and monitoring enthusiast Open-Source developer Clojure Libraries, OpenBSD, Riemann, Collectd, and more. @pyr Exoscale 3 . 1

Slide 4

Slide 4 text

TAMING JENKINS TAMING JENKINS Building a Cloud-Native CI environment 4 . 1

Slide 5

Slide 5 text

EXOSCALE EXOSCALE Infrastructure as a service Part of A1 Digital Zones in Frankfurt, Vienna, Zürich, Geneva 5 . 1

Slide 6

Slide 6 text

EXOSCALE EXOSCALE 6 . 1

Slide 7

Slide 7 text

EXOSCALE EXOSCALE provider "exoscale" { api_key = "${var.exoscale_api_key}" secret_key = "${var.exoscale_secret_key}" } resource "exoscale_instance" "web" { template = "Ubuntu 17.04" disk_size = "50g" profile = "medium" ssh_key = "production" } 7 . 1

Slide 8

Slide 8 text

I THOUGHT THIS WAS A DEV I THOUGHT THIS WAS A DEV CONFERENCE! CONFERENCE! 8 . 1

Slide 9

Slide 9 text

WHAT'S IN A CLOUD PROVIDER WHAT'S IN A CLOUD PROVIDER Datacenter operations So ware development 9 . 1

Slide 10

Slide 10 text

SOFTWARE AT EXOSCALE SOFTWARE AT EXOSCALE Object storage controller Network controller Internal SDN Customer management Metering system Billing Web portal 10 . 1

Slide 11

Slide 11 text

LANGUAGES AT EXOSCALE LANGUAGES AT EXOSCALE C & Go Clojure Python ClojureScript & JS 11 . 1

Slide 12

Slide 12 text

THE BUILD FACTORY THE BUILD FACTORY From code to artifact From artifact to deploy 12 . 1

Slide 13

Slide 13 text

BUILD FACTORY BUILD FACTORY 13 . 1

Slide 14

Slide 14 text

MAIN DESIGN IDEA MAIN DESIGN IDEA Security Reproducibility Observability Checkpoints Cloud-Native 14 . 1

Slide 15

Slide 15 text

SECURITY SECURITY No code download on production hosts Signed packages 15 . 1

Slide 16

Slide 16 text

REPRODUCIBILITY REPRODUCIBILITY Building once (and in chroots) ensures clean packages Reproducible builds make wide changes easier We need staging deploys and production deploys to be identical 16 . 1

Slide 17

Slide 17 text

OBSERVABILITY OBSERVABILITY When did we last build this? What did the output look like? What commit did it correspond to? 17 . 1

Slide 18

Slide 18 text

CHECKPOINTS CHECKPOINTS CD is great for test and staging We are wary of unattended production deploys There should be a clear (but simple) trigger 18 . 1

Slide 19

Slide 19 text

CLOUD-NATIVE CLOUD-NATIVE Configuration as code Fast infrastructure set-up and teardown No decay over time It should be easy to grow or shrink the factory's throughput 19 . 1

Slide 20

Slide 20 text

BUILD FACTORY BUILD FACTORY 20 . 1

Slide 21

Slide 21 text

BUILD FACTORY BUILD FACTORY 21 . 1

Slide 22

Slide 22 text

BUILD FACTORY BUILD FACTORY 22 . 1

Slide 23

Slide 23 text

BUILD FACTORY BUILD FACTORY 23 . 1

Slide 24

Slide 24 text

BUILD FACTORY BUILD FACTORY 24 . 1

Slide 25

Slide 25 text

BUILD FACTORY BUILD FACTORY 25 . 1

Slide 26

Slide 26 text

BUILD FACTORY BUILD FACTORY 26 . 1

Slide 27

Slide 27 text

BUILD FACTORY BUILD FACTORY 27 . 1

Slide 28

Slide 28 text

BUILD FACTORY BUILD FACTORY 28 . 1

Slide 29

Slide 29 text

ARE WE THERE YET? ARE WE THERE YET? 29 . 1

Slide 30

Slide 30 text

MOST OF IT IS FINE MOST OF IT IS FINE But legacy lurks in every environment. 30 . 1

Slide 31

Slide 31 text

THE CHECKLIST THE CHECKLIST Security ✔ Reproducibility ✔ Observability ✔ Checkpoints ✔ Cloud-Native ❌ 31 . 1

Slide 32

Slide 32 text

OUR JENKINS OUR JENKINS 32 . 1

Slide 33

Slide 33 text

OUR JENKINS OUR JENKINS A single machine, builds close to configuration Jenkins user has sudo access Which plugins are installed? How is config restored? 33 . 1

Slide 34

Slide 34 text

HOW DID WE GET HERE? HOW DID WE GET HERE? Jenkins makes automation hard Password in logs No clear configuration file standard 34 . 1

Slide 35

Slide 35 text

HOW DID WE GET HERE? HOW DID WE GET HERE? week in review false https://github.com/exoscale/wir/

Slide 36

Slide 36 text

XML-SERIALIZED OBJECTS XML-SERIALIZED OBJECTS

Slide 37

Slide 37 text

XML-SERIALIZED OBJECTS XML-SERIALIZED OBJECTS Lengthy descriptions No standard text to config Too many combinations to template 37 . 1

Slide 38

Slide 38 text

NO STANDARD HOOK MECHANISM NO STANDARD HOOK MECHANISM All job configuration is in-situ No way to externally say: run IRC notifications for all these jobs Even fewer avenues for templating 38 . 1

Slide 39

Slide 39 text

FEW MITIGATION STRATEGIES FEW MITIGATION STRATEGIES Jobs are edited through the web Backups to git as primary recovery mechanism 39 . 1

Slide 40

Slide 40 text

DIRECT CONSEQUENCE DIRECT CONSEQUENCE Jenkins down means no deploy Time to recovery: ??? 40 . 1

Slide 41

Slide 41 text

BE GENTLE BE GENTLE This was all started in 2012 41 . 1

Slide 42

Slide 42 text

JENKINS AUTOMATION, CIRCA 2012 JENKINS AUTOMATION, CIRCA 2012 42 . 1

Slide 43

Slide 43 text

JENKINS IN 2017: A NEW HOPE JENKINS IN 2017: A NEW HOPE Pipeline Jobs Job DSL Init via groovy 43 . 1

Slide 44

Slide 44 text

OUR IMPLEMENTATION OUR IMPLEMENTATION The road to a cloud-native CI environment 44 . 1

Slide 45

Slide 45 text

PIPELINE JOBS PIPELINE JOBS node { stage('build') { checkout scm sh 'lein compile :all' sh 'lein uberjar' archiveArtifacts artifacts: '**/target/*.jar', fingerprint: true } stage('test') { sh 'lein test' } } 45 . 1

Slide 46

Slide 46 text

PIPELINE JOBS PIPELINE JOBS . ├── Dockerfile ├── Jenkinsfile ├── project.clj ├── src │ └── foo │ └── core.clj └── test └── foo └── core_test.clj 4 directories, 5 files 46 . 1

Slide 47

Slide 47 text

PIPELINE JOBS PIPELINE JOBS Keeps build information close to code No impedance mismatch between job and code Easier to work in branches 47 . 1

Slide 48

Slide 48 text

JOB DSL JOB DSL def githubPipelineJob(org, name, private_repo=true) { pipelineJob name { definition { cpsScm { scriptPath 'JenkinsFile' git { remote { github "$org/$name" if (private_repo) { credential 'github-token' } } } } } } } githubPipelineJob('exoscale', 'portal') githubPipelineJob('exoscale', 'api') ... 48 . 1

Slide 49

Slide 49 text

JOB DSL JOB DSL A single job generates all jobs Updates / Removals when job list changes All Jenkins jobs can be expressed 49 . 1

Slide 50

Slide 50 text

REGISTERING BUILD HOSTS REGISTERING BUILD HOSTS Most plugins require some form of manual intervention Most are operated from the master 50 . 1

Slide 51

Slide 51 text

REGISTERING BUILD HOSTS REGISTERING BUILD HOSTS Reverse engineered API Private SSH key deployed on the master As a named credential Public SSH key deployed on host 51 . 1

Slide 52

Slide 52 text

THE LAST MISSING PIECE THE LAST MISSING PIECE Still need to bootstrap Jenkins Still need to provision additional configuration Plugins Perms Credentials 52 . 1

Slide 53

Slide 53 text

GROOVY INITIALIZATION SUPPORT GROOVY INITIALIZATION SUPPORT Code in $JENKINS_HOME/init.groovy.d/ gets picked up All Jenkins functionality can be accessed 53 . 1

Slide 54

Slide 54 text

LIVING THE DREAM LIVING THE DREAM executors: 0 url: "http://jenkins/" seedjob: git: "https://github.com/exoscale/jobs.git" name: "seed-job" credentials: "github-token" credentials: ssh-key: type: PRIVATE_KEY username: ubuntu description: "Private key for host SSH access" key: | ... github-token: type: USER_PASSWORD username: build-bot password: "..." description: "Github autogenerated token credentials for builds" perms: admin: password: '...' allowed: [ 'JENKINS_ADMINISTER' ] builder: password: '...' allowed: - 'JENKINS_READ'

Slide 55

Slide 55 text

- 'ITEM_BUILD' - 'ITEM_READ' - ... worker: password: '...' allowed: - 'JENKINS_READ' - 'COMPUTER_EXTENDED_READ' - 'COMPUTER_CONNECT' - ... 54 . 1

Slide 56

Slide 56 text

LIVING THE DREAM LIVING THE DREAM ant build-timeout email-ext extended-read-permission github-branch-source gradle job-dsl workflow-aggregator pipeline-github-lib ssh-slaves timestamper ws-cleanup blueocean matrix-auth authorize-project 55 . 1

Slide 57

Slide 57 text

GROOVY INITIAL SETUP GROOVY INITIAL SETUP def realm = new HudsonPrivateSecurityRealm(false) def strategy = new FullControlOnceLoggedInAuthorizationStrategy() realm.createAccount('admin', 'TemporaryLongPassword') jenkins.setSecurityRealm(realm) jenkins.setAuthorizationStrategy(strategy) jenkins.getDescriptor("jenkins.CLI").get().setEnabled(false) jenkins.save() 56 . 1

Slide 58

Slide 58 text

GROOVY PLUGIN PROVISIONING GROOVY PLUGIN PROVISIONING def path = System.getenv("JENKINS_PLUGIN_CONFIG") ?: "/etc/jenkins/plugins.list" plugins = new File(path).readLines() def pm = jenkins.getPluginManager() def uc = jenkins.getUpdateCenter() def install_list = plugins.findAll{ !pm.getPlugin(it)} install_list.each { def plugin = uc.getPlugin(it) if (plugin) { def ftr = plugin.deploy() ftr.get(1, TimeUnit.MINUTES) } } jenkins.save() if (install_list) { jenkins.restart() } 57 . 1

Slide 59

Slide 59 text

GROOVY YAML LOAD GROOVY YAML LOAD @Grab('org.yaml:snakeyaml:1.18') def path = System.getenv("JENKINS_YAML_CONFIG") ?: "/etc/jenkins/jenkins.yaml" config = new Yaml().load(new File(path).getText()) 58 . 1

Slide 60

Slide 60 text

GROOVY JOB DSL BOOTSTRAP GROOVY JOB DSL BOOTSTRAP dslScript = new ExecuteDslScripts(null) dslScript.setTargets("jobs.groovy") dslScript.setLookupStrategy(LookupStrategy.JENKINS_ROOT) remotes = GitSCM.createRepoList(config.seedjob.git, config.seedjob.credentials) branches = [new BranchSpec("*/master")] browser = new GithubWeb(config.seedjob.git) dslScm = new GitSCM(remotes, branches, false, [], browser, "git", []) job = new hudson.model.FreeStyleProject(jenkins, config.seedjob.name) job.scm = dslScm job.getPublishersList().add(dslScript); jenkins.add(job, config.seedjob.name) 59 . 1

Slide 61

Slide 61 text

ARE WE THERE YET? ARE WE THERE YET? 60 . 1

Slide 62

Slide 62 text

MUCH CLOSER MUCH CLOSER 61 . 1

Slide 63

Slide 63 text

LOOKING FORWARD LOOKING FORWARD Learning a bit more Groovy :-) Less dependencies on build hosts More artifact repositories Easier interaction with hosts 62 . 1

Slide 64

Slide 64 text

THANKS! THANKS! Questions? We are hiring! Try Exoscale out! on twitter and github @pyr 63 . 1

Slide 65

Slide 65 text

64 . 1