Tweet
Tweet

Slide 1

Slide 1 text

Common pitfalls in Jenkins security configuration… and how to avoid them Oleg Nenashev

Slide 2

Slide 2 text

© 2018 All Rights Reserved. 2 About me @oleg_nenashev oleg-nenashev ● Jenkins contributor since 2012 ● Jenkins Core maintainer ● Jenkins Ambassador ● CloudBees Core ● CloudBees Jenkins Support

Slide 3

Slide 3 text

© 2018 All Rights Reserved. 3 About me. Jenkins Security • Jenkins Security Team member since 2015 • Maintainer of Role Strategy, Job Restrictions and Ownership plugins • Maintained JEP-200 • Reported and fixed some defects • Interests: Static analysis, security scans @oleg_nenashev oleg-nenashev

Slide 4

Slide 4 text

© 2018 All Rights Reserved. 4 About you

Slide 5

Slide 5 text

© 2018 All Rights Reserved. 5 Do you think that your Jenkins instance is secure?

Slide 6

Slide 6 text

© 2018 All Rights Reserved. 6

Slide 7

Slide 7 text

© 2018 All Rights Reserved. 7 Disclaimer • The talk is based on the public information ONLY • Many Jenkins instances were harmed by the issues in the talk • Neither the presenter nor the Jenkins project are responsible for any impact on the instance

Slide 8

Slide 8 text

© 2018 All Rights Reserved. 8 ● CVE-2017-1000353 ● Fixed and announced in April 2017 ● Jenkins 2.46.2+ ● Still being exploited in the wild 1 year after https://research.checkpoint.com/jenkins-miner-one-biggest-mining-operations-ever-discovered/

Slide 9

Slide 9 text

© 2018 All Rights Reserved. 9

Slide 10

Slide 10 text

© 2018 All Rights Reserved. 10 Jenkins is a… remote execution engine (by design)

Slide 11

Slide 11 text

© 2018 All Rights Reserved. 11 Jenkins is a… remote execution engine (by design) • One can run code and system commands • Access to master system • Access to agents • Access to private/public clouds

Slide 12

Slide 12 text

© 2018 All Rights Reserved. 12 Jenkins… has access to sensitive data (by design)

Slide 13

Slide 13 text

© 2018 All Rights Reserved. 13 Jenkins… has access to sensitive data (by design) • Credentials • Private repositories • Artifacts, including release ones

Slide 14

Slide 14 text

© 2018 All Rights Reserved. 14 Jenkins is a… service (by design)

Slide 15

Slide 15 text

© 2018 All Rights Reserved. 15 Jenkins is a… service (by design) • Multiple users • Different expertise • Users may misuse permissions

Slide 16

Slide 16 text

© 2018 All Rights Reserved. 16 What does security mean? Jenkins security Intrusion and data theft protection Restrictions within organization

Slide 17

Slide 17 text

© 2018 All Rights Reserved. 17 Jenkins security Intrusion and data theft protection Restrictions within organization • Must-have in internet-facing instances • Paranoid mode is fine What does security mean?

Slide 18

Slide 18 text

© 2018 All Rights Reserved. 18 Jenkins security Intrusion and data theft protection Restrictions within organization • Better user experience • Protection from unintentional actions • Protection from lack of expertise What does security mean?

Slide 19

Slide 19 text

© 2018 All Rights Reserved. 19 “Should I care about security?” • YES – public-facing instances • YES – Projects and source code restricted inside organization • YES – Projects use external infra: update centers, DockerHub, etc. • ? - otherwise

Slide 20

Slide 20 text

© 2018 All Rights Reserved. 20 • Jenkins core and plugins • Security Officer: Daniel Beck • Security team members • https://jenkins.io/security/ Jenkins Security Team

Slide 21

Slide 21 text

© 2018 All Rights Reserved. 21 Security process under the hood • Private process for vulnerabilities • Restricted SECURITY project in Jenkins JIRA • Private repositories for fix development and reviews • Staged and pre-tested releases https://jenkins.io/security/

Slide 22

Slide 22 text

© 2018 All Rights Reserved. 22 Security process under the hood • Private process for vulnerabilities • Restricted SECURITY project in Jenkins JIRA • Private repositories for fix development and reviews • Staged and pre-tested releases • Coordination of releases with stakeholders • Cooperation with plugin maintainers to deliver fixes • Cooperation with maintainers of upstream components https://jenkins.io/security/

Slide 23

Slide 23 text

© 2018 All Rights Reserved. 23 https://jenkins.io/security/advisories/

Slide 24

Slide 24 text

© 2018 All Rights Reserved. 24 Security advisories Source: status update by Daniel Beck at the Jenkins Contributor Summit, Sep 17, 2018

Slide 25

Slide 25 text

© 2018 All Rights Reserved. 25 Severity of fixed issues Source: status update by Daniel Beck at the Jenkins Contributor Summit, Sep 17, 2018

Slide 26

Slide 26 text

Common pitfalls Organizational issues

Slide 27

Slide 27 text

© 2018 All Rights Reserved. 27 Not subscribing to security advisory notifications

Slide 28

Slide 28 text

© 2018 All Rights Reserved. 28 https://jenkins.io/security/advisories/

Slide 29

Slide 29 text

© 2018 All Rights Reserved. 29 Security advisories • It’s often easy to create an exploit based on advisory and the code • Instances may become vulnerable after public advisory/disclosure • We notify about security releases with 1-week advance • Admins can subscribe to notifications • https://groups.google.com/forum/#!forum/jenkinsci-advisories • https://feeds.feedburner.com/jenkins-security-advisories

Slide 30

Slide 30 text

© 2018 All Rights Reserved. 30 Not updating

Slide 31

Slide 31 text

© 2018 All Rights Reserved. 31 Keep Updating! • Frequent security releases •Weekly •Current LTS baseline • Security hardening in common releases 2.121.2. You might want to update ASAP

Slide 32

Slide 32 text

© 2018 All Rights Reserved. 32 Using insecure plugins

Slide 33

Slide 33 text

© 2018 All Rights Reserved. 33 Scriptocalypse https://jenkins.io/security/advisory/2017-04-10/ • Unlimited scripting • More than 30 plugins affected •Groovy Plugin •JobDSL Plugin •Grails Plugin •Scriptler Plugin • Some of them are blocked

Slide 34

Slide 34 text

© 2018 All Rights Reserved. 34 Fun facts – Plugins with Remote Code Execution Plugin Advisory date Number of installations* Build Flow Plugin April 10, 2017 6504 (-40%) Scriptler (last official release) April 10, 2017 8656 (-25%) Liquibase Runner Mar 03, 2018 278 (-10%) . . . * Stats from Aug 2018 ● It takes a while to get fixed actually installed ● The most of users - old instances not facing public ● Decline on new installations with recent Jenkins core

Slide 35

Slide 35 text

© 2018 All Rights Reserved. 35 Relying on administrative monitor • It is an advisory tool only • The instance may be vulnerable if you see the notifications • Custom update centers may not ship security warnings • Remote Code Execution exploits can disable notifications

Slide 36

Slide 36 text

© 2018 All Rights Reserved. 36 Mis-reporting security issues

Slide 37

Slide 37 text

© 2018 All Rights Reserved. 37

Slide 38

Slide 38 text

© 2018 All Rights Reserved. 38 JIRA, project=’JENKINS’

Slide 39

Slide 39 text

© 2018 All Rights Reserved. 39 SECURITY project ● https://jenkins.io/security/#reporting-vulnerabilities ● Rewards for proper reports

Slide 40

Slide 40 text

Infrastructure management Common Pitfalls

Slide 41

Slide 41 text

© 2018 All Rights Reserved. 41 I have a private instance, I am fine

Slide 42

Slide 42 text

© 2018 All Rights Reserved. 42 Web Interface is NOT the only way to get into a connected system

Slide 43

Slide 43 text

© 2018 All Rights Reserved. 43 Do you pull latest images from DockerHub?

Slide 44

Slide 44 text

© 2018 All Rights Reserved. 44 Do you pull latest images from DockerHub?

Slide 45

Slide 45 text

© 2018 All Rights Reserved. 45 •What’s inside? •Who can change them? •What if there is a malicious code? Do you pull latest images from DockerHub?

Slide 46

Slide 46 text

© 2018 All Rights Reserved. 46 Are Jenkins update centers different?

Slide 47

Slide 47 text

© 2018 All Rights Reserved. 47 They are not. Know what you use Monitor plugin versions and release notes • Beware of transient dependencies (!) • Also monitor JIRA Consider using locally managed sources • Internal Maven, Docker Registry • Custom Jenkins Update Center: Juseppe • https://github.com/yandex-qatools/juseppe Use static configurations • Configuration-as-Code

Slide 48

Slide 48 text

© 2018 All Rights Reserved. 48 Exposing Jenkins filesystem • JENKINS_HOME snapshot is enough to recreate the system • … and to steal credentials

Slide 49

Slide 49 text

© 2018 All Rights Reserved. 49 Master Filesystem • All secrets and credentials within JENKINS_HOME are encrypted • Encryption is powered by secret key files • Secret keys are also stored in JENKINS_HOME/secrets • Snapshot of JENKINS_HOME is enough to get all secrets in Jenkins • External credentials might help

Slide 50

Slide 50 text

© 2018 All Rights Reserved. 50 Read-only backups same as above...

Slide 51

Slide 51 text

© 2018 All Rights Reserved. 51 Running master or agent under privileged accounts

Slide 52

Slide 52 text

© 2018 All Rights Reserved. 52 Running master or agent under privileged accounts • Direct access to the system (e.g. from scripts) • Environment can be altered: registry, system processes, filesystem, etc., etc. • Access to other accounts

Slide 53

Slide 53 text

© 2018 All Rights Reserved. 53 Keep Jenkins in a sandbox Do not run masters/agents under system accounts Restrict access to non-required resources • Generic accounts • Read-only repositories Sandbox your scripts if possible

Slide 54

Slide 54 text

© 2018 All Rights Reserved. 54 Tips & Tricks • Single-shot agents • No environment pollution • Limit the infrastructure impact

Slide 55

Slide 55 text

© 2018 All Rights Reserved. 55 No disaster recovery plan

Slide 56

Slide 56 text

© 2018 All Rights Reserved. 56 No disaster recovery plan • Some exploits may corrupt the instance • Remote Code Execution • Filesystem access

Slide 57

Slide 57 text

© 2018 All Rights Reserved. 57 No disaster recovery plan • Some exploits may corrupt the instance • Remote Code Execution • Filesystem access • The Jenkins host may be compromised • Other infrastructure may be compromised if misconfigured

Slide 58

Slide 58 text

© 2018 All Rights Reserved. 58 No disaster recovery plan • Some exploits may corrupt the instance • Remote Code Execution • Filesystem access • The Jenkins host may be compromised • Other infrastructure may be compromised if misconfigured • It should be possible to rebuild the system from scratch

Slide 59

Slide 59 text

© 2018 All Rights Reserved. 59 Configuration as Code in Jenkins Jobs System Configurations

Slide 60

Slide 60 text

© 2018 All Rights Reserved. 60 System Configuration as Code Just examples… External tools Jenkins CLI and REST API python-jenkins jenkins-client (java) Configuration Management Ansible, Chef, … Docker, Docker Compose ... Solutions in Jenkins Groovy Boot Hooks JCasC . . . SCM Sync Configuration

Slide 61

Slide 61 text

© 2018 All Rights Reserved. 61 System Configuration as Code 1. Docker image: plugins.txt, Java flags, etc. 2. Groovy Init Scripts ● Overview: http://bit.ly/jenkins-groovy-hooks 3. OR: New Configuration-as-Code Plugin ● https://plugins.jenkins.io/configuration-as-code ● Allows configuring Jenkins from YAML

Slide 62

Slide 62 text

Common pitfalls Configuration issues

Slide 63

Slide 63 text

© 2018 All Rights Reserved. 63 Not configuring security in configuration-as-code

Slide 64

Slide 64 text

© 2018 All Rights Reserved. 64 Security & Configuration-as-Code • Security is configured by the Installation Wizard on first startup • Configuration-as-Code logic usually disables the Installation Wizard • “-DrunSetupWizard=false” flag in containers • JCasC skips installation wizard if config is passed • Admin is responsible to set defaults

Slide 65

Slide 65 text

© 2018 All Rights Reserved. 65 Security & Configuration-as-Code Must configure: • Authentication • Authorization jenkins: securityRealm: local: allowsSignup: false enableCaptcha: false # Users are configured elsewhere authorizationStrategy: roleBased: roles: global: - name: "admin" description: "Jenkins administrators" permissions: - "Overall/Administer" assignments: - "admin" - name: "readonly" ... jenkins.yaml https://github.com/oleg-nenashev/demo-j enkins-config-as-code/tree/casc-plugin

Slide 66

Slide 66 text

© 2018 All Rights Reserved. 66 Security & Configuration-as-Code Must configure: • Disable Remoting CLI • CSRF Protection • Slave-to-master security • Remoting protocols (before 2.138.1) jenkins: agentProtocols: - "JNLP4-connect" - "Ping" crumbIssuer: standard: excludeClientIPFromCrumb: true remotingSecurity: enabled: true security: remotingCLI: enabled: false https://github.com/oleg-nenashev/demo-jenkins-config-as-code/ jenkins.yaml

Slide 67

Slide 67 text

© 2018 All Rights Reserved. 67 Same with Groovy import hudson.security.csrf.DefaultCrumbIssuer import jenkins.model.Jenkins import jenkins.CLI import jenkins.security.s2m.AdminWhitelistRule import org.kohsuke.stapler.StaplerProxy CLI.get().enabled = false Jenkins.instance.agentProtocols = new HashSet(["JNLP4-connect"]) Jenkins.instance.getExtensionList(StaplerProxy.class) .get(AdminWhitelistRule.class) .masterKillSwitch = false if (Jenkins.instance.crumbIssuer == null) { println "CSRF protection is disabled, Enabling the default Crumb Issuer" Jenkins.instance.crumbIssuer = new DefaultCrumbIssuer(true) } https://github.com/oleg-nenashev/demo-jenkins-config-as-code/

Slide 68

Slide 68 text

© 2018 All Rights Reserved. 68 Disabling security features

Slide 69

Slide 69 text

© 2018 All Rights Reserved. 69 Disabling security features • Blacklisted classes in Class Deserialization • Blacklisted methods in Script Security • Agent-2-master security • Content Security Policy • Markup Editor • ...

Slide 70

Slide 70 text

© 2018 All Rights Reserved. 70 Permitting builds on the master

Slide 71

Slide 71 text

© 2018 All Rights Reserved. 71 Builds on the master • Builds have access to the master filesystem • Builds run under the Jenkins account • They can… • Read data from other builds/artifacts • Read secret hashes • Modify Jenkins system configuration • … • You don’t want that in 99% of cases

Slide 72

Slide 72 text

© 2018 All Rights Reserved. 72 Restricting builds on the master • Solution 1: • Set “0” executors on master • Another node running under different account • BUT: Fly-weight tasks are permitted

Slide 73

Slide 73 text

© 2018 All Rights Reserved. 73 Restricting builds on the master • Solution 1: • 0 executors on master • Another node running under different account • BUT: Fly-weight tasks are permitted • Solution 2: If you need executors on master • Job Restrictions Plugin • https://plugins.jenkins.io/job-restrictions

Slide 74

Slide 74 text

© 2018 All Rights Reserved. 74 Job Restrictions. Protecting the Master node • NEVER let users run jobs on master • Only use it for system jobs owned by admins

Slide 75

Slide 75 text

© 2018 All Rights Reserved. 75 Hint: Agent.Build permission Example for Ownership-Based security: http://bit.ly/ownership-based-security Available in Role Strategy Plugin, Matrix Authorization, CloudBees RBAC

Slide 76

Slide 76 text

© 2018 All Rights Reserved. 76 Not restricting Jenkins queue

Slide 77

Slide 77 text

© 2018 All Rights Reserved. 77 Not restricting Jenkins queue • By default Queue tasks run with the System account

Slide 78

Slide 78 text

© 2018 All Rights Reserved. 78 Not restricting Jenkins queue • By default Queue tasks run with the System account • Users may trigger ANY builds • Parameterized Trigger, Pipeline steps

Slide 79

Slide 79 text

© 2018 All Rights Reserved. 79 Not restricting Jenkins queue • By default Queue tasks run with the System account • Users may trigger ANY builds • Parameterized Trigger, Pipeline steps • Users can extract data from other builds • Copy Artifacts - artifacts and workspaces

Slide 80

Slide 80 text

© 2018 All Rights Reserved. 80 Not restricting Jenkins queue • By default Queue tasks run with the System account • Users may trigger ANY builds • Parameterized Trigger, Pipeline steps • Users can extract data from other builds • Copy Artifacts - artifacts and workspaces • Computer.Build is always on • Task can run on any agent

Slide 81

Slide 81 text

© 2018 All Rights Reserved. 81 Authorize Project Plugin Authorize builds • Global default • Whitelist of user-configurable strategies • Job properties https://plugins.jenkins.io/authorize-project

Slide 82

Slide 82 text

© 2018 All Rights Reserved. 82 Granting EXTENDED_READ to non-privileged users

Slide 83

Slide 83 text

© 2018 All Rights Reserved. 83 Granting EXTENDED_READ to non-privileged users • config.xml may include sensitive data • Mistakes by job developers • Security defects in plugins (e.g. plaintext passwords) • The permission is disabled for a reason

Slide 84

Slide 84 text

Summary

Slide 85

Slide 85 text

© 2018 All Rights Reserved. 85 Security advisories Source: status update by Daniel Beck at the Jenkins Contributor Summit, Sep 17, 2018

Slide 86

Slide 86 text

© 2018 All Rights Reserved. 86 Security advisories Source: status update by Daniel Beck at the Jenkins Contributor Summit, Sep 17, 2018 o_O

Slide 87

Slide 87 text

© 2018 All Rights Reserved. 87 •It’s not! •All fixes go to advisories •Technical debt cleaning for 1500+ components •50% of fixes - plugins installed on 3% instances or less FAQ: Is Jenkins security that bad?

Slide 88

Slide 88 text

© 2018 All Rights Reserved. 88 What’s next? • Continuous effort on improving security • Security cleanup in “long-tail” plugins • “Secure by default” in Cloud Native Jenkins • https://jenkins.io/blog/2018/08/31/shifting-gears/

Slide 89

Slide 89 text

© 2018 All Rights Reserved. 89 Do you think that your Jenkins instance is secure?

Slide 90

Slide 90 text

© 2018 All Rights Reserved. 90 Ask yourself ❑ Are you subscribed to Jenkins security advisories? ❑ Do you keep your Jenkins up to date? ❑ Is JENKINS_HOME or backups accessible? ❑ Do you initialize security in Configuration-as-code? ❑ Do you restrict jobs on the master? ❑ Do you use Authorize Project to restrict the queue?

Slide 91

Slide 91 text

© 2018 All Rights Reserved. 91 1. Follow the security advisories 2. Keep your Jenkins instances up to date 3. Check your Jenkins infrastructure and configuration 4. Be careful when using niche plugins 5. Try Configuration-as-code Takeaways

Slide 92

Slide 92 text

© 2018 All Rights Reserved. 92 •Security page: https://jenkins.io/security/ •Advisories: https://jenkins.io/security/advisories/ •Ownership-based security: http://bit.ly/ownership-based-security •Configurations-as-Code demo (see branches): https://hub.docker.com/r/onenashev/demo-jenkins-config-as-code/ •My previous talk about security best practices: https://speakerdeck.com/onenashev/spb-jenkins-meetup-number-9-managing-sec urity-in-jenkins-eng Links

Slide 93

Slide 93 text

© 2018 All Rights Reserved. 93 Thank you! Contacts: E-mail: onenashev@cloudbees.com GitHub: oleg-nenashev Twitter: @oleg_nenashev