Common pitfalls in Jenkins security configuration… and how to avoid them

Jenkins, an open source automation server with 15 years of history, still dominates the CI/CD market and remains a mission-critical part of the worldwide software supply chain. IBut Jenkins still has quite an old architecture, and one can consider it a remote execution engine which has access to project data… by design. So, how secure is your instance and supply chain when using Jenkins?

When it comes to Jenkins instances with thousands of users, it is hard to retain status quo between security itself and its impact on user experience due to the restrictions and performance degradation. I will talk about Jenkins security model, best practices and common non-newbie configuration mistakes which we often see on production instances. In particular we will discuss Groovy scripting, controller-to-agent communications and resource isolation. We will also review a few supply chain attack cases that happened through Jenkins, and how one could prevent them.

Target audience for this talk: experienced Jenkins administrators and users interested in Security. The learnings are applicable to other CI/CD tools, especially the self-hosted ones

Transcript

1. Common pitfalls in Jenkins security configuration… and how to avoid them Oleg Nenashev
2. © 2018 All Rights Reserved. 2 About me @oleg_nenashev oleg-nenashev • Jenkins contributor since 2012 • Jenkins Core maintainer • Jenkins Ambassador • CloudBees Core • CloudBees Jenkins Support
3. © 2018 All Rights Reserved. 3 About me. Jenkins Security • Jenkins Security Team member since 2015 • Maintainer of Role Strategy, Job Restrictions and Ownership plugins • Maintained JEP-200 • Reported and fixed some defects • Interests: Static analysis, security scans @oleg_nenashev oleg-nenashev
4. © 2018 All Rights Reserved. 4 About you
5. © 2018 All Rights Reserved. 5 Do you think that your Jenkins instance is secure?
6. © 2018 All Rights Reserved. 6
7. © 2018 All Rights Reserved. 7 Disclaimer • The talk is based on the public information ONLY • Many Jenkins instances were harmed by the issues in the talk • Neither the presenter nor the Jenkins project are responsible for any impact on the instance
8. © 2018 All Rights Reserved. 8 • CVE-2017-1000353 • Fixed and announced in April 2017 • Jenkins 2.46.2+ • Still being exploited in the wild 1 year after https://research.checkpoint.com/jenkins-miner-one-biggest-mining-operations-ever-discovered/
9. © 2018 All Rights Reserved. 9
10. © 2018 All Rights Reserved. 10 Jenkins is a… remote execution engine (by design)
11. © 2018 All Rights Reserved. 11 Jenkins is a… remote execution engine (by design) • One can run code and system commands • Access to master system • Access to agents • Access to private/public clouds
12. © 2018 All Rights Reserved. 12 Jenkins… has access to sensitive data (by design)
13. © 2018 All Rights Reserved. 13 Jenkins… has access to sensitive data (by design) • Credentials • Private repositories • Artifacts, including release ones
14. © 2018 All Rights Reserved. 14 Jenkins is a… service (by design)
15. © 2018 All Rights Reserved. 15 Jenkins is a… service (by design) • Multiple users • Different expertise • Users may misuse permissions
16. © 2018 All Rights Reserved. 16 What does security mean? Jenkins security Intrusion and data theft protection Restrictions within organization
17. © 2018 All Rights Reserved. 17 Jenkins security Intrusion and data theft protection Restrictions within organization • Must-have in internet-facing instances • Paranoid mode is fine What does security mean?
18. © 2018 All Rights Reserved. 18 Jenkins security Intrusion and data theft protection Restrictions within organization • Better user experience • Protection from unintentional actions • Protection from lack of expertise What does security mean?
19. © 2018 All Rights Reserved. 19 “Should I care about security?” • YES – public-facing instances • YES – Projects and source code restricted inside organization • YES – Projects use external infra: update centers, DockerHub, etc. • ? - otherwise
20. © 2018 All Rights Reserved. 20 • Jenkins core and plugins • Security Officer: Daniel Beck • Security team members • https://jenkins.io/security/ Jenkins Security Team
21. © 2018 All Rights Reserved. 21 Security process under the hood • Private process for vulnerabilities • Restricted SECURITY project in Jenkins JIRA • Private repositories for fix development and reviews • Staged and pre-tested releases https://jenkins.io/security/
22. © 2018 All Rights Reserved. 22 Security process under the hood • Private process for vulnerabilities • Restricted SECURITY project in Jenkins JIRA • Private repositories for fix development and reviews • Staged and pre-tested releases • Coordination of releases with stakeholders • Cooperation with plugin maintainers to deliver fixes • Cooperation with maintainers of upstream components https://jenkins.io/security/
23. © 2018 All Rights Reserved. 23 https://jenkins.io/security/advisories/
24. © 2018 All Rights Reserved. 24 Security advisories Source: status update by Daniel Beck at the Jenkins Contributor Summit, Sep 17, 2018
25. © 2018 All Rights Reserved. 25 Severity of fixed issues Source: status update by Daniel Beck at the Jenkins Contributor Summit, Sep 17, 2018
26. Common pitfalls Organizational issues
27. © 2018 All Rights Reserved. 27 Not subscribing to security advisory notifications
28. © 2018 All Rights Reserved. 28 https://jenkins.io/security/advisories/
29. © 2018 All Rights Reserved. 29 Security advisories • It’s often easy to create an exploit based on advisory and the code • Instances may become vulnerable after public advisory/disclosure • We notify about security releases with 1-week advance • Admins can subscribe to notifications • https://groups.google.com/forum/#!forum/jenkinsci-advisories • https://feeds.feedburner.com/jenkins-security-advisories
30. © 2018 All Rights Reserved. 30 Not updating
31. © 2018 All Rights Reserved. 31 Keep Updating! • Frequent security releases •Weekly •Current LTS baseline • Security hardening in common releases 2.121.2. You might want to update ASAP
32. © 2018 All Rights Reserved. 32 Using insecure plugins
33. © 2018 All Rights Reserved. 33 Scriptocalypse https://jenkins.io/security/advisory/2017-04-10/ • Unlimited scripting • More than 30 plugins affected •Groovy Plugin •JobDSL Plugin •Grails Plugin •Scriptler Plugin • Some of them are blocked
34. © 2018 All Rights Reserved. 34 Fun facts – Plugins with Remote Code Execution Plugin Advisory date Number of installations* Build Flow Plugin April 10, 2017 6504 (-40%) Scriptler (last official release) April 10, 2017 8656 (-25%) Liquibase Runner Mar 03, 2018 278 (-10%) . . . * Stats from Aug 2018 • It takes a while to get fixed actually installed • The most of users - old instances not facing public • Decline on new installations with recent Jenkins core
35. © 2018 All Rights Reserved. 35 Relying on administrative monitor • It is an advisory tool only • The instance may be vulnerable if you see the notifications • Custom update centers may not ship security warnings • Remote Code Execution exploits can disable notifications
36. © 2018 All Rights Reserved. 36 Mis-reporting security issues
37. © 2018 All Rights Reserved. 37
38. © 2018 All Rights Reserved. 38 JIRA, project=’JENKINS’
39. © 2018 All Rights Reserved. 39 SECURITY project • https://jenkins.io/security/#reporting-vulnerabilities • Rewards for proper reports
40. Infrastructure management Common Pitfalls
41. © 2018 All Rights Reserved. 41 I have a private instance, I am fine
42. © 2018 All Rights Reserved. 42 Web Interface is NOT the only way to get into a connected system
43. © 2018 All Rights Reserved. 43 Do you pull latest images from DockerHub?
44. © 2018 All Rights Reserved. 44 Do you pull latest images from DockerHub?
45. © 2018 All Rights Reserved. 45 •What’s inside? •Who can change them? •What if there is a malicious code? Do you pull latest images from DockerHub?
46. © 2018 All Rights Reserved. 46 Are Jenkins update centers different?
47. © 2018 All Rights Reserved. 47 They are not. Know what you use Monitor plugin versions and release notes • Beware of transient dependencies (!) • Also monitor JIRA Consider using locally managed sources • Internal Maven, Docker Registry • Custom Jenkins Update Center: Juseppe • https://github.com/yandex-qatools/juseppe Use static configurations • Configuration-as-Code
48. © 2018 All Rights Reserved. 48 Exposing Jenkins filesystem • JENKINS_HOME snapshot is enough to recreate the system • … and to steal credentials
49. © 2018 All Rights Reserved. 49 Master Filesystem • All secrets and credentials within JENKINS_HOME are encrypted • Encryption is powered by secret key files • Secret keys are also stored in JENKINS_HOME/secrets • Snapshot of JENKINS_HOME is enough to get all secrets in Jenkins • External credentials might help
50. © 2018 All Rights Reserved. 50 Read-only backups same as above...
51. © 2018 All Rights Reserved. 51 Running master or agent under privileged accounts
52. © 2018 All Rights Reserved. 52 Running master or agent under privileged accounts • Direct access to the system (e.g. from scripts) • Environment can be altered: registry, system processes, filesystem, etc., etc. • Access to other accounts
53. © 2018 All Rights Reserved. 53 Keep Jenkins in a sandbox Do not run masters/agents under system accounts Restrict access to non-required resources • Generic accounts • Read-only repositories Sandbox your scripts if possible
54. © 2018 All Rights Reserved. 54 Tips & Tricks • Single-shot agents • No environment pollution • Limit the infrastructure impact
55. © 2018 All Rights Reserved. 55 No disaster recovery plan
56. © 2018 All Rights Reserved. 56 No disaster recovery plan • Some exploits may corrupt the instance • Remote Code Execution • Filesystem access
57. © 2018 All Rights Reserved. 57 No disaster recovery plan • Some exploits may corrupt the instance • Remote Code Execution • Filesystem access • The Jenkins host may be compromised • Other infrastructure may be compromised if misconfigured
58. © 2018 All Rights Reserved. 58 No disaster recovery plan • Some exploits may corrupt the instance • Remote Code Execution • Filesystem access • The Jenkins host may be compromised • Other infrastructure may be compromised if misconfigured • It should be possible to rebuild the system from scratch
59. © 2018 All Rights Reserved. 59 Configuration as Code in Jenkins Jobs System Configurations
60. © 2018 All Rights Reserved. 60 System Configuration as Code Just examples… External tools Jenkins CLI and REST API python-jenkins jenkins-client (java) Configuration Management Ansible, Chef, … Docker, Docker Compose ... Solutions in Jenkins Groovy Boot Hooks JCasC . . . SCM Sync Configuration
61. © 2018 All Rights Reserved. 61 System Configuration as Code 1. Docker image: plugins.txt, Java flags, etc. 2. Groovy Init Scripts • Overview: http://bit.ly/jenkins-groovy-hooks 3. OR: New Configuration-as-Code Plugin • https://plugins.jenkins.io/configuration-as-code • Allows configuring Jenkins from YAML
62. Common pitfalls Configuration issues
63. © 2018 All Rights Reserved. 63 Not configuring security in configuration-as-code
64. © 2018 All Rights Reserved. 64 Security & Configuration-as-Code • Security is configured by the Installation Wizard on first startup • Configuration-as-Code logic usually disables the Installation Wizard • “-DrunSetupWizard=false” flag in containers • JCasC skips installation wizard if config is passed • Admin is responsible to set defaults
65. © 2018 All Rights Reserved. 65 Security & Configuration-as-Code Must configure: • Authentication • Authorization jenkins: securityRealm: local: allowsSignup: false enableCaptcha: false # Users are configured elsewhere authorizationStrategy: roleBased: roles: global: - name: "admin" description: "Jenkins administrators" permissions: - "Overall/Administer" assignments: - "admin" - name: "readonly" ... jenkins.yaml https://github.com/oleg-nenashev/demo-j enkins-config-as-code/tree/casc-plugin
66. © 2018 All Rights Reserved. 66 Security & Configuration-as-Code Must configure: • Disable Remoting CLI • CSRF Protection • Slave-to-master security • Remoting protocols (before 2.138.1) jenkins: agentProtocols: - "JNLP4-connect" - "Ping" crumbIssuer: standard: excludeClientIPFromCrumb: true remotingSecurity: enabled: true security: remotingCLI: enabled: false https://github.com/oleg-nenashev/demo-jenkins-config-as-code/ jenkins.yaml
67. © 2018 All Rights Reserved. 67 Same with Groovy import hudson.security.csrf.DefaultCrumbIssuer import jenkins.model.Jenkins import jenkins.CLI import jenkins.security.s2m.AdminWhitelistRule import org.kohsuke.stapler.StaplerProxy CLI.get().enabled = false Jenkins.instance.agentProtocols = new HashSet<String>(["JNLP4-connect"]) Jenkins.instance.getExtensionList(StaplerProxy.class) .get(AdminWhitelistRule.class) .masterKillSwitch = false if (Jenkins.instance.crumbIssuer == null) { println "CSRF protection is disabled, Enabling the default Crumb Issuer" Jenkins.instance.crumbIssuer = new DefaultCrumbIssuer(true) } https://github.com/oleg-nenashev/demo-jenkins-config-as-code/
68. © 2018 All Rights Reserved. 68 Disabling security features
69. © 2018 All Rights Reserved. 69 Disabling security features • Blacklisted classes in Class Deserialization • Blacklisted methods in Script Security • Agent-2-master security • Content Security Policy • Markup Editor • ...
70. © 2018 All Rights Reserved. 70 Permitting builds on the master
71. © 2018 All Rights Reserved. 71 Builds on the master • Builds have access to the master filesystem • Builds run under the Jenkins account • They can… • Read data from other builds/artifacts • Read secret hashes • Modify Jenkins system configuration • … • You don’t want that in 99% of cases
72. © 2018 All Rights Reserved. 72 Restricting builds on the master • Solution 1: • Set “0” executors on master • Another node running under different account • BUT: Fly-weight tasks are permitted
73. © 2018 All Rights Reserved. 73 Restricting builds on the master • Solution 1: • 0 executors on master • Another node running under different account • BUT: Fly-weight tasks are permitted • Solution 2: If you need executors on master • Job Restrictions Plugin • https://plugins.jenkins.io/job-restrictions
74. © 2018 All Rights Reserved. 74 Job Restrictions. Protecting the Master node • NEVER let users run jobs on master • Only use it for system jobs owned by admins
75. © 2018 All Rights Reserved. 75 Hint: Agent.Build permission Example for Ownership-Based security: http://bit.ly/ownership-based-security Available in Role Strategy Plugin, Matrix Authorization, CloudBees RBAC
76. © 2018 All Rights Reserved. 76 Not restricting Jenkins queue
77. © 2018 All Rights Reserved. 77 Not restricting Jenkins queue • By default Queue tasks run with the System account
78. © 2018 All Rights Reserved. 78 Not restricting Jenkins queue • By default Queue tasks run with the System account • Users may trigger ANY builds • Parameterized Trigger, Pipeline steps
79. © 2018 All Rights Reserved. 79 Not restricting Jenkins queue • By default Queue tasks run with the System account • Users may trigger ANY builds • Parameterized Trigger, Pipeline steps • Users can extract data from other builds • Copy Artifacts - artifacts and workspaces
80. © 2018 All Rights Reserved. 80 Not restricting Jenkins queue • By default Queue tasks run with the System account • Users may trigger ANY builds • Parameterized Trigger, Pipeline steps • Users can extract data from other builds • Copy Artifacts - artifacts and workspaces • Computer.Build is always on • Task can run on any agent
81. © 2018 All Rights Reserved. 81 Authorize Project Plugin Authorize builds • Global default • Whitelist of user-configurable strategies • Job properties https://plugins.jenkins.io/authorize-project
82. © 2018 All Rights Reserved. 82 Granting EXTENDED_READ to non-privileged users
83. © 2018 All Rights Reserved. 83 Granting EXTENDED_READ to non-privileged users • config.xml may include sensitive data • Mistakes by job developers • Security defects in plugins (e.g. plaintext passwords) • The permission is disabled for a reason
84. Summary
85. © 2018 All Rights Reserved. 85 Security advisories Source: status update by Daniel Beck at the Jenkins Contributor Summit, Sep 17, 2018
86. © 2018 All Rights Reserved. 86 Security advisories Source: status update by Daniel Beck at the Jenkins Contributor Summit, Sep 17, 2018 o_O
87. © 2018 All Rights Reserved. 87 •It’s not! •All fixes go to advisories •Technical debt cleaning for 1500+ components •50% of fixes - plugins installed on 3% instances or less FAQ: Is Jenkins security that bad?
88. © 2018 All Rights Reserved. 88 What’s next? • Continuous effort on improving security • Security cleanup in “long-tail” plugins • “Secure by default” in Cloud Native Jenkins • https://jenkins.io/blog/2018/08/31/shifting-gears/
89. © 2018 All Rights Reserved. 89 Do you think that your Jenkins instance is secure?
90. © 2018 All Rights Reserved. 90 Ask yourself ❑ Are you subscribed to Jenkins security advisories? ❑ Do you keep your Jenkins up to date? ❑ Is JENKINS_HOME or backups accessible? ❑ Do you initialize security in Configuration-as-code? ❑ Do you restrict jobs on the master? ❑ Do you use Authorize Project to restrict the queue?
91. © 2018 All Rights Reserved. 91 1. Follow the security advisories 2. Keep your Jenkins instances up to date 3. Check your Jenkins infrastructure and configuration 4. Be careful when using niche plugins 5. Try Configuration-as-code Takeaways
92. © 2018 All Rights Reserved. 92 •Security page: https://jenkins.io/security/ •Advisories: https://jenkins.io/security/advisories/ •Ownership-based security: http://bit.ly/ownership-based-security •Configurations-as-Code demo (see branches): https://hub.docker.com/r/onenashev/demo-jenkins-config-as-code/ •My previous talk about security best practices: https://speakerdeck.com/onenashev/spb-jenkins-meetup-number-9-managing-sec urity-in-jenkins-eng Links
93. © 2018 All Rights Reserved. 93 Thank you! Contacts: E-mail: onenashev@cloudbees.com GitHub: oleg-nenashev Twitter: @oleg_nenashev