Tweet
Tweet

Slide 1

Slide 1 text

Managing security in Jenkins. Best practices and Ownership-Based Security Oleg Nenashev CloudBees, Inc. St. Petersburg Jenkins Meetup November 28, 2017

Slide 2

Slide 2 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 2 About me @oleg_nenashev oleg-nenashev LibreCores project St. Petersburg Polytechnic University Jenkins meetups

Slide 3

Slide 3 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 3 Oleg’s “Hall of Shame”(c) • Plugins • Jenkins Core • Windows Service Wrapper • Remoting • Security Team

Slide 4

Slide 4 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 4 https://jenkins.io/security/ Fixes in the core and plugins Jenkins Security Team Board Core Team Security LTS Events INFRA Website

Slide 5

Slide 5 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 5 About you Do you administer Jenkins instances?

Slide 6

Slide 6 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 6 About you Do you administer Jenkins instances? Do you have more than 20 users?

Slide 7

Slide 7 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 7 Agenda 1. Introduction to Jenkins Security 2. Protecting Jenkins. Best practices 3. Ownership-based security Disclaimer: • Presentation represent the author’s personal opinion • Author’s personal opinion may differ from official positions of CloudBees and/or Jenkins community • Many Jenkins instances were harmed, use carefully

Slide 8

Slide 8 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 8 1. Most popular CI/CD tool in the world 2. Generic automation server 3. Flexible and extensible 4. It’s open source, big community 5. Commercial support vendors 6. … Who is Mr. Jenkins? https://jenkins.io

Slide 9

Slide 9 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 9

Slide 10

Slide 10 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 10 Jenkins… is a remote execution engine (by design)

Slide 11

Slide 11 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 11 Jenkins… is a remote execution engine (by design) • One can run code and system commands • Access to master system • Access to agents • Access to private/public clouds

Slide 12

Slide 12 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 12 Jenkins… has access to sensitive data (by design)

Slide 13

Slide 13 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 13 Jenkins… has access to sensitive data (by design) • Credentials • Private repositories • Artifacts, including release ones

Slide 14

Slide 14 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 14 Jenkins… is a shared service (by design)

Slide 15

Slide 15 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 15 Jenkins… is a shared service (by design) • Multiple users • Different expertise • Users may misuse permissions

Slide 16

Slide 16 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 16 What does security mean? Jenkins security Intrusion and data theft protection Restrictions within organization

Slide 17

Slide 17 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 17 Jenkins security Intrusion and data theft protection Restrictions within organization • Must-have in internet-facing instances • Paranoid mode is fine What does security mean?

Slide 18

Slide 18 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 18 Jenkins security Intrusion and data theft protection Restrictions within organization • Better user experience • Protection from unintentional actions • Protection from lack of expertise What does security mean?

Slide 19

Slide 19 text

Protecting Jenkins. Best practices

Slide 20

Slide 20 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 20 Limited number of admins Permissions Security audit Rule #0. Use security!

Slide 21

Slide 21 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 21 Rule #1. Keep Updating Frequent security releases • Weekly • Current LTS baseline Info sources • https://jenkins.io/security/advisories/ • jenkinsci-advisories mailing list (including announcements) • RSS feed 2.46.2 Exploits are in the wild, update ASAP

Slide 22

Slide 22 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 22 LTS is only 12 weeks… Not enough?

Slide 23

Slide 23 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 23 Build your own core (custom fork) •“mvn clean package” in the root •HINT: Join the security team to get info about changes in advance LTS is only 12 weeks… Not enough?

Slide 24

Slide 24 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 24 Build your own core (custom fork) JEP it, help to maintain! • People is interested in longer LTS LTS is only 12 weeks… Not enough?

Slide 25

Slide 25 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 25 Build your own core (custom fork) JEP it, help to maintain! Use custom versions from vendors: • https://wiki.jenkins-ci.org/display/JENKINS/Commercial+Support • CloudBees Jenkins Enterprise LTS is only 12 weeks… Not enough?

Slide 26

Slide 26 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 26 Do you pull latest images from DockerHub?

Slide 27

Slide 27 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 27 Do you pull latest images from DockerHub?

Slide 28

Slide 28 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 28 What’s inside? Who can change them? What if there is a malicious code? Do you pull latest images from DockerHub?

Slide 29

Slide 29 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 29 What’s inside? Who can change them? What if there is a malicious code? How is it different from other package sources? Do you pull latest images from DockerHub?

Slide 30

Slide 30 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 30 Plugins may contain defects Rule #2. Know what you use

Slide 31

Slide 31 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 31 Rule #2. Know what you use Monitor plugin versions and release notes • Beware of transient dependencies (!) • Also monitor JIRA

Slide 32

Slide 32 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 32 Rule #2. Know what you use Monitor plugin versions and release notes Consider using locally managed sources • Internal Maven • Docker Registry • Custom Jenkins Update Center: Juseppe • https://github.com/yandex-qatools/juseppe

Slide 33

Slide 33 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 33 Rule #2. Know what you use Monitor plugin versions and release notes Consider using locally managed sources Use static configurations • Configuration-as-Code • One cannot simply break it

Slide 34

Slide 34 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 34 Previous Jenkins Meetup… http://bit.ly/jenkins_msk_3_groovy_hooks

Slide 35

Slide 35 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 35 System Configuration… as Code External tools Jenkins CLI and REST API python- jenkins jenkins-client (java) Configuration Management Ansible, Chef, … Docker, Docker Compose ... Solutions in Jenkins Groovy Boot Hooks Config as Code Plugin (alpha) SCM Sync Configuration Just examples…

Slide 36

Slide 36 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 36 Disabling setup wizard… turns off security, SURPRISE! Configuration-as-code. Keep in mind!

Slide 37

Slide 37 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 37 Disabling setup wizard… turns off security, SURPRISE! You are responsible for configuring security Configuration-as-code. Keep in mind!

Slide 38

Slide 38 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 38 Disabling setup wizard… turns off security, SURPRISE! You are responsible for configuring security Configuration-as-code. Keep in mind!

Slide 39

Slide 39 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 39 You are responsible for configuring security Examples: • Authentication/Authorization • CSRF • Slave2Master security • Remoting protocols Configuration-as-code. Keep in mind!

Slide 40

Slide 40 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 40 https://hub.docker.com/r/onenashev/ demo-jenkins-config-as-code/ Groovy Hooks

Slide 41

Slide 41 text

Is it enough?

Slide 42

Slide 42 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 42 Rule #3. Keep Jenkins in a sandbox

Slide 43

Slide 43 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 43 Rule #3. Keep Jenkins in a sandbox Do not run masters/agents under system accounts • BAD - Local Administrator in Windows • BAD - Root in Unix • NOT BAD? - Root in Docker

Slide 44

Slide 44 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 44 Rule #3. Keep Jenkins in a sandbox Do not run masters/agents under system accounts Restrict access to non- required resources • Generic accounts • Read-only repositories

Slide 45

Slide 45 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 45 Rule #3. Keep Jenkins in a sandbox Do not run masters/agents under system accounts Restrict access to non- required resources Sandbox your scripts as well

Slide 46

Slide 46 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 46 Scryptocalypse https://jenkins.io/security/advisory/2017-04-10/ • Unrestricted scripting • More than 30 plugins affected •Groovy Plugin •JobDSL Plugin •Grails Plugin •Scriptler Plugin • Some of them are blocked, even now

Slide 47

Slide 47 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 47 It is not only about Groovy… Tcl Plugin •Blacklisted in April 2017 Jenkins Core 2.73.3 •Command Computer Launcher •Runs command on a master •Any user with a mode edit permission… was able to run commands on the master

Slide 48

Slide 48 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 48 DIY

Slide 49

Slide 49 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 49 Jenkins Script Security https://plugins.jenkins.io/ script-security DIY

Slide 50

Slide 50 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 50 Script Security Plugin Used in [almost] all Groovy plugins https://plugins.jenkins.io/script-security

Slide 51

Slide 51 text

Is it enough to become secure?

Slide 52

Slide 52 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 52 Rule #4. Do not Run Jobs on master

Slide 53

Slide 53 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 53 Builds have access to the master filesystem Examples: • Read data from other builds/artifacts • Read secret hashes • Modify Jenkins system configuration Rule #4. Do not Run Jobs on master

Slide 54

Slide 54 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 54 Rule #4. Do not Run Jobs on master • Solution 1: • Set “0” executors on master • Another node running under different account • BUT: Does not protect from fly-weight tasks

Slide 55

Slide 55 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 55 Rule #4. Do not Run Jobs on master • Solution 1: • 0 executors on master • Another node running under different account • BUT: Does not protect from fly-weight tasks • Solution 2: • Job Restrictions Plugin • Details: later

Slide 56

Slide 56 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 56 Ø By default builds run with the System account Ø Users may trigger wrong builds Ø Users can extract data Rule #5. Do not trust your builds

Slide 57

Slide 57 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 57 Authorize Project Plugin Authorize builds • Global default • Whitelist of user- configurable strategies • Job properties https://plugins.jenkins.io/authorize-project

Slide 58

Slide 58 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 58 Audit Trail – logging of actions •https://plugins.jenkins.io/audit-trail Security Inspector – permission checks •https://plugins.jenkins.io/security-inspector … Rule #6. Audit your security

Slide 59

Slide 59 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 59 Security Inspector Plugin https://plugins.jenkins.io/security-inspector Reports for jobs, agents and users

Slide 60

Slide 60 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 60 1. Assign leads to jobs and agents 2. Share the maintenance effort with them 3. Make the ownership explicit Rule #7. Make the responsibilities explicit

Slide 61

Slide 61 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 61 Common strategies do not “just work” Project Matrix Authorization Strategy • Can be managed on Job/Folder level • Hard to manage every item • WAS: No support of Node permissions

Slide 62

Slide 62 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 62 Common strategies do not “just work” Project Matrix Authorization Strategy • Hard to manage • No support of Node permissions Role-Based Strategy • Regular expression for each role • Performance: Hundreds of Regex checks every request • Web UI easily hangs

Slide 63

Slide 63 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 63 Role Strategy, it’s FUN!

Slide 64

Slide 64 text

Ownership-Based Security

Slide 65

Slide 65 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 65 Ownership-based Security Role- Strategy Ownership Job Restrictions • Assign owners of jobs/nodes • Fancy UI • Auth strategy • Macro engine • Restrict runs for jobs and nodes http://bit.ly/ownership-based-security + Authorize Project

Slide 66

Slide 66 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 66 Ownership Plugin • Primary and Secondary Owners • Summary Boxes, View filters, etc. • Environment variables • Integration with Security plugins Customizable layout https://plugins.jenkins.io/ownership

Slide 67

Slide 67 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 67 Ownership Info. Definition and Inheritance Folders Jobs Nodes Runs Sub- Projects Inherits

Slide 68

Slide 68 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 68 Demo. What’s inside? Ownership 0.10.0 Job Restrictions 0.6 Security Inspector 0.4 Jenkins core 2.73.3 (minimal – 1.625) Authorize Project 1.3.0 Dynamic Search View 0.2.2 Role Strategy 2.6.1

Slide 69

Slide 69 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 69 https://hub.docker.com/r/onenashev/ demo-jenkins-config-as-code/

Slide 70

Slide 70 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 70 Setting ownership info

Slide 71

Slide 71 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 71 Ownership Info. What Do you get? • Ownership Summary Boxes • Ownership View Columns • View Filters • Also: @Me macro Customizable layout

Slide 72

Slide 72 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 72 Example: Quick administration contacts Customizable template

Slide 73

Slide 73 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 73 Ownership-Based Security. Role-Based Strategy Settings Roles [1/2]

Slide 74

Slide 74 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 74 Ownership-Based Security. Role-Based Strategy Settings Roles [2/2]

Slide 75

Slide 75 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 75 Ownership-Based Security. Role-Based Strategy Settings Assignments

Slide 76

Slide 76 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 76 Jobs. Securing access Untrusted secondary owners!

Slide 77

Slide 77 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 77 Jobs. Authorize Project Jobs get authenticated as owners => • Permissions • Node access (Computer.BUILD)

Slide 78

Slide 78 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 78 Using Data in Jobs. Freestyle

Slide 79

Slide 79 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 79 Using Data in Jobs. Pipeline

Slide 80

Slide 80 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 80 Jenkins nodes • Similar Ownership Management • Special permission • Node Ownership Monitor • => info in the table

Slide 81

Slide 81 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 81 Securing Nodes

Slide 82

Slide 82 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 82 Job Restrictions. Protecting the Master node • NEVER let users run jobs on master • Only use it for system jobs owned by admins

Slide 83

Slide 83 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 83 Ownership-Based Security: Links Plugins: • https://plugins.jenkins.io/ownership • https://plugins.jenkins.io/role-strategy • https://plugins.jenkins.io/job-restrictions • https://plugins.jenkins.io/authorize-project Ownership-based security: • http://bit.ly/ownership-based-security Demo • https://github.com/oleg-nenashev/demo-jenkins-config-as-code

Slide 84

Slide 84 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 84 Ø Item-specific security Ø Ownership-based restrictions for triggering jobs Ø Ownership assignment policy on create/copy Ø “sudo” mode for admins Ownership-Based Security: Out of the scope http://bit.ly/ownership-based-security

Slide 85

Slide 85 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 85 1. Subscribe to security advisories 2. Use Security plugins 3. Keep your Jenkins up to date Takeaways

Slide 86

Slide 86 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 86 1. There are existing solutions for large-scale 2. They are not documented sometimes… 3. Google ’em all Rule #-1. Explore

Slide 87

Slide 87 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 87 Security page • https://jenkins.io/security/ Advisories • https://jenkins.io/security/advisories/ Ownership-based security • http://bit.ly/ownership-based-security Demo • https://hub.docker.com/r/onenashev/demo-jenkins-config-as-code/ Links

Slide 88

Slide 88 text

@oleg_nenashev, @jenkins_spb © 2017 CloudBees, Inc. All Rights Reserved. 88 Thank you! Contacts: E-mail: onenashev@cloudbees.com GitHub: oleg-nenashev Twitter: @oleg_nenashev