Mar 07, 2018 Oleg Nenashev Jenkins contributor, core maintainer https://github.com/oleg-nenashev/ 1 JEP-200 Update “Switch Remoting/XStream from blacklist to whitelist”

2 ● Deserialization attacks are real ● CVE-2017-1000353 ● Fixed in April 2017 ● Jenkins 2.46.2+ https://research.checkpoint.com/jenkins-miner-one-biggest-mining-operations-ever-discovered/

Why did it happen? 1. Class deserialization attack 2. Unauthenticated Remote Code Execution via Remoting CLI • CVE-2017-1000353 • https://jenkins.io/security/advisory/2017-04-26/ 3. Mitigation: • Vulnerable classes have been blacklisted • Remoting CLI was deprecated & disabled in 2.46.2+ • Jenkins project issues several reminders about disabling CLI 4. Somebody created an exploit and used it • Impact: Public instances with old core versions 3

Blacklists are NOT enough We used to blacklist classes • Remoting's ClassFilter • Several advisories in 2015 – 2017 4

Blacklists are NOT enough We used to blacklist classes • Remoting's ClassFilter • Several advisories in 2015 – 2017 Blacklisting == Whack-A-Mole • Unknown classes vulnerable to deserialization attacks 5

JEP-200 6

JEP-200: “just” the biggest incompatible change to Jenkins in recent memory 7 © Jesse Glick, JEP-200 Sponsor

JEP-200 • Apr 2017 - Planning started • Oct 2017 - JEP-200 draft published • Jan 2018 - Released in Jenkins weekly 2.102 • https://jenkins.io/blog/2018/01/13/jep-200/ • Mar 01, 2018 • Adoption: ~12% of Jenkins installations • >90% of regressions are fixed and released • Good community ratings for 5 weekly releases • Mar 15, 2018 - Availability in LTS? 8

Time to upgrade? 9

IT HAS BEEN 0 DAYS SINCE THE LAST JEP-200 REGRESSION REPORT 10

Affected plugins 11 More than 50 plugins affected ● Pipeline: API, Pipeline: Declarative, JobDSL ● Monitoring, Config File Provider, Mesos ● Maven Integration, Artifactory, Build Name Setter, Gerrit Trigger, Build Failure Analyzer, Publish Over .*, Analysis Core, .. ● Build Flow Plugin (deprecated & depublished) https://wiki.jenkins.io/display/JENKINS/Plugins+affected+by+fix+for+JEP-200

JEP-200 Example Build Name Setter 1.6.7 (fix applied in 1.6.8), Default configuration 12

JEP-200 Example Build Name Setter 1.6.7 (fix applied in 1.6.8), Default configuration 13

14 Upgrading to Jenkins 2.107+

FAQ 1. Are the issues real? • Yes, JEP-200 reveals REAL functional issues • Some collateral damage 2. Am I safe? • Yes, there is no known class deserialization exploits • JEP-200 is a security hardening 3. Do I have to upgrade immediately on Mar 15? • No 15

Upgrading to 2.107.1+ 1. Read https://jenkins.io/blog/2018/01/13/jep-200 2. Backup your instance 3. Update all affected plugins 4. Apply workarounds if needed 5. Monitor your instance • Configuration loading on startup • Build execution 16 https://jenkins.io/blog/2018/01/13/jep-200/#for-jenkins-administrators

Applying workarounds Some classes can be whitelisted: -Dhudson.remoting.ClassFilter=pkg.and.Class1,pkg.and.Class2 But: • Applying workaround may require several iterations • Some classes are blacklisted, e.g. net.sf.json.JSONObject • You may need to rollback 17 https://jenkins.io/blog/2018/01/13/jep-200/#for-jenkins-administrators

I see a stacktrace in logs, is it JEP-200? https://jenkins.io/redirect/class-filter/ in text? 18 YES Yes, it is likely JEP-200 Report it (see below) NO Most likely, NO Report it for a common triage

Reporting issues • JEP-200 maintainers are ready to triage issues (and fix them) • Monitoring issues until May 1st, 2018 • We need your help - report issues • https://issues.jenkins-ci.org/ • Use the “JEP-200” label • Issue examples: here 19

Takeaways 1. Read https://jenkins.io/blog/2018/01/13/jep-200/ 2. Be prepared • March 15 - availability in LTS: 2.107.1 • JEP-200 maintainers will be monitoring JIRA 3. Test your instances • RC: http://mirrors.jenkins.io/war-stable-rc/2.107.1/ • Report issues with “JEP-200” label 4. If you are a plugin maintainer, test your plugin(s) • See below 20

Takeaways 1. Read https://jenkins.io/blog/2018/01/13/jep-200/ 2. Be prepared 3. Test your instances 4. If you are a plugin maintainer, test your plugin(s) 5. Keep Updating! 21

Thanks! 22 Q&A: ● Jenkins IRC Channel ○ https://jenkins.io/chat/ ● Jenkins Developer and User mailing lists ○ https://jenkins.io/mailing-lists/

23 JEP-200 for Plugin Maintainers

Class Deserialization 101 public class Foo implements Serializable { private String command; private Object readResolve() { Process p = Runtime.getRuntime().exec(command); return this; } } Details / Real examples: ● https://www.slideshare.net/codewhitesec/exploiting-deserialization-vulnerabilities-in-java-54707478 ● https://www.christian-schneider.net/JavaDeserializationSecurityFAQ.html ● https://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-appl ication-have-in-common-this-vulnerability/ 24

In the Last 90 Days JEP-200 Maintainers... • Tested more than 100 plugins • Facelifted >70 plugins • Plugin refreshes (POM update, etc.) • Support for Plugin Compatibility Tester • Fixed >50 plugins (Wiki) • JEP-200 exposes structural mistakes in plugins/core that could affect security, performance, upgradability, etc. © Jesse • Some collateral damage 25

JEP-200. What may break? • Classes from 3rd-party libraries • HOT: Model objects ▸ MavenInformation from Maven core - Maven Plugin ▸ 3rd-party TestResult classes - TestLink/TAP Plugin • Non-whitelisted Java classes • https://github.com/jenkinsci/jenkins/blob/master/core/src/m ain/resources/jenkins/security/whitelisted-classes.txt • E.g. Calendar or DateFormatter are not whitelisted • Non-whitelisted classes from Jenkins core libs • E.g. Guava Collections 26

How to test JEP-200? Classic approach: • Use Plugin Compat Tester (aka PCT) • Run Acceptance Test Harness • Check plugins on Test servers (if possible) 27

PCT. Quick Start docker run --rm -v maven-repo:/root/.m2 -v $(pwd)/out:/pct/out -v jenkins-2.107.1-rc.war:/pct/jenkins.war:ro -v plugin-sources/ssh-slaves-plugin:/pct/plugin-src:ro -e ARTIFACT_ID=ssh-slaves jenkins/pct More documentation: https://github.com/jenkinsci/plugin-compat-tester 28

PCT. Checking the report DEMO: PCT Report Look for: • Stacktraces “https://jenkins.io/redirect/class-filter/” references • Regression against current baselines 29

What to test? XStream • All classes being persisted on the disk • Hot areas: • Run/Project actions: data stored by plugins • Improperly cached objects (no “transient” / “static”) • Historic data is a subject for loading issues • May even cause DoS :( 30

What to test? Remoting • HOT: All callables, especially anonymous inner classes • MasterToSlaveCallable • MasterToSlaveFileCallable • hudson.remoting.Callable in old plugins • SlaveToMasterCallable • Not just a single type whitelist • Type AND fields should be whitelisted: ▸ Implementation classes ▸ All upstream abstract classes • Passing final variables to anonymous callable classes 31