Jenkins JEP-200 - Status update and heads up

This presentation provides heads-up about the incoming release of JEP-200 "Switch Remoting/XStream blacklist to whitelist". The change will land in Jenkins 2.107.1. Target audience: Jenkins administrators and plugin maintainers. What to expect from JEP-200? How to upgrade? How to test changes? And how to verify your plugins?

Video: https://www.youtube.com/watch?v=Vfnc9t1RuYA

Transcript

1. Mar 07, 2018 Oleg Nenashev Jenkins contributor, core maintainer https://github.com/oleg-nenashev/ 1 JEP-200 Update “Switch Remoting/XStream from blacklist to whitelist”
2. 2 • Deserialization attacks are real • CVE-2017-1000353 • Fixed in April 2017 • Jenkins 2.46.2+ https://research.checkpoint.com/jenkins-miner-one-biggest-mining-operations-ever-discovered/
3. Why did it happen? 1. Class deserialization attack 2. Unauthenticated Remote Code Execution via Remoting CLI • CVE-2017-1000353 • https://jenkins.io/security/advisory/2017-04-26/ 3. Mitigation: • Vulnerable classes have been blacklisted • Remoting CLI was deprecated & disabled in 2.46.2+ • Jenkins project issues several reminders about disabling CLI 4. Somebody created an exploit and used it • Impact: Public instances with old core versions 3
4. Blacklists are NOT enough We used to blacklist classes • Remoting's ClassFilter • Several advisories in 2015 – 2017 4
5. Blacklists are NOT enough We used to blacklist classes • Remoting's ClassFilter • Several advisories in 2015 – 2017 Blacklisting == Whack-A-Mole • Unknown classes vulnerable to deserialization attacks 5
6. JEP-200 6
7. JEP-200: “just” the biggest incompatible change to Jenkins in recent memory 7 © Jesse Glick, JEP-200 Sponsor
8. JEP-200 • Apr 2017 - Planning started • Oct 2017 - JEP-200 draft published • Jan 2018 - Released in Jenkins weekly 2.102 • https://jenkins.io/blog/2018/01/13/jep-200/ • Mar 01, 2018 • Adoption: ~12% of Jenkins installations • >90% of regressions are fixed and released • Good community ratings for 5 weekly releases • Mar 15, 2018 - Availability in LTS? 8
9. Time to upgrade? 9
10. IT HAS BEEN 0 DAYS SINCE THE LAST JEP-200 REGRESSION REPORT 10
11. Affected plugins 11 More than 50 plugins affected • Pipeline: API, Pipeline: Declarative, JobDSL • Monitoring, Config File Provider, Mesos • Maven Integration, Artifactory, Build Name Setter, Gerrit Trigger, Build Failure Analyzer, Publish Over .*, Analysis Core, .. • Build Flow Plugin (deprecated & depublished) https://wiki.jenkins.io/display/JENKINS/Plugins+affected+by+fix+for+JEP-200
12. JEP-200 Example Build Name Setter 1.6.7 (fix applied in 1.6.8), Default configuration 12
13. JEP-200 Example Build Name Setter 1.6.7 (fix applied in 1.6.8), Default configuration 13
14. 14 Upgrading to Jenkins 2.107+
15. FAQ 1. Are the issues real? • Yes, JEP-200 reveals REAL functional issues • Some collateral damage 2. Am I safe? • Yes, there is no known class deserialization exploits • JEP-200 is a security hardening 3. Do I have to upgrade immediately on Mar 15? • No 15
16. Upgrading to 2.107.1+ 1. Read https://jenkins.io/blog/2018/01/13/jep-200 2. Backup your instance 3. Update all affected plugins 4. Apply workarounds if needed 5. Monitor your instance • Configuration loading on startup • Build execution 16 https://jenkins.io/blog/2018/01/13/jep-200/#for-jenkins-administrators
17. Applying workarounds Some classes can be whitelisted: -Dhudson.remoting.ClassFilter=pkg.and.Class1,pkg.and.Class2 But: • Applying workaround may require several iterations • Some classes are blacklisted, e.g. net.sf.json.JSONObject • You may need to rollback 17 https://jenkins.io/blog/2018/01/13/jep-200/#for-jenkins-administrators
18. I see a stacktrace in logs, is it JEP-200? https://jenkins.io/redirect/class-filter/ in text? 18 YES Yes, it is likely JEP-200 Report it (see below) NO Most likely, NO Report it for a common triage
19. Reporting issues • JEP-200 maintainers are ready to triage issues (and fix them) • Monitoring issues until May 1st, 2018 • We need your help - report issues • https://issues.jenkins-ci.org/ • Use the “JEP-200” label • Issue examples: here 19
20. Takeaways 1. Read https://jenkins.io/blog/2018/01/13/jep-200/ 2. Be prepared • March 15 - availability in LTS: 2.107.1 • JEP-200 maintainers will be monitoring JIRA 3. Test your instances • RC: http://mirrors.jenkins.io/war-stable-rc/2.107.1/ • Report issues with “JEP-200” label 4. If you are a plugin maintainer, test your plugin(s) • See below 20
21. Takeaways 1. Read https://jenkins.io/blog/2018/01/13/jep-200/ 2. Be prepared 3. Test your instances 4. If you are a plugin maintainer, test your plugin(s) 5. Keep Updating! 21
22. Thanks! 22 Q&A: • Jenkins IRC Channel ◦ https://jenkins.io/chat/ • Jenkins Developer and User mailing lists ◦ https://jenkins.io/mailing-lists/
23. 23 JEP-200 for Plugin Maintainers
24. Class Deserialization 101 public class Foo implements Serializable { private String command; private Object readResolve() { Process p = Runtime.getRuntime().exec(command); return this; } } Details / Real examples: • https://www.slideshare.net/codewhitesec/exploiting-deserialization-vulnerabilities-in-java-54707478 • https://www.christian-schneider.net/JavaDeserializationSecurityFAQ.html • https://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-appl ication-have-in-common-this-vulnerability/ 24
25. In the Last 90 Days JEP-200 Maintainers... • Tested more than 100 plugins • Facelifted >70 plugins • Plugin refreshes (POM update, etc.) • Support for Plugin Compatibility Tester • Fixed >50 plugins (Wiki) • JEP-200 exposes structural mistakes in plugins/core that could affect security, performance, upgradability, etc. © Jesse • Some collateral damage 25
26. JEP-200. What may break? • Classes from 3rd-party libraries • HOT: Model objects ▸ MavenInformation from Maven core - Maven Plugin ▸ 3rd-party TestResult classes - TestLink/TAP Plugin • Non-whitelisted Java classes • https://github.com/jenkinsci/jenkins/blob/master/core/src/m ain/resources/jenkins/security/whitelisted-classes.txt • E.g. Calendar or DateFormatter are not whitelisted • Non-whitelisted classes from Jenkins core libs • E.g. Guava Collections 26
27. How to test JEP-200? Classic approach: • Use Plugin Compat Tester (aka PCT) • Run Acceptance Test Harness • Check plugins on Test servers (if possible) 27
28. PCT. Quick Start docker run --rm -v maven-repo:/root/.m2 -v $(pwd)/out:/pct/out -v jenkins-2.107.1-rc.war:/pct/jenkins.war:ro -v plugin-sources/ssh-slaves-plugin:/pct/plugin-src:ro -e ARTIFACT_ID=ssh-slaves jenkins/pct More documentation: https://github.com/jenkinsci/plugin-compat-tester 28
29. PCT. Checking the report DEMO: PCT Report Look for: • Stacktraces “https://jenkins.io/redirect/class-filter/” references • Regression against current baselines 29
30. What to test? XStream • All classes being persisted on the disk • Hot areas: • Run/Project actions: data stored by plugins • Improperly cached objects (no “transient” / “static”) • Historic data is a subject for loading issues • May even cause DoS :( 30
31. What to test? Remoting • HOT: All callables, especially anonymous inner classes • MasterToSlaveCallable<Type> • MasterToSlaveFileCallable<Type> • hudson.remoting.Callable in old plugins • SlaveToMasterCallable • Not just a single type whitelist • Type AND fields should be whitelisted: ▸ Implementation classes ▸ All upstream abstract classes • Passing final variables to anonymous callable classes 31