Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Jenkins JEP-200 - Status update and heads up

Jenkins JEP-200 - Status update and heads up

This presentation provides heads-up about the incoming release of JEP-200 "Switch Remoting/XStream blacklist to whitelist". The change will land in Jenkins 2.107.1. Target audience: Jenkins administrators and plugin maintainers. What to expect from JEP-200? How to upgrade? How to test changes? And how to verify your plugins?

Video: https://www.youtube.com/watch?v=Vfnc9t1RuYA

568e3391c8b528f2b255443e4cca27ca?s=128

Oleg Nenashev

March 07, 2018
Tweet

Transcript

  1. Mar 07, 2018 Oleg Nenashev Jenkins contributor, core maintainer https://github.com/oleg-nenashev/

    1 JEP-200 Update “Switch Remoting/XStream from blacklist to whitelist”
  2. 2 • Deserialization attacks are real • CVE-2017-1000353 • Fixed

    in April 2017 • Jenkins 2.46.2+ https://research.checkpoint.com/jenkins-miner-one-biggest-mining-operations-ever-discovered/
  3. Why did it happen? 1. Class deserialization attack 2. Unauthenticated

    Remote Code Execution via Remoting CLI • CVE-2017-1000353 • https://jenkins.io/security/advisory/2017-04-26/ 3. Mitigation: • Vulnerable classes have been blacklisted • Remoting CLI was deprecated & disabled in 2.46.2+ • Jenkins project issues several reminders about disabling CLI 4. Somebody created an exploit and used it • Impact: Public instances with old core versions 3
  4. Blacklists are NOT enough We used to blacklist classes •

    Remoting's ClassFilter • Several advisories in 2015 – 2017 4
  5. Blacklists are NOT enough We used to blacklist classes •

    Remoting's ClassFilter • Several advisories in 2015 – 2017 Blacklisting == Whack-A-Mole • Unknown classes vulnerable to deserialization attacks 5
  6. JEP-200 6

  7. JEP-200: “just” the biggest incompatible change to Jenkins in recent

    memory 7 © Jesse Glick, JEP-200 Sponsor
  8. JEP-200 • Apr 2017 - Planning started • Oct 2017

    - JEP-200 draft published • Jan 2018 - Released in Jenkins weekly 2.102 • https://jenkins.io/blog/2018/01/13/jep-200/ • Mar 01, 2018 • Adoption: ~12% of Jenkins installations • >90% of regressions are fixed and released • Good community ratings for 5 weekly releases • Mar 15, 2018 - Availability in LTS? 8
  9. Time to upgrade? 9

  10. IT HAS BEEN 0 DAYS SINCE THE LAST JEP-200 REGRESSION

    REPORT 10
  11. Affected plugins 11 More than 50 plugins affected • Pipeline:

    API, Pipeline: Declarative, JobDSL • Monitoring, Config File Provider, Mesos • Maven Integration, Artifactory, Build Name Setter, Gerrit Trigger, Build Failure Analyzer, Publish Over .*, Analysis Core, .. • Build Flow Plugin (deprecated & depublished) https://wiki.jenkins.io/display/JENKINS/Plugins+affected+by+fix+for+JEP-200
  12. JEP-200 Example Build Name Setter 1.6.7 (fix applied in 1.6.8),

    Default configuration 12
  13. JEP-200 Example Build Name Setter 1.6.7 (fix applied in 1.6.8),

    Default configuration 13
  14. 14 Upgrading to Jenkins 2.107+

  15. FAQ 1. Are the issues real? • Yes, JEP-200 reveals

    REAL functional issues • Some collateral damage 2. Am I safe? • Yes, there is no known class deserialization exploits • JEP-200 is a security hardening 3. Do I have to upgrade immediately on Mar 15? • No 15
  16. Upgrading to 2.107.1+ 1. Read https://jenkins.io/blog/2018/01/13/jep-200 2. Backup your instance

    3. Update all affected plugins 4. Apply workarounds if needed 5. Monitor your instance • Configuration loading on startup • Build execution 16 https://jenkins.io/blog/2018/01/13/jep-200/#for-jenkins-administrators
  17. Applying workarounds Some classes can be whitelisted: -Dhudson.remoting.ClassFilter=pkg.and.Class1,pkg.and.Class2 But: •

    Applying workaround may require several iterations • Some classes are blacklisted, e.g. net.sf.json.JSONObject • You may need to rollback 17 https://jenkins.io/blog/2018/01/13/jep-200/#for-jenkins-administrators
  18. I see a stacktrace in logs, is it JEP-200? https://jenkins.io/redirect/class-filter/

    in text? 18 YES Yes, it is likely JEP-200 Report it (see below) NO Most likely, NO Report it for a common triage
  19. Reporting issues • JEP-200 maintainers are ready to triage issues

    (and fix them) • Monitoring issues until May 1st, 2018 • We need your help - report issues • https://issues.jenkins-ci.org/ • Use the “JEP-200” label • Issue examples: here 19
  20. Takeaways 1. Read https://jenkins.io/blog/2018/01/13/jep-200/ 2. Be prepared • March 15

    - availability in LTS: 2.107.1 • JEP-200 maintainers will be monitoring JIRA 3. Test your instances • RC: http://mirrors.jenkins.io/war-stable-rc/2.107.1/ • Report issues with “JEP-200” label 4. If you are a plugin maintainer, test your plugin(s) • See below 20
  21. Takeaways 1. Read https://jenkins.io/blog/2018/01/13/jep-200/ 2. Be prepared 3. Test your

    instances 4. If you are a plugin maintainer, test your plugin(s) 5. Keep Updating! 21
  22. Thanks! 22 Q&A: • Jenkins IRC Channel ◦ https://jenkins.io/chat/ •

    Jenkins Developer and User mailing lists ◦ https://jenkins.io/mailing-lists/
  23. 23 JEP-200 for Plugin Maintainers

  24. Class Deserialization 101 public class Foo implements Serializable { private

    String command; private Object readResolve() { Process p = Runtime.getRuntime().exec(command); return this; } } Details / Real examples: • https://www.slideshare.net/codewhitesec/exploiting-deserialization-vulnerabilities-in-java-54707478 • https://www.christian-schneider.net/JavaDeserializationSecurityFAQ.html • https://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-appl ication-have-in-common-this-vulnerability/ 24
  25. In the Last 90 Days JEP-200 Maintainers... • Tested more

    than 100 plugins • Facelifted >70 plugins • Plugin refreshes (POM update, etc.) • Support for Plugin Compatibility Tester • Fixed >50 plugins (Wiki) • JEP-200 exposes structural mistakes in plugins/core that could affect security, performance, upgradability, etc. © Jesse • Some collateral damage 25
  26. JEP-200. What may break? • Classes from 3rd-party libraries •

    HOT: Model objects ▸ MavenInformation from Maven core - Maven Plugin ▸ 3rd-party TestResult classes - TestLink/TAP Plugin • Non-whitelisted Java classes • https://github.com/jenkinsci/jenkins/blob/master/core/src/m ain/resources/jenkins/security/whitelisted-classes.txt • E.g. Calendar or DateFormatter are not whitelisted • Non-whitelisted classes from Jenkins core libs • E.g. Guava Collections 26
  27. How to test JEP-200? Classic approach: • Use Plugin Compat

    Tester (aka PCT) • Run Acceptance Test Harness • Check plugins on Test servers (if possible) 27
  28. PCT. Quick Start docker run --rm -v maven-repo:/root/.m2 -v $(pwd)/out:/pct/out

    -v jenkins-2.107.1-rc.war:/pct/jenkins.war:ro -v plugin-sources/ssh-slaves-plugin:/pct/plugin-src:ro -e ARTIFACT_ID=ssh-slaves jenkins/pct More documentation: https://github.com/jenkinsci/plugin-compat-tester 28
  29. PCT. Checking the report DEMO: PCT Report Look for: •

    Stacktraces “https://jenkins.io/redirect/class-filter/” references • Regression against current baselines 29
  30. What to test? XStream • All classes being persisted on

    the disk • Hot areas: • Run/Project actions: data stored by plugins • Improperly cached objects (no “transient” / “static”) • Historic data is a subject for loading issues • May even cause DoS :( 30
  31. What to test? Remoting • HOT: All callables, especially anonymous

    inner classes • MasterToSlaveCallable<Type> • MasterToSlaveFileCallable<Type> • hudson.remoting.Callable in old plugins • SlaveToMasterCallable • Not just a single type whitelist • Type AND fields should be whitelisted: ▸ Implementation classes ▸ All upstream abstract classes • Passing final variables to anonymous callable classes 31