A Survey of RubyGems CVEs

Transcript

1. A Survey of RubyGems CVEs Samuel Giddins 1
2. Your intrepid presenter @segiddins Samuel Giddins RubyGems, Bundler, RubyGems.org maintainer 10+ year bug contributor Samuel Giddins 2
3. Security Engineer in Residence at Ruby Central Samuel Giddins 3
4. <h3 style="view-transition-name:one;">Goals</h3> 1. Shed light on the history of vulnerabilities in a well-tested piece of infrastructure 2. Make you aware of big bad dangerous world out there 3. Give you nightmare fuel to send to your boss as fuel to ask for more support for our work <div data-marpit-fragment>Too honest? Sorry. I'll just lean into #1 then 😅 </div> Samuel Giddins 4
5. A Survey of RubyGems CVEs Samuel Giddins 5
6. <h1 style="view-transition-name:cve;"> So, what's a CVE?</h1> Samuel Giddins 6
7. <h3 style="view-transition-name:cve;"> So, what's a CVE?</h3> C ommon V ulnerabilities and E xposures Samuel Giddins 7
8. <h3 style="view-transition-name:cve;"> So, what's a CVE?</h3> The mission of the CVE® Program is to identify, define, and catalog publicly disclosed cybersecurity vulnerabilities. Samuel Giddins 8
9. <h3 style="view-transition-name:cve;"> So, what's a CVE?</h3> CVE Identifier (CVE ID) An alphanumeric string that identifies a Publicly Disclosed vulnerability. Samuel Giddins 9
10. <h3 style="view-transition-name:cve;"> So, what's a CVE?</h3> An instance of one or more weaknesses in a Product that can be exploited, causing a negative impact to confidentiality, integrity, or availability; a set of conditions or behaviors that allows the violation of an explicit or implicit security policy. Weakness: something bad Exploited: it's bad unintentionally! negative impact: what makes it bad allows the violation of an explicit or implicit security policy: it breaks a promise about how the Product works Samuel Giddins 10
11. <h3 style="view-transition-name:two;"> Some Q&A</h3> Show of hands if you've ever run gem install Samuel Giddins 11
12. <h3 style="view-transition-name:two;"> Some Q&A</h3> Show of hands if you're ever put source "https://rubygems.org" in a Gemfile Samuel Giddins 12
13. <h2 style="view-transition-name:guess;">Guess What</h3> <div data- marpit-fragment> You've used the "products" that are RubyGems, Bundler, and RubyGems.org, and their CVEs could affect you. </div> Samuel Giddins 13
14. <h2 style="view-transition-name:guess;">Guess What</h3> RubyGems, like any sufficiently-used piece of software, has its fair share of bugs. Samuel Giddins 14
15. <h2 style="view-transition-name:guess;">Guess What</h3> Being a package manager (and gem host), many of those bugs turn out to have security implications. Samuel Giddins 15
16. The first RubyGems CVE CVE-2007-0469 The extract_files function in installer.rb in RubyGems before 0.9.1 does not check whether files exist before overwriting them, which allows user-assisted remote attackers to overwrite arbitrary files, cause a denial of service, or execute arbitrary code via crafted GEM packages. Samuel Giddins 16
17. CVE-2007-0469 manifested a common weakness: Directory traversal Fixed via Improved input validation Samuel Giddins 17
18. In this case RubyGems expected files in a gem to all be under the gem's directory and not be absolute paths or .. RubyGems didn't check that assumption when unpacking gems Samuel Giddins 18
19. Aside Our first CVE was from 2007. RubyGems (and associated projects) have been around a while. In part, they date to an earlier, kinder era of the internet. Samuel Giddins 19
20. Our Worst CVE January 30, 2013 RubyGems.org went down for multiple days after an uploaded exploit gem got arbitrary remote code execution Samuel Giddins 20
21. Funny enough, it didn't even get a CVE ID. The root cause was CVE-2013-0156 aka the great YAML RCE vuln. Everyone was too busy fixing & verifying nothing was tampered with. And also rebuilding 100% of the RubyGems.org infrastructure. Ooops. Samuel Giddins 21
22. Almost half of that response (and all future RubyGems.org CVE responses) was spent on verifying that (there is no evidence that) the vulnerability was exploited. Samuel Giddins 22
23. Check checksums of files in S3, make sure they didn't change Use trusted mirrors Use SHAs stored in the RubyGems.org database & database dumps Verify there were no (other) instances matching the pattern of the vulnerability No rogue YAML No published versions matching the bad pattern No failed requests matching the bad pattern Samuel Giddins 23
24. I've run this playbook a dozen times since joining the RubyGems security team. It's been around since January 2013. Samuel Giddins 24
25. Our Most Common Weaknesses Samuel Giddins 25
26. RubyGems Directory traversal unpacking gems tar entry filenames tar entry symlinks name / version / platform Symlink directory traversal unpacking gems Arbitrary YAML deserialization Samuel Giddins 26
27. Terminal control character injection XSS in embedded servers DOS ReDOS Making the client "sanitize" incredibly long strings Negative numbers in tar files Samuel Giddins 27
28. Bundler Source/dependency confusion you download a public gem instead of your private one lack of namespacing Shell injection git CLI Everything upstream from RubyGems Samuel Giddins 28
29. RubyGems.org Content overwriting / cache poisoning name / version / platform Access control bypass Using full-name collisions to yank other people's gems Abandoned email squatting Reset password without MFA Arbitrary YAML deserialization Samuel Giddins 29
30. DOS ReDOS Tar bombs YAML bombs Negative numbers in tar files Samuel Giddins 30
31. How do we handle this reality? Samuel Giddins 31
32. <h1 style="view-transition-name:twp;">Constant security concerns</h1> Samuel Giddins 32
33. <h1 style="view-transition-name:twp;">Constant security concerns</h1> <span data-marpit-fragment>Software</span> <span data-marpit- fragment>Supply Chain</span> <span data-marpit- fragment>Security</span> Samuel Giddins 33
34. <h1 style="view-transition-name:twp;">Constant security concerns</h1> Governments now care about this. See: The US Government saying C is bad Samuel Giddins 34
35. The security of software used by the Government is vital to the Government’s ability to perform its critical functions. The development of commercial software often lacks transparency, sufficient focus on the ability of the software to resist attack, and adequate controls to prevent tampering by malicious actors. There is a pressing need to implement more rigorous and predictable mechanisms for ensuring that products function securely, and as intended. Samuel Giddins 35
36. The security and integrity of “critical software” — software that performs functions critical to trust (such as affording or requiring elevated system privileges or direct access to networking and computing resources) — is a particular concern. Accordingly, the Government must take action to rapidly improve the security and integrity of the software supply chain, with a priority on addressing critical software. Samuel Giddins 36
37. <h1 style="view-transition-name:twp;">Constant security concerns</h1> Like it or not, our random RubyGems have been critical commercial software, according to the US Government <div data-marpit-fragment> How do we deal with this reality? </div> Samuel Giddins 37
38. <h1 style="view-transition-name:twp;">Constant security concerns</h1> 430 HackerOne reports Responsible disclosure & bug bounty program Each report takes time to triage Some reports require a lot of testing 64 real issues fixed So, so, so many reports that are... <span data-marpit- fragment>useless</span> Samuel Giddins 38
39. <h1 style="view-transition-name:twp;">Constant security concerns</h1> CISA, US Dept Homeland Security Coordination on publishing of Principles for Package Repository Security Samuel Giddins 39
40. <h1 style="view-transition-name:twp;">Constant security concerns</h1> Completed a 3rd-party audit of RubyGems.org Scoping work Staying within a very limited budget Coordinating on verification of remediation Samuel Giddins 40
41. <h1 style="view-transition-name:twp;">Constant security concerns</h1> Attacks Malicious gems published Typo squatting Dependency confusion Data exfiltration Samuel Giddins 41
42. <h1 style="view-transition-name:twp;">Constant security concerns</h1> Attacks ? Hey, why is the site getting so many 500s right now? Why am I getting paged? Why is there one IP making 10k requests per second? Why is the site down? Oh, a security researcher is literally hitting every endpoint for every gem on the whole system Added a missing index, asked the researcher to use the DB dumps instead Samuel Giddins 42
43. <h1 style="view-transition-name:twp;">Constant security concerns</h1> RubyGems Research Every gem, every file, indexed Full-text search File-level diffs between versions Fast response to xz Search every gem for anything related to xz or liblzma Samuel Giddins 43
44. We run an official security program Bug Bounty / Reporting via HackerOner hackerone.com/rubygems security@rubygems.org 24/7 oncall rotation Get in touch privately if you suspect there's a vulnerability in Bundler RubyGems.org RubyGems Report issues with individual gems to their authors Samuel Giddins 44
45. Vuln Lifecycle Report Triage Stop the bleeding Fix Verify fix Assess impact Backfill fix Disclose Samuel Giddins 45
46. <h1 style="view-transition-name:two;">Improvements</h1> Sigstore <h3 data-marpit-fragment>sign. verify. protect.</h3> <div data-marpit- fragment> Making sure your software is what it claims to be. </div> Samuel Giddins 46
47. <h1 style="view-transition-name:two;">All this is... $$$$$</h1> Supported by our generous sponsors AWS gives us $165,000/year in credits (offsetting infrastructure costs) Fastly gives us $1,000,000/year in donated services (estimated at retail rate) DataDog donates monitoring services Honeybadger provides error tracking Samuel Giddins 47
48. <h1 style="view-transition-name:two;">All this is... $$$$$</h1> Supported by donations to Ruby Central Shopify $1 million over 4 years to support the security & reliability of RubyGems & RubyGems.org Directly funds Open Source team work German Sovereign Technology Fund €863,000 over about 2 years Funded general maintenance & security-focused improvements Samuel Giddins 48
49. <h1 style="view-transition-name:two;">All this is... $$$$$</h1> Supported by donations to Ruby Central OpenSSF Alpha-Omega $100,000 security audit $150,000 to add organizations to RubyGems.org AWS Credits Sponsor my role as Security Engineer in Residence Pay for me to be here today! Samuel Giddins 49
50. <h1 style="view-transition-name:two;">Improvements</h1> Trusted Publishing Automate publishing gems from CI No more persistent credentials No more 2fa dance Samuel Giddins 50
51. <h1 style="view-transition-name:two;">Improvements</h1> Trusted Publishing <div data-marpit-fragment> Come set it up for your gems with me this afternoon! </div> <div data-marpit-fragment> github.com/rubygems/configure_trusted_publisher </div> Samuel Giddins 51
52. This is all made possible by contribution from users like you, your companies, and security-minded organizations like the German government (STF), OpenSSF (Alpha-Omega), AWS, Shopify, and more. Samuel Giddins 52
53. Thank you @segiddins Security Engineer in Residence @ Ruby Central Samuel Giddins 53