Your intrepid presenter
@segiddins
Samuel Giddins
RubyGems, Bundler, RubyGems.org maintainer
10+ year bug contributor
Samuel Giddins 2
Slide 3
Slide 3 text
Security Engineer in Residence at Ruby Central
Samuel Giddins 3
Slide 4
Slide 4 text
Goals
1. Shed light on the history of vulnerabilities in a well-tested piece of
infrastructure
2. Make you aware of big bad dangerous world out there
3. Give you nightmare fuel to send to your boss as fuel to ask for more
support for our work
Too honest? Sorry. I'll just lean into #1 then
đ
Samuel Giddins 4
Slide 5
Slide 5 text
A Survey of RubyGems CVEs
Samuel Giddins 5
Slide 6
Slide 6 text
So, what's a CVE?
Samuel Giddins 6
Slide 7
Slide 7 text
So, what's a CVE?
C ommon
V ulnerabilities
and
E xposures
Samuel Giddins 7
Slide 8
Slide 8 text
So, what's a CVE?
The mission of the CVEÂŽ Program is to identify, define,
and catalog publicly disclosed cybersecurity
vulnerabilities.
Samuel Giddins 8
Slide 9
Slide 9 text
So, what's a CVE?
CVE Identifier (CVE ID)
An alphanumeric string that identifies a Publicly Disclosed
vulnerability.
Samuel Giddins 9
Slide 10
Slide 10 text
So, what's a CVE?
An instance of one or more weaknesses in a Product that
can be exploited, causing a negative impact to
confidentiality, integrity, or availability; a set of conditions
or behaviors that allows the violation of an explicit or
implicit security policy.
Weakness: something bad
Exploited: it's bad unintentionally!
negative impact: what makes it bad
allows the violation of an explicit or implicit security policy: it breaks a
promise about how the Product works
Samuel Giddins 10
Slide 11
Slide 11 text
Some Q&A
Show of hands if you've ever run gem install
Samuel Giddins 11
Slide 12
Slide 12 text
Some Q&A
Show of hands if you're ever put
source "https://rubygems.org"
in a Gemfile
Samuel Giddins 12
Slide 13
Slide 13 text
Guess What
You've used the "products" that are RubyGems, Bundler,
and RubyGems.org, and their CVEs could affect you.
Samuel Giddins 13
Slide 14
Slide 14 text
Guess What
RubyGems, like any sufficiently-used piece of software, has its fair share of
bugs.
Samuel Giddins 14
Slide 15
Slide 15 text
Guess What
Being a package manager (and gem host), many of those bugs turn out to
have security implications.
Samuel Giddins 15
Slide 16
Slide 16 text
The first RubyGems CVE
CVE-2007-0469
The extract_files function in installer.rb in
RubyGems before 0.9.1 does not check whether files exist
before overwriting them, which allows user-assisted
remote attackers to overwrite arbitrary files, cause a
denial of service, or execute arbitrary code via crafted
GEM packages.
Samuel Giddins 16
Slide 17
Slide 17 text
CVE-2007-0469 manifested a common weakness:
Directory traversal
Fixed via
Improved input validation
Samuel Giddins 17
Slide 18
Slide 18 text
In this case
RubyGems expected files in a gem to all be under the gem's directory
and not be absolute paths
or ..
RubyGems didn't check that assumption when unpacking gems
Samuel Giddins 18
Slide 19
Slide 19 text
Aside
Our first CVE was from 2007.
RubyGems (and associated projects) have been around a while.
In part, they date to an earlier, kinder era of the internet.
Samuel Giddins 19
Slide 20
Slide 20 text
Our Worst CVE
January 30, 2013
RubyGems.org went down for multiple days after an uploaded exploit gem
got arbitrary remote code execution
Samuel Giddins 20
Slide 21
Slide 21 text
Funny enough, it didn't even get a CVE ID.
The root cause was CVE-2013-0156 aka the great YAML RCE vuln.
Everyone was too busy fixing & verifying nothing was tampered with.
And also rebuilding 100% of the RubyGems.org infrastructure. Ooops.
Samuel Giddins 21
Slide 22
Slide 22 text
Almost half of that response
(and all future RubyGems.org CVE responses)
was spent on verifying that (there is no evidence that)
the vulnerability was exploited.
Samuel Giddins 22
Slide 23
Slide 23 text
Check checksums of files in S3, make sure they didn't change
Use trusted mirrors
Use SHAs stored in the RubyGems.org database & database dumps
Verify there were no (other) instances matching the pattern of the
vulnerability
No rogue YAML
No published versions matching the bad pattern
No failed requests matching the bad pattern
Samuel Giddins 23
Slide 24
Slide 24 text
I've run this playbook a dozen times since joining the RubyGems security
team.
It's been around since January 2013.
Samuel Giddins 24
Slide 25
Slide 25 text
Our Most Common Weaknesses
Samuel Giddins 25
Slide 26
Slide 26 text
RubyGems
Directory traversal unpacking gems
tar entry filenames
tar entry symlinks
name / version / platform
Symlink directory traversal unpacking gems
Arbitrary YAML deserialization
Samuel Giddins 26
Slide 27
Slide 27 text
Terminal control character injection
XSS in embedded servers
DOS
ReDOS
Making the client "sanitize" incredibly long strings
Negative numbers in tar files
Samuel Giddins 27
Slide 28
Slide 28 text
Bundler
Source/dependency confusion
you download a public gem instead of your private one
lack of namespacing
Shell injection
git CLI
Everything upstream from RubyGems
Samuel Giddins 28
Slide 29
Slide 29 text
RubyGems.org
Content overwriting / cache poisoning
name / version / platform
Access control bypass
Using full-name collisions to yank other people's gems
Abandoned email squatting
Reset password without MFA
Arbitrary YAML deserialization
Samuel Giddins 29
Slide 30
Slide 30 text
DOS
ReDOS
Tar bombs
YAML bombs
Negative numbers in tar files
Samuel Giddins 30
Slide 31
Slide 31 text
How do we handle this reality?
Samuel Giddins 31
Slide 32
Slide 32 text
Constant security concerns
Samuel Giddins 32
Slide 33
Slide 33 text
Constant security concerns
SoftwareSupply ChainSecurity
Samuel Giddins 33
Slide 34
Slide 34 text
Constant security concerns
Governments now care about this.
See: The US Government saying C is bad
Samuel Giddins 34
Slide 35
Slide 35 text
The security of software used by the Government is vital
to the Governmentâs ability to perform its critical
functions. The development of commercial software often
lacks transparency, sufficient focus on the ability of the
software to resist attack, and adequate controls to
prevent tampering by malicious actors. There is a pressing
need to implement more rigorous and predictable
mechanisms for ensuring that products function securely,
and as intended.
Samuel Giddins 35
Slide 36
Slide 36 text
The security and integrity of âcritical softwareâ â
software that performs functions critical to trust (such as
affording or requiring elevated system privileges or direct
access to networking and computing resources) â is a
particular concern. Accordingly, the Government must take
action to rapidly improve the security and integrity of the
software supply chain, with a priority on addressing
critical software.
Samuel Giddins 36
Slide 37
Slide 37 text
Constant security concerns
Like it or not, our random RubyGems have been critical commercial software,
according to the US Government
How do we deal with this reality?
Samuel Giddins 37
Slide 38
Slide 38 text
Constant security concerns
430 HackerOne reports
Responsible disclosure & bug bounty program
Each report takes time to triage
Some reports require a lot of testing
64 real issues fixed
So, so, so many reports that are... useless
Samuel Giddins 38
Slide 39
Slide 39 text
Constant security concerns
CISA, US Dept Homeland Security
Coordination on publishing of
Principles for Package Repository Security
Samuel Giddins 39
Slide 40
Slide 40 text
Constant security concerns
Completed a 3rd-party audit of RubyGems.org
Scoping work
Staying within a very limited budget
Coordinating on verification of remediation
Samuel Giddins 40
Slide 41
Slide 41 text
Constant security concerns
Attacks
Malicious gems published
Typo squatting
Dependency confusion
Data exfiltration
Samuel Giddins 41
Slide 42
Slide 42 text
Constant security concerns
Attacks ?
Hey, why is the site getting so many 500s right now?
Why am I getting paged?
Why is there one IP making 10k requests per second?
Why is the site down?
Oh, a security researcher is literally hitting every endpoint for every gem on
the whole system
Added a missing index, asked the researcher to use the DB dumps instead
Samuel Giddins 42
Slide 43
Slide 43 text
Constant security concerns
RubyGems Research
Every gem, every file, indexed
Full-text search
File-level diffs between versions
Fast response to xz
Search every gem for anything related to xz or liblzma
Samuel Giddins 43
Slide 44
Slide 44 text
We run an official security program
Bug Bounty / Reporting via HackerOner
hackerone.com/rubygems
[email protected]
24/7 oncall rotation
Get in touch privately if you suspect there's a vulnerability in
Bundler
RubyGems.org
RubyGems
Report issues with individual gems to their authors
Samuel Giddins 44
Slide 45
Slide 45 text
Vuln Lifecycle
Report
Triage
Stop the bleeding
Fix
Verify fix
Assess impact
Backfill fix
Disclose
Samuel Giddins 45
Slide 46
Slide 46 text
Improvements
Sigstore
sign. verify. protect.
Making sure your software is what it claims to be.
Samuel Giddins 46
Slide 47
Slide 47 text
All this is... $$$$$
Supported by our generous sponsors
AWS gives us $165,000/year in credits (offsetting infrastructure costs)
Fastly gives us $1,000,000/year in donated services (estimated at retail
rate)
DataDog donates monitoring services
Honeybadger provides error tracking
Samuel Giddins 47
Slide 48
Slide 48 text
All this is... $$$$$
Supported by donations to Ruby Central
Shopify
$1 million over 4 years to support the security & reliability of RubyGems
& RubyGems.org
Directly funds Open Source team work
German Sovereign Technology Fund
âŹ863,000 over about 2 years
Funded general maintenance & security-focused improvements
Samuel Giddins 48
Slide 49
Slide 49 text
All this is... $$$$$
Supported by donations to Ruby Central
OpenSSF Alpha-Omega
$100,000 security audit
$150,000 to add organizations to RubyGems.org
AWS
Credits
Sponsor my role as Security Engineer in Residence
Pay for me to be here today!
Samuel Giddins 49
Slide 50
Slide 50 text
Improvements
Trusted Publishing
Automate publishing gems from CI
No more persistent credentials
No more 2fa dance
Samuel Giddins 50
Slide 51
Slide 51 text
Improvements
Trusted Publishing
Come set it up for your gems with me this afternoon!
github.com/rubygems/configure_trusted_publisher
Samuel Giddins 51
Slide 52
Slide 52 text
This is all made possible by contribution from users like you, your companies,
and security-minded organizations like the German government (STF),
OpenSSF (Alpha-Omega), AWS, Shopify, and more.
Samuel Giddins 52
Slide 53
Slide 53 text
Thank you
@segiddins
Security Engineer in Residence @ Ruby Central
Samuel Giddins 53