Keeping the Gems Sparkling

Transcript

1. Keeping The Gems Sparkling Samuel Giddins 1
2. Your intrepid presenter @segiddins Samuel Giddins RubyGems, Bundler, RubyGems.org maintainer 10+ year bug contributor Samuel Giddins 2
3. Security Engineer in Residence at Ruby Central Samuel Giddins 3
4. 1. Entertain you all, who make this ecosystem so vibrant 2. Shock you as much as possible with big numbers 3. Give you slides to send to your boss as fuel to ask for more support for our work Too honest? Sorry. I'll just lean into #1 then 😅 Goals Samuel Giddins 4
5. 1. Bring industry best practices into the Ruby ecosystem 2. Keep Ruby a vibrant community used by everyone, from the hobbyist to the Fortune 500 3. Convince you to do things that make me look successful to my boss 4. See midnight sun Goals Samuel Giddins 5
6. It's really big! No, like, really big! RubyGems.org Samuel Giddins 6
7. Show of hands if you've ever run gem install Some Q&A Samuel Giddins 7
8. Show of hands if you're ever put source "https://rubygems.org" in a Gemfile Some Q&A Samuel Giddins 8
9. Guess What You've helped contribute to the traffic that RubyGems.org serves Samuel Giddins 9
10. 10 - 11 million unique IPs per month 23,302,340,473 requests per month 20,000 peak Requests per Second 167,947,962,107 gem downloads RubyGems.org Stats Samuel Giddins 10
11. RubyGems.org Stats Samuel Giddins 11
12. Story Time aka the weekend sam got paged aka the time we hit 225k rps Samuel Giddins 12
13. Story Time Samuel Giddins 13
14. RubyGems.org Stats Samuel Giddins 14
15. What have we been up to? Samuel Giddins 15
16. Big Stuff Trusted Publishing OIDC GitHub Actions for secure gem build & push Bundler Checksums (in alpha) importmaps + Stimulus.js What have we been up to? Samuel Giddins 16
17. Upcoming Redesign! Sigstore Organizations Namespacing What have we been up to? Samuel Giddins 17
18. Small Stuff We removed avatars from rubygems.org …for privacy (using gravatar leaks user emails) then we put them back (proxied through rubygems.org) Up to 8x the speed for gem installs and pristine What have we been up to? Samuel Giddins 18
19. Day-to-Day What have we been up to? Samuel Giddins 19
20. Deploying (Dis)continuous Manual Deployment Multiple times per day (or week if I forget...) Zero Downtime Absolutely Zero, no questions asked, no excuses, end of discussion 0-locking database migrations & upgrades Watch for more info from @simi for how he pulled this off repeatedly with PGBouncer and RDS What have we been up to? Samuel Giddins 20
21. 24x7 Oncall Follow the sun Coordinating between Australia, India, Czech Republic, Spain, USA Sometimes staying up late & waking up early so we get some face time What have we been up to? Samuel Giddins 21
22. Supported by our generous sponsors AWS gives us $165,000/year in credits (offsetting infrastructure costs) Fastly gives us $1,000,000/year in donated services (estimated at retail rate) DataDog donates monitoring services Honeybadger provides error tracking All this is... $$$$$ Samuel Giddins 22
23. Supported by donations to Ruby Central Shopify $1 million over 4 years to support the security & reliability of RubyGems & RubyGems.org Directly funds Open Source team work German Sovereign Technology Fund €863,000 over about 2 years Funded general maintenance & security-focused improvements All this is... $$$$$ Samuel Giddins 23
24. Supported by donations to Ruby Central OpenSSF Alpha-Omega $100,000 security audit $150,000 to add organizations to RubyGems.org AWS Credits Sponsor my role as Security Engineer in Residence Pay for me to be here today! All this is... $$$$$ Samuel Giddins 24
25. 5 engineers on staff All part-time except me (thanks AWS!) This is a second job for most of my teammates All this requires work Samuel Giddins 25
26. A close-knit support team from other companies @hsbt @ AndPad & Ruby Core Eric * Jenny @ Shopify Maciej @ Mend.io All this requires work Samuel Giddins 26
27. Individual contributors like YOU! We receive support, feedback, and Pull Requests from all around the Ruby Community! Thank you! All this requires work Samuel Giddins 27
28. Expertise is hard-won I've been working on Bundler & RubyGems for 10 years We want to retain our staff Financial support ebbs and flows All this requires work Samuel Giddins 28
29. Infrastructure always falls behind Samuel Giddins 29
30. New versions of: PostgresQL Kubernetes Memcached Opensearch Infrastructure always falls behind Samuel Giddins 30
31. Constant deprecations inside AWS Infrastructure always falls behind Samuel Giddins 31
32. New versions of gems (& Rails) RubyGems.org is one continually maintained app since 2009 Started as a Sinatra app on Ruby 1.8 Converted to Rails 2, then 3, 4, 5, 6, and 7 A canonical open source Rails app Infrastructure always falls behind Samuel Giddins 32
33. New best practices Don't use that, use this sprockets propshaft webpacker importmaps sass tailwind Infrastructure always falls behind Samuel Giddins 33
34. Keeping up with RubyGems & Bundler improvements RubyGems.org compact index makes dependency resolution faster Developed in coordination with RubyGems.org Infrastructure always falls behind Samuel Giddins 34
35. Constant security concerns Samuel Giddins 35
36. Software Supply Chain Security Constant security concerns Samuel Giddins 36
37. Governments now care about this. See: The US Government saying C is bad Constant security concerns Samuel Giddins 37
38. The security of software used by the Government is vital to the Government’s ability to perform its critical functions. The development of commercial software often lacks transparency, sufficient focus on the ability of the software to resist attack, and adequate controls to prevent tampering by malicious actors. There is a pressing need to implement more rigorous and predictable mechanisms for ensuring that products function securely, and as intended. Samuel Giddins 38
39. The security and integrity of “critical software” — software that performs functions critical to trust (such as affording or requiring elevated system privileges or direct access to networking and computing resources) — is a particular concern. Accordingly, the Government must take action to rapidly improve the security and integrity of the software supply chain, with a priority on addressing critical software. Samuel Giddins 39
40. Like it or not, our random RubyGems have been critical commercial software, according to the US Government How do we deal with this reality? Constant security concerns Samuel Giddins 40
41. 430 HackerOne reports Responsible disclosure & bug bounty program Each report takes time to triage Some reports require a lot of testing 64 real issues fixed So, so, so many reports that are... useless Constant security concerns Samuel Giddins 41
42. CISA, US Dept Homeland Security Coordination on publishing of Principles for Package Repository Security Constant security concerns Samuel Giddins 42
43. Starting the process of a 3rd-party audit Scoping work Staying within a very limited budget Coordinating on verification of remediation Constant security concerns Samuel Giddins 43
44. Attacks Malicious gems published Typo squatting Dependency confusion Data exfiltration Constant security concerns Samuel Giddins 44
45. Attacks ? Hey, why is the site getting so many 500s right now? Why am I getting paged? Why is there one IP making 10k requests per second? Why is the site down? Oh, a security researcher is literally hitting every endpoint for every gem on the whole system Added a missing index, asked the researcher to use the DB dumps instead Constant security concerns Samuel Giddins 45
46. RubyGems Research Every gem, every file, indexed Full-text search File-level diffs between versions Fast response to xz Search every gem for anything related to xz or liblzma Constant security concerns Samuel Giddins 46
47. Sigstore sign. verify. protect. Making sure your software is what it claims to be. Improvements Samuel Giddins 47
48. Trusted Publishing Automate publishing gems from CI No more persistent credentials No more 2fa dance Improvements Samuel Giddins 48
49. Trusted Publishing Come set it up for __your__ gems at my campfire this afternoon! Improvements Samuel Giddins 49
50. This is all made possible by contribution from users like you, your companies, and security-minded organizations like the German government (STF), OpenSSF (Alpha-Omega), AWS, Shopify, and more. Samuel Giddins 50
51. This talk is made possible by the support of my wife, who has allowed me to spend the week after our wedding in Scandinavia. Samuel Giddins 51
52. Samuel Giddins 52
53. Sorry it isn't warmer. Samuel Giddins 53
54. Thank you @segiddins Security Engineer in Residence @ Ruby Central Samuel Giddins 54