Securing the Open Source Software Supply Chain

Popular app frameworks like Docker, NodeJS, React, Ruby on Rails, & Grails (Java) have thousands of dependencies. Devs routinely add dependencies that may be unsafe. In this talk, you will learn how malicious actors use open source libraries to attack and a proactive dependency management approach.

Watch video at https://www.youtube.com/watch?time_continue=1&v=dVc8qv991kY&feature=emb_logo.

Transcript

1. Securing the Open Source Software Supply Chain Trusting third party code, CVEs, common attacks, and keeping up-to-date safely Frank S. Rietta, MS Information Security CEO of Rietta.com, a Security Focused Web Application Firm
2. 2 @FrankRietta #HellaConf What we’ll cover today • Challenges with complex dependency trees • Why companies take 34+ days to patch known vulnerabilities in production, leaving time to get hacked • Review of incidents of malicious code in RubyGems and NPM ecosystems and lessons that apply to any software project – Three types of attacks on distribution – Checking hashes is not a panacea – Code signing is not a panacea • How test driven development & continuous integration / continuous deployment ease many issues with the supply chain • Practical development steps to reduce risk from dependencies
3. 3 @FrankRietta #HellaConf Challenges • All software depends on more third party code • Deployments into cloud environments – introduce additional entanglements, often proprietary and not auditable – Move traditional threat boundaries • Lack of comprehensive tools to track assets among many disparate programming languages and operating systems makes it hard to centrally manage risk at the Enterprise level • Open source software is often the world class solution, but – Who is reviewing the code? – How are developers to ensure they are using the right package? • Dependencies means the need to redeploy/patch production frequently
4. 4 @FrankRietta #HellaConf Many Dependencies • Before a dev does any work at all, there are: – 64 Java run time libraries in a new Grails 4 project – 75 Gems in a brand new Rails 6 project – 996 node modules in a brand new React project • These libraries are produced by volunteer open source maintainers, many of whom are not paid anything to work on open source
5. 5 @FrankRietta #HellaConf Many Sources for Packages • Language-specific package manager – Node Package Manager (NPM) – JavaScript – NuGet – Microsoft .NET languages – Maven – Java – RubyGems – Ruby / Ruby on Rails • Native dependencies in OS distribution – apt (Debian-based Linuxes) and rpm (Red Hat / CentOS) – Microsoft Window’s various mechanisms – Java ecosystem • Docker images that contain dependencies • Cloud-based dependencies • Proprietary packages with ad hoc installations
6. 6 @FrankRietta #HellaConf Make Sure It’s Worth It • Open source dependencies are not cost free – Every dependency is an entanglement with code outside of your control that introduces brittleness – For very simple functionality, it’s often better to write the code yourself in your own project rather than include a third party library • For packages that are sensible to use, contribute – Reviews of the code looking for issues – Code back to the project – Financial support to the maintainers so that they are able to keep the package maintained in the future
7. 7 @FrankRietta #HellaConf Pillars of Dependency Management Minimize Dependencies During Development Contribute to Critical Open Source Infrastructure Monitor Packages for CVEs & Updates Patch & Update Regularly Write Automated Integration Tests Implement CI/CD
8. 8 @FrankRietta #HellaConf Why Companies are Slow at Patching Production
9. 9 @FrankRietta #HellaConf Patching production takes too long Severity Level Average Days to Patch All (Regardless of Severity) 38 days High 34 days Medium 39 days Low 54 days Oldest unpatched CVE 340 days Source: TCell 2018
10. 10 @FrankRietta #HellaConf Reasons for Slow Patching • No one is around to do any work • Change is hard • Change breaks things • Don’t know all the dependencies! • Lack of confidence • It’s a manual process
11. 11 @FrankRietta #HellaConf Key Observation for the Future A computer should deploy to production without human intervention after all the automated tests pass.
12. 12 @FrankRietta #HellaConf Security Lessons from Ruby Language Dependencies via RubyGems.org
13. 13 @FrankRietta #HellaConf What are Ruby Gems? • Gems are Ruby language libraries, often open source, and distributed by https://RubyGems.org. • Ruby developers reference Gems for their projects in a Gemfile manifest that functions the same as package.json in NPM. • A package manager called bundler automatically resolves and installs dependencies, downloading unattended from rubygems.org and other sources.
14. 14 @FrankRietta #HellaConf Attack Patterns Against Gems • Observed Attacks – Backdoors in Popular Gems Distributed via Compromised Maintainer Credentials – Malicious Typo-Squatted / Soundalike Gems • Theoretical Attack – Maintainership Transfer to a Malicious Actor • Happen with event-stream, a Node package, in 2018.
15. 15 @FrankRietta #HellaConf Backdoors in Popular Gems Distributed via Compromised Maintainer Credentials
16. 16 @FrankRietta #HellaConf Incidents of Malicious Code in Popular Mainstream Ruby Gems in 2019 Month Gem Version CVE CVSS Score April 2019 bootstrap-sass 3.2.0.3 CVE-2019 -10842 10 July 2019 strong_password 0.0.7 CVE-2019 -13354 7.5 August 2019 rest-client 1.6.10, 1.6.11, 1.6.12, 1.6.13, 1.6.14 CVE-2019 -15224 7.5 In these three cases, credential theft was involved and if the maintainers had 2FA enabled then the attack would have not succeeded in the way it did. Source: https://rietta.com/blog/rubygems-supply-chain-vulnerability
17. 17 @FrankRietta #HellaConf Rest-client Malicious Code Actions • sent the URL of the infected host to the attacker. • sent the environment variables of the infected host to the attacker. Depending on your set-up this can include credentials of services that you use e.g. database, payment service provider. • allowed to eval Ruby code on the infected host. Attacker needed to send a signed (using the attacker’s own key) cookie with the Ruby code to run. • overloaded the #authenticate method on the Identity class. Every time the method gets called it will send the email/password to the attacker. However I’m unsure which libraries use the Identity class though, maybe some Source: https://github.com/rest-client/rest-client/issues/713
18. 18 @FrankRietta #HellaConf Interview with rest-client maintainer According to Matt Manning, the maintainer of rest-client who’s credentials were used in the attack: “I probably hadn’t logged into the rubygems web UI since 2011⁄2012. I don’t know if they had 2fa back then, and I wasn’t disciplined about using a password manager then. I use 1password now, but that login was so old that I didn’t even have it in 1pass, so I didn’t catch it when I audited dupes, etc there. I probably haven’t pushed a public gem since 2014. I guess my api key was cached for that.” Source: https://rietta.com/blog/rest-client-cve
19. 19 @FrankRietta #HellaConf Malicious Typo-Squatted / Soundalike Gems
20. 20 @FrankRietta #HellaConf Typo Squatter / Sound-a-likes • Targets new projects where a developer is searching for the Gem to include, not having already referenced the legitimate one in their Gemfile – Clones the legitimate Gem and then publishes a variant with a similar name to cause confusion – Duplicates aspects of the cloned Gem’s meta data – Massive abuse of confusion between – and _ in Gem names • Targets older Gems or major versions that have not been published in some time • Difficult to distinguish from legitimate open source activities, such as forked projects
21. 21 @FrankRietta #HellaConf Reversing Labs 4/16/2020 • Reversing Labs recently published a data set about a massive 725 typo RubyGem attack in February 2020. • All victim Gems had significantly fewer than 10,000,000 downloads because RubyGems.org computed the levenshtein string distance between all Gems and rejected ones too close to protected Gems. • RubyGems.org removed this protection on March 7, 2020, because of issues with false positives.
22. 22 @FrankRietta #HellaConf Download Popularity Analysis of February 2020 Attack 100 or Less 101 - 1000 1001 - 3000 3001 - 5000 5001 - 7000 4001 - 9999 10k - 20k 20k to 9 mil 9 mil to 10 mil Protected / 10 mil+ 0 20 40 60 80 100 120 140 160 180 200 Typo Squatted Gem Download Counts in Reversing Labs 4/16/2020 Data Set Count Number of Victim Gem Downloads (it's Popularity) Number of Malicious Gems
23. 23 @FrankRietta #HellaConf Dash/Underscore Confusion Dash/Underscore (-/_) Confusion Names capitalizing on variations of - and _ to confuse victims. True False
24. 24 @FrankRietta #HellaConf Look at Multi Character Variations Malicious Name Victim Gem Levensh tein Victim Download Count active-model_serializers-matchers active_model_serializer-matchers 2 14,868 actionview-link-to_block actionview-link-to_blank 2 145 active-pubsub action-pubsub 2 144 a1437ky-bmi3 a1437ky_bmi2 2 84 font_awesome-sass font_awesome-less 2 74 fluent_plugin-datadog-statsd fluent-plugin-datatdog-statsd 2 17 Note that a1437ky-bmi3 was a clone of a Gem really named a1437ky-bmi3, but it also matches a variant with _ and version 2. Typo detection is tricky business.
25. 25 @FrankRietta #HellaConf Age of legitimate Gem when Malicious Typo-squat was Published Based on my own analysis of publicly available data, with the exception of awesome- bot and coin_base, none of these typo-squatted Gems had been updated in years. Malicious Gem Victim Gem Victim Downloads (April 2020 Data) Age Since Legitimate Update (Sept 2019 Data) capistrano-colors capistrano_colors 14,17,288 7 years 10 mons 11 days doge-coin doge_coin 8,004 5 years 5 mons 7 days lita_coin lita-coin 78,594 5 years 4 mons 19 days blockchain_wallet blockchain-wallet 11,702 5 years 3 mons 21 days omniauth_amazon omniauth-amazon 296,014 3 years 4 mons 3 days coming-soon coming_soon 4737 2 years 2 mons 13 days coin_base coinbase 125,634 5 mons 15 days awesome-bot awesome_bot 253,789 -1 mons -1 days
26. 27 @FrankRietta #HellaConf Lessons • Checking file hashes only provides marginal protection when a malicious library is published in a legitimate channel • Code signing is supported by RubyGems but not in common usage like it is in other environments like Maven. However, in open source code signing does not address legitimate forks of software or typo squatters • When depending on an open source library, you’re depending on the code quality and the opsec of volunteer contributors • We’ve been lucky that eagle eyed individuals have noticed these malicious Gems; there is minimal systematic protection in the RubyGems.org infrastructure
27. 28 @FrankRietta #HellaConf Test Driven Development & Continuous Integration / Continuous Deployment
28. 29 @FrankRietta #HellaConf Developers Write Tests “It is irresponsible for a developer to ship a line of code that [he or she] has not executed any unit test for, and one of the best ways to make sure you have not shipped a line of code without testing is to practice TDD” Robert Martin in his 2012 debate on TDD with Jim Coplien, video at https://www.youtube.com/watch?v=KtHQGs3zFAM&t=14%3A42
29. 30 @FrankRietta #HellaConf The Three Rules of TDD 1)Don’t write a line of production code without having a corresponding failing test 2)Don’t write too many failing tests without writing production code 3)Don’t write more production code than is sufficient to pass the currently failing test Source: Robert Martin http://butunclebob.com/ArticleS.UncleBob.TheThreeRulesOfTdd (2005)
30. 31 @FrankRietta #HellaConf Types of Tests • Experimenting in the interactive debugger • Two general categories of automated tests – Integration Testing • User Interface / Browser Tests for Web UI testing as a human sees it • Request Tests (Interaction with Full Web Server) for API • Controller Tests – Unit Testing • Automated code analysis (SAST) for security and code style • Unit / Model • View tests • Feedback from production via Exception monitoring
31. 32 @FrankRietta #HellaConf Source: https://twitter.com/d3veducation/status/813056723271450624
32. 34 @FrankRietta #HellaConf
33. 35 @FrankRietta #HellaConf Changing Software to Fix Defects • When there is a security defect, how do the developers change the code without breaking the software? • Two industry approaches: – Edit and Pray – Cover and Modify
34. 36 @FrankRietta #HellaConf Feather's Legacy Code Change Process When you have to make a change in a legacy code base: 1)Identify change points. 2)Find test points. 3)Break dependencies. 4)Write tests. 5)Make changes and refactor Feathers, Michael. Working Effectively with Legacy Code (Robert C. Martin Series) (Kindle Locations 624-628). Pearson Education. Kindle Edition.
35. 37 @FrankRietta #HellaConf Team Requirements for CI/CD • Developers write detailed automated tests • Containerized deployment runs tests on demand • Configure tests to run for each pull request • Make Master branch protected so that tests MUST PASS before code can be merged into it • Another dev reviews code and asserts that the tests seem on-the-face to cover intended functionality
36. 38 @FrankRietta #HellaConf Pull Request Driven Git Flow Source: https://medium.com/bdo-digital/git-workflows-99f7174fe461
37. 40 @FrankRietta #HellaConf Protect the Master Branch
38. 41 @FrankRietta #HellaConf CI is Part of the Code Review Process
39. 42 @FrankRietta #HellaConf Automated Security Tests are an Acceptance Criteria When automated CVE checking or SAST tests fail, then the pull request fails code review and cannot be merged into the Master branch.
40. 43 @FrankRietta #HellaConf Practical Dev Steps to Reduce Risk • Minimize dependency on third party libraries – Every dependency is an entanglement with code outside of your control that introduces brittleness – For very simple functionality, it’s often better to write the code yourself in your own project rather than include a third party Gem • Run Common Vulnerability (CVE) checking and Static Analysis (SAST) tools and make them part of your CI/CD
41. 44 @FrankRietta #HellaConf Practical Dev Steps to Reduce Risk • Carefully review libraries that you do choose to include – consider the age & number of downloads – search for variations and see if it appears to be squatting on a typo for a more popular package • Pay extreme attention to variations of – and _ characters; • Or a new appended number – there is some safety in waiting to upgrade to a new release unless you must upgrade to patch a known CVE that is reported
42. 45 @FrankRietta #HellaConf Practical Dev Steps to Reduce Risk • In Ruby: – Avoid using the ~>, >=, and > operators in your Gemfile, preferring to instead specific specific versions of each Gem – Update specific Gems intentionally like bundle update nokogiri and avoid bulk updating via running bundle update
43. 46 @FrankRietta #HellaConf Principles Apply to All Platforms • Software update supply chain attacks have been a big trend in cybercrime in recent years and is expected to grow in frequency. • It is important to know that just because a library is trustworthy today does not mean the next published version will not contain vulnerabilities or malicious code. • It is best to wait a little while before upgrading to the absolute latest release of a library to see if others detect an issue unless, and this is important, upgrading is necessary to fix a known published vulnerability. • The same is true for Node NPM and any other package ecosystems from which you obtain software dependencies.
44. 47 @FrankRietta #HellaConf Questions?
45. 48 @FrankRietta #HellaConf Thank you! • Find Me on Social Media – @FrankRietta on Twitter – @fsrietta on LinkedIn • Subscribe to my Blog and Newsletter – The Rietta Blog (since 2005) @ https://rietta.com/blog – Rietta on Security (monthly newsletter) at https://rietta.com/on-security • If you see me in person, ask for a “Code Review before Pen Test” sticker!
46. 49 @FrankRietta #HellaConf Book Recommendations • Feathers, Michael. Working Effectively with Legacy Code – Amazon – Barnes and Noble • Fowler, Martin. Refactoring: Improving the Design of Existing Code (2nd Edition) – Amazon – Barnes and Noble • Metz, Sandi & Owen, Katrina. 99 Bottles of OOP. – https://www.sandimetz.com/99bottles • Martin, Robert. Clean Architecture: A Craftsman's Guide to Software Structure and Design – Amazon – Barnes & Noble