Data driven dependency management

Analysis of 10 years of RubySec data and 50,000 dependency upgrade PRs

Transcript

1. Data driven dependency management Analysing 10 years of RubySec data, and 50,000 upgrade PRs
2. • Grey Baker (@greysteil on GitHub, @greybaker on Twitter) • 6 years programming Ruby • Bundler core team member (since August!) • 52,000 gem update PRs merged in the last 12 months
3. • Dependabot automatically creates dependency update PRs • I built it, and work on it full-time • GoCardless, GOV.UK and thoughtbot use it. You should too • Creates interesting data since it updates one gem at a time
4. What’s the best strategy for keeping your gems secure?
5. Always up-to-date Late adopter Reactive STRATEGY 1 • Use the latest version at all times • For analysis assume they’re always 1 day behind STRATEGY 2 • Wait a month after new major or minor versions • Always on the latest patch release STRATEGY 3 • Use the latest version when you add, then forget • Equally likely to have added or updated the gem at any time in the last year Three common gem strategies
6. • 10 years of vulnerability advisories • Includes the date of disclosure • Allows us to determine whether a given version is secure The data to test them Ruby Advisory Database • History of gem releases over time • Includes date of each release • Tells us which version each strategy would have been using at any time Rubygems API • Combining, we can tell what version each strategy would have been on at the time a vulnerability was disclosed, and whether that version is secure • Full analysis at https://github.com/dependabot/gem-vulnerability-analysis
7. Always keeping your gems up-to- date is the most secure strategy (just) Required response to a new vulnerability disclosure Always up-to-date Late adopter Reactive 47% 47% 47% 3% 4% 4% 42% 35% 35% 8% 15% 16% Not affected Fix by upgrading Fix by downgrading No fix on day zero • In addition, remember, the late adopter has a harder job with “fix by upgrading” • Full analysis at https://github.com/dependabot/gem-vulnerability-analysis
8. How buggy / incompatible are new gem versions?
9. Dependabot listens for the CI results of updates, so we have data!
10. Major update a.b.c -> x.0.0 Minor update x.b.c -> x.y.0 Patch x.y.c -> x.y.z • Based on CI info for 1,750 updates • Many major updates are just dropping old Ruby support, etc. Minor and patch versions rarely have any incompatibilities or new bugs • Based on CI info for 12,000 updates • Pre-1.0.0 updates excluded • Based on CI info for 17,000 updates • Pre-1.0.0 updates excluded Updates with passing CI Notes • SemVer might not work in theory, but it does work in practice • Minor updates are nearly as easy to upgrade as patch releases
11. • Thanks for listening! • Security analysis available at: https://github.com/dependabot/gem-vulnerability-analysis • Check out Dependabot at: https://dependabot.com