Data driven dependency management Analysing 10 years of RubySec data, and 50,000 upgrade PRs 

• Grey Baker (@greysteil on GitHub, @greybaker on Twitter) • 6 years programming Ruby • Bundler core team member (since August!) • 52,000 gem update PRs merged in the last 12 months 

• Dependabot automatically creates dependency update PRs • I built it, and work on it full-time • GoCardless, GOV.UK and thoughtbot use it. You should too • Creates interesting data since it updates one gem at a time 

What’s the best strategy for keeping your gems secure? 

Always up-to-date Late adopter Reactive STRATEGY 1 • Use the latest version at all times • For analysis assume they’re always 1 day behind STRATEGY 2 • Wait a month after new major or minor versions • Always on the latest patch release STRATEGY 3 • Use the latest version when you add, then forget • Equally likely to have added or updated the gem at any time in the last year Three common gem strategies 

• 10 years of vulnerability advisories • Includes the date of disclosure • Allows us to determine whether a given version is secure The data to test them Ruby Advisory Database • History of gem releases over time • Includes date of each release • Tells us which version each strategy would have been using at any time Rubygems API • Combining, we can tell what version each strategy would have been on at the time a vulnerability was disclosed, and whether that version is secure • Full analysis at https://github.com/dependabot/gem-vulnerability-analysis 

Always keeping your gems up-to- date is the most secure strategy (just) Required response to a new vulnerability disclosure Always up-to-date Late adopter Reactive 47% 47% 47% 3% 4% 4% 42% 35% 35% 8% 15% 16% Not affected Fix by upgrading Fix by downgrading No fix on day zero • In addition, remember, the late adopter has a harder job with “fix by upgrading” • Full analysis at https://github.com/dependabot/gem-vulnerability-analysis 

How buggy / incompatible are new gem versions? 

Dependabot listens for the CI results of updates, so we have data! 

Major update a.b.c -> x.0.0 Minor update x.b.c -> x.y.0 Patch x.y.c -> x.y.z • Based on CI info for 1,750 updates • Many major updates are just dropping old Ruby support, etc. Minor and patch versions rarely have any incompatibilities or new bugs • Based on CI info for 12,000 updates • Pre-1.0.0 updates excluded • Based on CI info for 17,000 updates • Pre-1.0.0 updates excluded Updates with passing CI Notes • SemVer might not work in theory, but it does work in practice • Minor updates are nearly as easy to upgrade as patch releases 

• Thanks for listening! • Security analysis available at: https://github.com/dependabot/gem-vulnerability-analysis • Check out Dependabot at: https://dependabot.com