Measuring dependency freshness in software systems Joël Cox, Eric Bouwers, Marko van Eekelen and Joost Visser 1 

2 Outline • Introduction • Dependency freshness at the component-level • Dependency freshness at the system-level • Validation • Conclusion 

What is dependency freshness? • Dependency: third-party (Java) software components • Actual used version vs. ideal version of a dependency 3 

Why is dependency freshness relevant? • Security • Flexibility • Stability • Compatibility 4 Why is dependency freshness hard? • Big testing efforts • Different priorities • Implicit dependencies 

Ultimately, updating a dependency is a trade-off between risk and effort 5 

Research context 6 

Research questions • RQ1 How can we measure the dependency freshness of a single dependency? • RQ2 How can we measure the dependency freshness of a system as a whole? • Goal To quantify the dependency freshness of a given software system from the point of view of external quality evaluators. 7 

Dependency freshness at the component-level 8 

Dependencies 9 

Dependency freshness 10 t t + 1 t + 2 s s s d 1 d 1 d 1 d 2 d 2 1 1 1 2 2 depends on succeeded by 0 0 0 

Dependency freshness 11 t t + 1 t + 2 s s s d 1 d 1 d 1 d 2 d 2 1 1 1 2 2 depends on succeeded by 0 0 0 

Dependency freshness 12 t t + 1 t + 2 s s s d 1 d 1 d 1 d 2 d 2 1 1 1 2 2 depends on succeeded by 0 0 0 

Dependency freshness 13 t t + 1 t + 2 s s s d 1 d 1 d 1 d 2 d 2 1 1 1 2 2 depends on succeeded by 0 0 0 

What makes a good metric given our research context? • Technology independent • Ease of implementation • Simple to understand • Enable root-cause analysis 14 

1. Version sequence number 15 Release 1 Release 2 Release 3 2 

2. Version release date 16 10/03/2014 02/04/2014 30/06/1992 113 

3. Version number delta 17 1.2.0 1.2.1 1.3.0 (0, 1, 1) 

Metric overview 18 Version sequence number Version release date Version number delta Technology independent + + - Ease of implementation + + + Simple to understand + + - Enable root-cause analysis + + + 

Datasets 19 Industry systems repository Maven components repository Available dependencies Used dependencies Tooling Rating Thresholds 75 systems, 30 clients 3107 dependencies, 8718 versions 23431 additional dependencies 75% release date hit rate 

Dependency freshness in practice 20 Freshness measured by release date distance Frequency 0 500 1000 1500 2000 2500 3000 0 200 400 600 Freshness measured by release sequence distance Frequency 0 20 40 60 80 100 120 0 200 400 600 800 1000 1200 

Dependency freshness at the system-level 21 

• What is good dependency management? • Use a benchmark approach to compare systems. Normative vs. descriptive 22 

Defining a system-level measurement 23 {6, 21, 8, 11, 7, 31, 17, 3} Component-level measurements Risk profile Star rating Alves, et. al. Deriving metric thresholds from benchmark data. ICSM 2010 
 Alves, et. al. Benchmark-based aggregation of metrics to ratings. IWSM-MENSURA 2011 

Validation 24 

Three different studies 1. Reported security vulnerabilities in dependencies 2. Interviews with practitioners 3. Longitudinal analysis 25 

1. Reported security vulnerabilities 26 ● 0 1 2 3 4 1 2 3 4 5 Number of dependencies with reported vulnerability Dependency freshness rating 0.0 0.2 0.4 0.6 1 2 3 4 5 Dependency freshness rating Density Vulnerable FALSE TRUE 

2. Interviews 27 Qualitative results • High usefulness
 All interviewees considered the metric useful when assessing systems • High reliability
 All interviewees were able top predict the rating ±1 star • High actionability
 Most interviewees would follow up on the results with clients Metric ranking System Subject # Rating Rank 1 2 3 4 5 1108 5.053 5 5 5 5 5 5 1994 4.105 4 4 2 4 4 4 850 3.248 3 3 3 3 3 3 362 2.188 2 2 4 2 2 2 181 1.427 1 1 1 1 1 1 

3. Longitudinal analysis (1) 28 1 2 3 4 5 2010 2011 2012 2013 2014 Time Dependency freshness rating 

3. Longitudinal analysis (2) 29 1 2 3 4 5 2010 2011 2012 2013 2014 Time Dependency freshness rating 

3. Longitudinal analysis (3) 30 1 2 3 4 5 2010 2011 2012 2013 2014 Time Dependency freshness rating 

3. Longitudinal analysis (4) 31 1 2 3 4 5 2010 2011 2012 2013 2014 Time Dependency freshness rating 

Summary • Serves as an indicator for security • Metric is considered useful in practice • Allows for monitoring through time 32 

Conclusion 33 

Contributions • Component-level metric for dependency freshness • Analysis of dependency freshness on industry systems • System-level metric for dependency freshness, that can help stakeholders make decisions about dependency management • Validated for usefulness, reliability and its relation to security 34 

In practice • Inspect systems on a case-by-case • Monitor systems over a time • Remediate underperforming systems 35 


 Thank you! [email protected] @joelcox 36