Historical and Impact Analysis of API Breaking Changes: A Large Scale Study (SANER 2017)

Change is a routine in software development. Like any system, libraries also evolve over time. As a consequence, clients are compelled to update and, thus, benefit from the available API improvements. However, some of these API changes may break contracts previously established, resulting in compilation errors and behavioral changes. In this paper, we study a set of questions regarding API breaking changes. Our goal is to measure the amount of breaking changes on real-world libraries and its impact on clients at a large-scale level. We assess (i) the frequency of breaking changes, (ii) the behavior of these changes over time, (iii) the impact on clients, and (iv) the characteristics of libraries with high frequency of breaking changes. Our largescale analysis on 317 real-world Java libraries, 9K releases, and 260K client applications shows that (i) 14.78% of the API changes break compatibility with previous versions, (ii) the frequency of breaking changes increases over time, (iii) 2.54% of their clients are impacted, and (iv) systems with higher frequency of breaking changes are larger, more popular, and more active. Based on these results, we provide a set of lessons to better support library and client developers in their maintenance tasks.

Transcript

1. Historical and Impact Analysis of API Breaking Changes: A Large-Scale Study Laerte Xavier, Aline Brito, André Hora, Marco Tulio Valente Federal University of Minas Gerais, Brazil SANER 2017
2. 2 API
3. APIs are constantly changing! 3
5. 5 default Query<R> setHibernateFlushMode(FlushMode flushMode) { setFlushMode( flushMode ); return this; } default org.hibernate.query.Query<R> setHibernateFlushMode(FlushMode flushMode) { setFlushMode( flushMode ); return (org.hibernate.query.Query) this; } hibernate/hibernate-orm Version 5.2.0
6. 6 Version 5.2.0 default org.hibernate.query.Query<R> setHibernateFlushMode(FlushMode flushMode) { setFlushMode( flushMode ); return (org.hibernate.query.Query) this; }
7. 7 Breaking Changes Elements removal Method signature changes Non-Breaking Changes Changes in deprecated elements Elements addition Elements deprecation Field changes (Dig and Johnson, ICSM, 2006)
8. 8 Related Work Hora and Valente, ICSME, 2015 Hora et al., CSMR, 2014 Henkel and Diwan, ICSE, 2005 Raemaekers et al., ICSM, 2012 API Evolution Change Impact Brito et al., SANER, 2016 McDonnel et al., ICSE, 2013
9. Trabalhos Relacionados Hora and Valente, 2015; Hora et al., 2014; Henkel and Diwan, 2005. Raemaekers et al., 2012; Evolução de APIs Impacto de Mudanças Brito et al., 2016; McDonnel et al., 2013. Open Questions 9 To what extent are clients affected by API breaking changes?
10. Trabalhos Relacionados Hora and Valente, 2015; Hora et al., 2014; Henkel and Diwan, 2005. Raemaekers et al., 2012; Evolução de APIs Impacto de Mudanças Brito et al., 2016; McDonnel et al., 2013. To what extent are clients affected by API breaking changes? Open Questions 10 Is this a problem only faced by newer libraries?
11. 11 RQ1: What is the frequency of API breaking changes? RQ2: How do API breaking changes evolve over time? RQ3: What is the impact of API breaking changes in client applications? RQ4: What are the characteristics of libraries with high and low frequency of breaking changes?
12. 12 TOP-317 GITHUB JAVA LIBRARIES
13. 13 TOP-317 1,792 STARS GITHUB JAVA LIBRARIES
14. 14 TOP-317 1,792 STARS 15 RELEASES GITHUB JAVA LIBRARIES
15. 15 TOP-317 1,792 STARS 15 RELEASES 3.4 YEARS GITHUB JAVA LIBRARIES
16. 16 RQ1: What is the frequency of API breaking changes? RQ2: How do API breaking changes evolve over time? RQ3: What is the impact of API breaking changes in client applications? RQ4: What are the characteristics of libraries with high and low frequency of breaking changes?
17. 17 ... 1 2 N N-1 Design 3 N-2 First Commit August 2016
18. 18 ... 1 2 N N-1 Design 3 N-2 First Commit August 2016 DIFF
19. 19 501,645 LIBRARY CHANGES 28% BREAKING CHANGES
20. 20 501,645 LIBRARY CHANGES 28% BREAKING CHANGES
21. 21 501,645 LIBRARY CHANGES 28% BREAKING CHANGES
22. 22 Libraries often break backward compatibility!
23. 23 RQ1: What is the frequency of API breaking changes? RQ2: How do API breaking changes evolve over time? RQ3: What is the impact of API breaking changes in client applications? RQ4: What are the characteristics of libraries with high and low frequency of breaking changes?
24. 24 ... 1 2 N N-1 Design 3 N-2 First Commit August 2016
25. 25 ... 1 2 N N-1 Design 3 N-2 First Commit August 2016 DIFF
26. DIFF 26 ... 1 2 N N-1 Design 3 N-2 First Commit August 2016 DIFF
27. DIFF DIFF 27 ... 1 2 N N-1 Design 3 N-2 First Commit August 2016 DIFF
28. DIFF DIFF DIFF DIFF DIFF DIFF 28 1 ... 2 N N-1 Design 3 N-2 First Commit August 2016 DIFF
29. 29 9,329 RELEASES 5 YEARS
30. 30 9,329 RELEASES YEARS 20% 5
31. 31 Breaking changes frequency increases over time!
32. 32 RQ1: What is the frequency of API breaking changes? RQ2: How do API breaking changes evolve over time? RQ3: What is the impact of API breaking changes in client applications? RQ4: What are the characteristics of libraries with high and low frequency of breaking changes?
33. DIFF 33 N N-1 Design JAVALI 263,425 Projects 16,386,193 Files http://java.labsoft.dcc.ufmg.br/javali Boa
34. 34 140,460 BREAKING CHANGES 16,291 CHANGED TYPES 1,290 TYPES WITH POSSIBLE IMPACT
35. 35 140,460 BREAKING CHANGES 16,291 CHANGED TYPES 1,290 TYPES WITH POSSIBLE IMPACT
36. 36 Most breaking changes do not have a massive impact on clients!
37. 37 RQ1: What is the frequency of API breaking changes? RQ2: How do API breaking changes evolve over time? RQ3: What is the impact of API breaking changes in client applications? RQ4: What are the characteristics of libraries with high and low frequency of breaking changes?
38. Design 38
39. Design Top (58) Bottom (58) 0% BC 100% BC 39
40. Design Top (58) Bottom (58) 0% BC 100% BC (Mann-Whitney U Test & Cliff’s Delta) 40
41. Bold: p-value < 0.05 (statistically significant different) and d > 0.147 (at least small size effect) 41
42. Bold: p-value < 0.05 (statistically significant different) and d > 0.147 (at least small size effect) 42
43. 43 Development and social coding measures are associated with API breaking changes!
44. Trabalhos Relacionados Hora and Valente, 2015; Hora et al., 2014; Henkel and Diwan, 2005. Raemaekers et al., 2012; Evolução de APIs Impacto de Mudanças Brito et al., 2016; McDonnel et al., 2013. To what extent are clients affected by API breaking changes? Open Questions 44 Is this a problem only faced by newer libraries?
45. 45 Libraries often break backward compatibility
46. 46 Libraries often break backward compatibility Breaking changes frequency increases over time
47. 47 Libraries often break backward compatibility Breaking changes frequency increases over time Most breaking changes do not have a massive impact on clients
48. 48 Libraries often break backward compatibility Breaking changes frequency increases over time Most breaking changes do not have a massive impact on clients Development and social coding measures are associated with API breaking changes
49. 49 Would you need tools to measure the impact of breaking changes? As an API developer...
50. 50 Would you use stability metrics to choose an API? Would you need tools to measure the impact of breaking changes? As an API developer... As an API client...
51. Historical and Impact Analysis of API Breaking Changes: A Large-Scale Study Laerte Xavier, Aline Brito, André Hora, Marco Tulio Valente Federal University of Minas Gerais, Brazil SANER 2017