Historical and Impact Analysis of API Breaking Changes: A Large Scale Study (SANER 2017)

Historical and Impact Analysis of API Breaking Changes: A Large Scale Study (SANER 2017)

Change is a routine in software development. Like any system, libraries also evolve over time. As a consequence, clients are compelled to update and, thus, benefit from the available API improvements. However, some of these API changes may break contracts previously established, resulting in compilation errors and behavioral changes. In this paper, we study a set of questions regarding API breaking changes. Our goal is to measure the amount of breaking changes on real-world libraries and its impact on clients at a large-scale level. We assess (i) the frequency of breaking changes, (ii) the behavior of these changes over time, (iii) the impact on clients, and (iv) the characteristics of libraries with high frequency of breaking changes. Our largescale analysis on 317 real-world Java libraries, 9K releases, and 260K client applications shows that (i) 14.78% of the API changes break compatibility with previous versions, (ii) the frequency of breaking changes increases over time, (iii) 2.54% of their clients are impacted, and (iv) systems with higher frequency of breaking changes are larger, more popular, and more active. Based on these results, we provide a set of lessons to better support library and client developers in their maintenance tasks.

13beaa3b7239eca3319d54c6a9f3a85a?s=128

ASERG, DCC, UFMG

February 22, 2017
Tweet

Transcript

  1. Historical and Impact Analysis of API Breaking Changes: A Large-Scale

    Study Laerte Xavier, Aline Brito, André Hora, Marco Tulio Valente Federal University of Minas Gerais, Brazil SANER 2017
  2. 2 API

  3. APIs are constantly changing! 3

  4. 4

  5. 5 default Query<R> setHibernateFlushMode(FlushMode flushMode) { setFlushMode( flushMode ); return

    this; } default org.hibernate.query.Query<R> setHibernateFlushMode(FlushMode flushMode) { setFlushMode( flushMode ); return (org.hibernate.query.Query) this; } hibernate/hibernate-orm Version 5.2.0
  6. 6 Version 5.2.0 default org.hibernate.query.Query<R> setHibernateFlushMode(FlushMode flushMode) { setFlushMode( flushMode

    ); return (org.hibernate.query.Query) this; }
  7. 7 Breaking Changes Elements removal Method signature changes Non-Breaking Changes

    Changes in deprecated elements Elements addition Elements deprecation Field changes (Dig and Johnson, ICSM, 2006)
  8. 8 Related Work Hora and Valente, ICSME, 2015 Hora et

    al., CSMR, 2014 Henkel and Diwan, ICSE, 2005 Raemaekers et al., ICSM, 2012 API Evolution Change Impact Brito et al., SANER, 2016 McDonnel et al., ICSE, 2013
  9. Trabalhos Relacionados Hora and Valente, 2015; Hora et al., 2014;

    Henkel and Diwan, 2005. Raemaekers et al., 2012; Evolução de APIs Impacto de Mudanças Brito et al., 2016; McDonnel et al., 2013. Open Questions 9 To what extent are clients affected by API breaking changes?
  10. Trabalhos Relacionados Hora and Valente, 2015; Hora et al., 2014;

    Henkel and Diwan, 2005. Raemaekers et al., 2012; Evolução de APIs Impacto de Mudanças Brito et al., 2016; McDonnel et al., 2013. To what extent are clients affected by API breaking changes? Open Questions 10 Is this a problem only faced by newer libraries?
  11. 11 RQ1: What is the frequency of API breaking changes?

    RQ2: How do API breaking changes evolve over time? RQ3: What is the impact of API breaking changes in client applications? RQ4: What are the characteristics of libraries with high and low frequency of breaking changes?
  12. 12 TOP-317 GITHUB JAVA LIBRARIES

  13. 13 TOP-317 1,792 STARS GITHUB JAVA LIBRARIES

  14. 14 TOP-317 1,792 STARS 15 RELEASES GITHUB JAVA LIBRARIES

  15. 15 TOP-317 1,792 STARS 15 RELEASES 3.4 YEARS GITHUB JAVA

    LIBRARIES
  16. 16 RQ1: What is the frequency of API breaking changes?

    RQ2: How do API breaking changes evolve over time? RQ3: What is the impact of API breaking changes in client applications? RQ4: What are the characteristics of libraries with high and low frequency of breaking changes?
  17. 17 ... 1 2 N N-1 Design 3 N-2 First

    Commit August 2016
  18. 18 ... 1 2 N N-1 Design 3 N-2 First

    Commit August 2016 DIFF
  19. 19 501,645 LIBRARY CHANGES 28% BREAKING CHANGES

  20. 20 501,645 LIBRARY CHANGES 28% BREAKING CHANGES

  21. 21 501,645 LIBRARY CHANGES 28% BREAKING CHANGES

  22. 22 Libraries often break backward compatibility!

  23. 23 RQ1: What is the frequency of API breaking changes?

    RQ2: How do API breaking changes evolve over time? RQ3: What is the impact of API breaking changes in client applications? RQ4: What are the characteristics of libraries with high and low frequency of breaking changes?
  24. 24 ... 1 2 N N-1 Design 3 N-2 First

    Commit August 2016
  25. 25 ... 1 2 N N-1 Design 3 N-2 First

    Commit August 2016 DIFF
  26. DIFF 26 ... 1 2 N N-1 Design 3 N-2

    First Commit August 2016 DIFF
  27. DIFF DIFF 27 ... 1 2 N N-1 Design 3

    N-2 First Commit August 2016 DIFF
  28. DIFF DIFF DIFF DIFF DIFF DIFF 28 1 ... 2

    N N-1 Design 3 N-2 First Commit August 2016 DIFF
  29. 29 9,329 RELEASES 5 YEARS

  30. 30 9,329 RELEASES YEARS 20% 5

  31. 31 Breaking changes frequency increases over time!

  32. 32 RQ1: What is the frequency of API breaking changes?

    RQ2: How do API breaking changes evolve over time? RQ3: What is the impact of API breaking changes in client applications? RQ4: What are the characteristics of libraries with high and low frequency of breaking changes?
  33. DIFF 33 N N-1 Design JAVALI 263,425 Projects 16,386,193 Files

    http://java.labsoft.dcc.ufmg.br/javali Boa
  34. 34 140,460 BREAKING CHANGES 16,291 CHANGED TYPES 1,290 TYPES WITH

    POSSIBLE IMPACT
  35. 35 140,460 BREAKING CHANGES 16,291 CHANGED TYPES 1,290 TYPES WITH

    POSSIBLE IMPACT
  36. 36 Most breaking changes do not have a massive impact

    on clients!
  37. 37 RQ1: What is the frequency of API breaking changes?

    RQ2: How do API breaking changes evolve over time? RQ3: What is the impact of API breaking changes in client applications? RQ4: What are the characteristics of libraries with high and low frequency of breaking changes?
  38. Design 38

  39. Design Top (58) Bottom (58) 0% BC 100% BC 39

  40. Design Top (58) Bottom (58) 0% BC 100% BC (Mann-Whitney

    U Test & Cliff’s Delta) 40
  41. Bold: p-value < 0.05 (statistically significant different) and d >

    0.147 (at least small size effect) 41
  42. Bold: p-value < 0.05 (statistically significant different) and d >

    0.147 (at least small size effect) 42
  43. 43 Development and social coding measures are associated with API

    breaking changes!
  44. Trabalhos Relacionados Hora and Valente, 2015; Hora et al., 2014;

    Henkel and Diwan, 2005. Raemaekers et al., 2012; Evolução de APIs Impacto de Mudanças Brito et al., 2016; McDonnel et al., 2013. To what extent are clients affected by API breaking changes? Open Questions 44 Is this a problem only faced by newer libraries?
  45. 45 Libraries often break backward compatibility

  46. 46 Libraries often break backward compatibility Breaking changes frequency increases

    over time
  47. 47 Libraries often break backward compatibility Breaking changes frequency increases

    over time Most breaking changes do not have a massive impact on clients
  48. 48 Libraries often break backward compatibility Breaking changes frequency increases

    over time Most breaking changes do not have a massive impact on clients Development and social coding measures are associated with API breaking changes
  49. 49 Would you need tools to measure the impact of

    breaking changes? As an API developer...
  50. 50 Would you use stability metrics to choose an API?

    Would you need tools to measure the impact of breaking changes? As an API developer... As an API client...
  51. Historical and Impact Analysis of API Breaking Changes: A Large-Scale

    Study Laerte Xavier, Aline Brito, André Hora, Marco Tulio Valente Federal University of Minas Gerais, Brazil SANER 2017