Historical and Impact Analysis of API Breaking Changes: A Large Scale Study (SANER 2017)

Change is a routine in software development. Like any system, libraries also evolve over time. As a consequence, clients are compelled to update and, thus, benefit from the available API improvements. However, some of these API changes may break contracts previously established, resulting in compilation errors and behavioral changes. In this paper, we study a set of questions regarding API breaking changes. Our goal is to measure the amount of breaking changes on real-world libraries and its impact on clients at a large-scale level. We assess (i) the frequency of breaking changes, (ii) the behavior of these changes over time, (iii) the impact on clients, and (iv) the characteristics of libraries with high frequency of breaking changes. Our largescale analysis on 317 real-world Java libraries, 9K releases, and 260K client applications shows that (i) 14.78% of the API changes break compatibility with previous versions, (ii) the frequency of breaking changes increases over time, (iii) 2.54% of their clients are impacted, and (iv) systems with higher frequency of breaking changes are larger, more popular, and more active. Based on these results, we provide a set of lessons to better support library and client developers in their maintenance tasks.

Transcript

1. Historical and Impact Analysis of API
   Breaking Changes: A Large-Scale Study
   Laerte Xavier, Aline Brito, André Hora, Marco Tulio Valente
   Federal University of Minas Gerais, Brazil
   SANER 2017
   

   View Slide

2. 2
   API
   

   View Slide

3. APIs are constantly
   changing!
   3
   

   View Slide

4. View Slide

5. 5
   default Query setHibernateFlushMode(FlushMode flushMode) {
   setFlushMode( flushMode );
   return this;
   }
   default org.hibernate.query.Query setHibernateFlushMode(FlushMode
   flushMode) {
   setFlushMode( flushMode );
   return (org.hibernate.query.Query) this;
   }
   hibernate/hibernate-orm
   Version 5.2.0
   

   View Slide

6. 6
   Version 5.2.0
   default org.hibernate.query.Query setHibernateFlushMode(FlushMode
   flushMode) {
   setFlushMode( flushMode );
   return (org.hibernate.query.Query) this;
   }
   

   View Slide

7. 7
   Breaking Changes
   Elements removal
   Method signature changes
   Non-Breaking
   Changes
   Changes in deprecated elements
   Elements addition
   Elements deprecation
   Field changes
   (Dig and Johnson, ICSM, 2006)
   

   View Slide

8. 8
   Related Work
   Hora and Valente, ICSME, 2015
   Hora et al., CSMR, 2014
   Henkel and Diwan, ICSE, 2005
   Raemaekers et al., ICSM, 2012
   API Evolution Change Impact
   Brito et al., SANER, 2016
   McDonnel et al., ICSE, 2013
   

   View Slide

9. Trabalhos Relacionados
   Hora and Valente, 2015;
   Hora et al., 2014;
   Henkel and Diwan, 2005.
   Raemaekers et al., 2012;
   Evolução de APIs Impacto de Mudanças
   Brito et al., 2016;
   McDonnel et al., 2013.
   Open Questions
   9
   To what extent are clients affected by
   API breaking changes?
   

   View Slide

10. Trabalhos Relacionados
    Hora and Valente, 2015;
    Hora et al., 2014;
    Henkel and Diwan, 2005.
    Raemaekers et al., 2012;
    Evolução de APIs Impacto de Mudanças
    Brito et al., 2016;
    McDonnel et al., 2013.
    To what extent are clients affected by
    API breaking changes?
    Open Questions
    10
    Is this a problem only faced by newer
    libraries?
    

    View Slide

11. 11
    RQ1: What is the frequency of API breaking changes?
    RQ2: How do API breaking changes evolve over time?
    RQ3: What is the impact of API breaking changes in
    client applications?
    RQ4: What are the characteristics of libraries with high
    and low frequency of breaking changes?
    

    View Slide

12. 12
    TOP-317
    GITHUB JAVA LIBRARIES
    

    View Slide

13. 13
    TOP-317
    1,792
    STARS
    GITHUB JAVA LIBRARIES
    

    View Slide

14. 14
    TOP-317
    1,792
    STARS
    15
    RELEASES
    GITHUB JAVA LIBRARIES
    

    View Slide

15. 15
    TOP-317
    1,792
    STARS
    15
    RELEASES
    3.4
    YEARS
    GITHUB JAVA LIBRARIES
    

    View Slide

16. 16
    RQ1: What is the frequency of API breaking changes?
    RQ2: How do API breaking changes evolve over time?
    RQ3: What is the impact of API breaking changes in
    client applications?
    RQ4: What are the characteristics of libraries with high
    and low frequency of breaking changes?
    

    View Slide

17. 17
    ...
    1 2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016
    

    View Slide

18. 18
    ...
    1 2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016
    DIFF
    

    View Slide

19. 19
    501,645
    LIBRARY
    CHANGES
    28%
    BREAKING
    CHANGES
    

    View Slide

20. 20
    501,645
    LIBRARY
    CHANGES
    28%
    BREAKING
    CHANGES
    

    View Slide

21. 21
    501,645
    LIBRARY
    CHANGES
    28%
    BREAKING
    CHANGES
    

    View Slide

22. 22
    Libraries often break
    backward compatibility!
    

    View Slide

23. 23
    RQ1: What is the frequency of API breaking changes?
    RQ2: How do API breaking changes evolve over time?
    RQ3: What is the impact of API breaking changes in
    client applications?
    RQ4: What are the characteristics of libraries with high
    and low frequency of breaking changes?
    

    View Slide

24. 24
    ...
    1 2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016
    

    View Slide

25. 25
    ...
    1 2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016
    DIFF
    

    View Slide

26. DIFF
    26
    ...
    1 2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016
    DIFF
    

    View Slide

27. DIFF
    DIFF
    27
    ...
    1 2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016
    DIFF
    

    View Slide

28. DIFF
    DIFF
    DIFF
    DIFF
    DIFF
    DIFF
    28
    1
    ...
    2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016
    DIFF
    

    View Slide

29. 29
    9,329
    RELEASES
    5
    YEARS
    

    View Slide

30. 30
    9,329
    RELEASES YEARS
    20%
    5
    

    View Slide

31. 31
    Breaking changes frequency
    increases over time!
    

    View Slide

32. 32
    RQ1: What is the frequency of API breaking changes?
    RQ2: How do API breaking changes evolve over time?
    RQ3: What is the impact of API breaking changes in
    client applications?
    RQ4: What are the characteristics of libraries with high
    and low frequency of breaking changes?
    

    View Slide

33. DIFF
    33
    N
    N-1
    Design
    JAVALI
    263,425 Projects
    16,386,193
    Files
    http://java.labsoft.dcc.ufmg.br/javali
    Boa
    

    View Slide

34. 34
    140,460
    BREAKING CHANGES
    16,291
    CHANGED TYPES
    1,290
    TYPES WITH
    POSSIBLE IMPACT
    

    View Slide

35. 35
    140,460
    BREAKING CHANGES
    16,291
    CHANGED TYPES
    1,290
    TYPES WITH
    POSSIBLE IMPACT
    

    View Slide

36. 36
    Most breaking changes
    do not have a massive
    impact on clients!
    

    View Slide

37. 37
    RQ1: What is the frequency of API breaking changes?
    RQ2: How do API breaking changes evolve over time?
    RQ3: What is the impact of API breaking changes in
    client applications?
    RQ4: What are the characteristics of libraries with high
    and low frequency of breaking changes?
    

    View Slide

38. Design
    38
    

    View Slide

39. Design
    Top (58)
    Bottom (58)
    0% BC
    100%
    BC
    39
    

    View Slide

40. Design
    Top (58)
    Bottom (58)
    0% BC
    100%
    BC
    (Mann-Whitney U
    Test & Cliff’s Delta)
    40
    

    View Slide

41. Bold: p-value < 0.05 (statistically significant different) and
    d > 0.147 (at least small size effect)
    41
    

    View Slide

42. Bold: p-value < 0.05 (statistically significant different) and
    d > 0.147 (at least small size effect)
    42
    

    View Slide

43. 43
    Development and social
    coding measures are
    associated with API
    breaking changes!
    

    View Slide

44. Trabalhos Relacionados
    Hora and Valente, 2015;
    Hora et al., 2014;
    Henkel and Diwan, 2005.
    Raemaekers et al., 2012;
    Evolução de APIs Impacto de Mudanças
    Brito et al., 2016;
    McDonnel et al., 2013.
    To what extent are clients affected by
    API breaking changes?
    Open Questions
    44
    Is this a problem only faced by newer
    libraries?
    

    View Slide

45. 45
    Libraries often break backward compatibility
    

    View Slide

46. 46
    Libraries often break backward compatibility
    Breaking changes frequency increases over time
    

    View Slide

47. 47
    Libraries often break backward compatibility
    Breaking changes frequency increases over time
    Most breaking changes do not have a massive
    impact on clients
    

    View Slide

48. 48
    Libraries often break backward compatibility
    Breaking changes frequency increases over time
    Most breaking changes do not have a massive
    impact on clients
    Development and social coding measures
    are associated with API breaking changes
    

    View Slide

49. 49
    Would you need tools to
    measure the impact of
    breaking changes?
    As an API developer...
    

    View Slide

50. 50
    Would you use stability
    metrics to choose an API?
    Would you need tools to
    measure the impact of
    breaking changes?
    As an API developer...
    As an API client...
    

    View Slide

51. Historical and Impact Analysis of API
    Breaking Changes: A Large-Scale Study
    Laerte Xavier, Aline Brito, André Hora, Marco Tulio Valente
    Federal University of Minas Gerais, Brazil
    SANER 2017
    

    View Slide