$30 off During Our Annual Pro Sale. View Details »

Historical and Impact Analysis of API Breaking Changes: A Large Scale Study (SANER 2017)

Historical and Impact Analysis of API Breaking Changes: A Large Scale Study (SANER 2017)

Change is a routine in software development. Like any system, libraries also evolve over time. As a consequence, clients are compelled to update and, thus, benefit from the available API improvements. However, some of these API changes may break contracts previously established, resulting in compilation errors and behavioral changes. In this paper, we study a set of questions regarding API breaking changes. Our goal is to measure the amount of breaking changes on real-world libraries and its impact on clients at a large-scale level. We assess (i) the frequency of breaking changes, (ii) the behavior of these changes over time, (iii) the impact on clients, and (iv) the characteristics of libraries with high frequency of breaking changes. Our largescale analysis on 317 real-world Java libraries, 9K releases, and 260K client applications shows that (i) 14.78% of the API changes break compatibility with previous versions, (ii) the frequency of breaking changes increases over time, (iii) 2.54% of their clients are impacted, and (iv) systems with higher frequency of breaking changes are larger, more popular, and more active. Based on these results, we provide a set of lessons to better support library and client developers in their maintenance tasks.

ASERG, DCC, UFMG

February 22, 2017
Tweet

More Decks by ASERG, DCC, UFMG

Other Decks in Research

Transcript

  1. Historical and Impact Analysis of API
    Breaking Changes: A Large-Scale Study
    Laerte Xavier, Aline Brito, André Hora, Marco Tulio Valente
    Federal University of Minas Gerais, Brazil
    SANER 2017

    View Slide

  2. 2
    API

    View Slide

  3. APIs are constantly
    changing!
    3

    View Slide

  4. 4

    View Slide

  5. 5
    default Query setHibernateFlushMode(FlushMode flushMode) {
    setFlushMode( flushMode );
    return this;
    }
    default org.hibernate.query.Query setHibernateFlushMode(FlushMode
    flushMode) {
    setFlushMode( flushMode );
    return (org.hibernate.query.Query) this;
    }
    hibernate/hibernate-orm
    Version 5.2.0

    View Slide

  6. 6
    Version 5.2.0
    default org.hibernate.query.Query setHibernateFlushMode(FlushMode
    flushMode) {
    setFlushMode( flushMode );
    return (org.hibernate.query.Query) this;
    }

    View Slide

  7. 7
    Breaking Changes
    Elements removal
    Method signature changes
    Non-Breaking
    Changes
    Changes in deprecated elements
    Elements addition
    Elements deprecation
    Field changes
    (Dig and Johnson, ICSM, 2006)

    View Slide

  8. 8
    Related Work
    Hora and Valente, ICSME, 2015
    Hora et al., CSMR, 2014
    Henkel and Diwan, ICSE, 2005
    Raemaekers et al., ICSM, 2012
    API Evolution Change Impact
    Brito et al., SANER, 2016
    McDonnel et al., ICSE, 2013

    View Slide

  9. Trabalhos Relacionados
    Hora and Valente, 2015;
    Hora et al., 2014;
    Henkel and Diwan, 2005.
    Raemaekers et al., 2012;
    Evolução de APIs Impacto de Mudanças
    Brito et al., 2016;
    McDonnel et al., 2013.
    Open Questions
    9
    To what extent are clients affected by
    API breaking changes?

    View Slide

  10. Trabalhos Relacionados
    Hora and Valente, 2015;
    Hora et al., 2014;
    Henkel and Diwan, 2005.
    Raemaekers et al., 2012;
    Evolução de APIs Impacto de Mudanças
    Brito et al., 2016;
    McDonnel et al., 2013.
    To what extent are clients affected by
    API breaking changes?
    Open Questions
    10
    Is this a problem only faced by newer
    libraries?

    View Slide

  11. 11
    RQ1: What is the frequency of API breaking changes?
    RQ2: How do API breaking changes evolve over time?
    RQ3: What is the impact of API breaking changes in
    client applications?
    RQ4: What are the characteristics of libraries with high
    and low frequency of breaking changes?

    View Slide

  12. 12
    TOP-317
    GITHUB JAVA LIBRARIES

    View Slide

  13. 13
    TOP-317
    1,792
    STARS
    GITHUB JAVA LIBRARIES

    View Slide

  14. 14
    TOP-317
    1,792
    STARS
    15
    RELEASES
    GITHUB JAVA LIBRARIES

    View Slide

  15. 15
    TOP-317
    1,792
    STARS
    15
    RELEASES
    3.4
    YEARS
    GITHUB JAVA LIBRARIES

    View Slide

  16. 16
    RQ1: What is the frequency of API breaking changes?
    RQ2: How do API breaking changes evolve over time?
    RQ3: What is the impact of API breaking changes in
    client applications?
    RQ4: What are the characteristics of libraries with high
    and low frequency of breaking changes?

    View Slide

  17. 17
    ...
    1 2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016

    View Slide

  18. 18
    ...
    1 2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016
    DIFF

    View Slide

  19. 19
    501,645
    LIBRARY
    CHANGES
    28%
    BREAKING
    CHANGES

    View Slide

  20. 20
    501,645
    LIBRARY
    CHANGES
    28%
    BREAKING
    CHANGES

    View Slide

  21. 21
    501,645
    LIBRARY
    CHANGES
    28%
    BREAKING
    CHANGES

    View Slide

  22. 22
    Libraries often break
    backward compatibility!

    View Slide

  23. 23
    RQ1: What is the frequency of API breaking changes?
    RQ2: How do API breaking changes evolve over time?
    RQ3: What is the impact of API breaking changes in
    client applications?
    RQ4: What are the characteristics of libraries with high
    and low frequency of breaking changes?

    View Slide

  24. 24
    ...
    1 2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016

    View Slide

  25. 25
    ...
    1 2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016
    DIFF

    View Slide

  26. DIFF
    26
    ...
    1 2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016
    DIFF

    View Slide

  27. DIFF
    DIFF
    27
    ...
    1 2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016
    DIFF

    View Slide

  28. DIFF
    DIFF
    DIFF
    DIFF
    DIFF
    DIFF
    28
    1
    ...
    2 N
    N-1
    Design
    3 N-2
    First
    Commit
    August
    2016
    DIFF

    View Slide

  29. 29
    9,329
    RELEASES
    5
    YEARS

    View Slide

  30. 30
    9,329
    RELEASES YEARS
    20%
    5

    View Slide

  31. 31
    Breaking changes frequency
    increases over time!

    View Slide

  32. 32
    RQ1: What is the frequency of API breaking changes?
    RQ2: How do API breaking changes evolve over time?
    RQ3: What is the impact of API breaking changes in
    client applications?
    RQ4: What are the characteristics of libraries with high
    and low frequency of breaking changes?

    View Slide

  33. DIFF
    33
    N
    N-1
    Design
    JAVALI
    263,425 Projects
    16,386,193
    Files
    http://java.labsoft.dcc.ufmg.br/javali
    Boa

    View Slide

  34. 34
    140,460
    BREAKING CHANGES
    16,291
    CHANGED TYPES
    1,290
    TYPES WITH
    POSSIBLE IMPACT

    View Slide

  35. 35
    140,460
    BREAKING CHANGES
    16,291
    CHANGED TYPES
    1,290
    TYPES WITH
    POSSIBLE IMPACT

    View Slide

  36. 36
    Most breaking changes
    do not have a massive
    impact on clients!

    View Slide

  37. 37
    RQ1: What is the frequency of API breaking changes?
    RQ2: How do API breaking changes evolve over time?
    RQ3: What is the impact of API breaking changes in
    client applications?
    RQ4: What are the characteristics of libraries with high
    and low frequency of breaking changes?

    View Slide

  38. Design
    38

    View Slide

  39. Design
    Top (58)
    Bottom (58)
    0% BC
    100%
    BC
    39

    View Slide

  40. Design
    Top (58)
    Bottom (58)
    0% BC
    100%
    BC
    (Mann-Whitney U
    Test & Cliff’s Delta)
    40

    View Slide

  41. Bold: p-value < 0.05 (statistically significant different) and
    d > 0.147 (at least small size effect)
    41

    View Slide

  42. Bold: p-value < 0.05 (statistically significant different) and
    d > 0.147 (at least small size effect)
    42

    View Slide

  43. 43
    Development and social
    coding measures are
    associated with API
    breaking changes!

    View Slide

  44. Trabalhos Relacionados
    Hora and Valente, 2015;
    Hora et al., 2014;
    Henkel and Diwan, 2005.
    Raemaekers et al., 2012;
    Evolução de APIs Impacto de Mudanças
    Brito et al., 2016;
    McDonnel et al., 2013.
    To what extent are clients affected by
    API breaking changes?
    Open Questions
    44
    Is this a problem only faced by newer
    libraries?

    View Slide

  45. 45
    Libraries often break backward compatibility

    View Slide

  46. 46
    Libraries often break backward compatibility
    Breaking changes frequency increases over time

    View Slide

  47. 47
    Libraries often break backward compatibility
    Breaking changes frequency increases over time
    Most breaking changes do not have a massive
    impact on clients

    View Slide

  48. 48
    Libraries often break backward compatibility
    Breaking changes frequency increases over time
    Most breaking changes do not have a massive
    impact on clients
    Development and social coding measures
    are associated with API breaking changes

    View Slide

  49. 49
    Would you need tools to
    measure the impact of
    breaking changes?
    As an API developer...

    View Slide

  50. 50
    Would you use stability
    metrics to choose an API?
    Would you need tools to
    measure the impact of
    breaking changes?
    As an API developer...
    As an API client...

    View Slide

  51. Historical and Impact Analysis of API
    Breaking Changes: A Large-Scale Study
    Laerte Xavier, Aline Brito, André Hora, Marco Tulio Valente
    Federal University of Minas Gerais, Brazil
    SANER 2017

    View Slide