Work Practices, Challenges, Research Opportunities in Continuous Integration

Work Practices, Challenges, Research Opportunities in Continuous Integration

D0270498e20bd573441f1f48f2e425cf?s=128

Gustavo Pinto

July 10, 2018
Tweet

Transcript

  1. 11.

    @gustavopinto Continuous Integration doesn’t get rid of bugs, but it

    does make them dramatically easier to find and remove
  2. 12.

    @gustavopinto Continuous Integration doesn’t get rid of bugs, but it

    does make them dramatically easier to find and remove Continuous Integration is one of the key XP practices. I’ve been selling books talking about it since the 90s
  3. 13.

    Continuous Integration doesn’t get rid of bugs, but it does

    make them dramatically easier to find and remove Continuous Integration is one of the key XP practices. I’ve been selling books talking about it since the 90s Continuous Integration make sure that our test case suite is in good shape! @gustavopinto
  4. 17.

    C C++ C# Clojure CoffeeScript Erlang Go Haskell Java Jav

    JavaScript Objective-C PHP Perl Python Ruby Scala TypeScript Most popular PLs on @gustavopinto
  5. 18.

    C C++ C# Clojure CoffeeScript Erlang Go Haskell Java Jav

    JavaScript Objective-C PHP Perl Python Ruby Scala TypeScript Most popular PLs on @gustavopinto
  6. 19.

    C C++ C# Clojure CoffeeScript Erlang Go Haskell Java Jav

    JavaScript Objective-C PHP Perl Python Ruby Scala TypeScript Most popular PLs on 50 most popular projects … … … … … … … … … … … … 750 most popular projects @gustavopinto
  7. 24.

    666 38% 48% 32% 13% 18% About 25% of all

    builds have failed! @gustavopinto 17%
  8. 27.

    Dear Andrei, we saw that your pull-request #33292 broke the

    build of the rails/rails project. Since we are studying the reasons behind broken builds, would you mind to answer a few questions? Link here! Thank you, Gustavo Pinto Hi Gustavo! Sure, I’m happy to help. Andrei @gustavopinto
  9. 28.

    Background Experience with CI (Q1) Current position (Q2) Work for

    (Q3) How often contribute to open source? (Q4) How familiar with CI? (Q5) How many OSS projects contributed to with CI? (Q6) Have configured any project to use CI? (Q7) Why do you use CI? @gustavopinto
  10. 29.

    Reasons for build breakage? (Q12) What are the technical reasons?

    (Q13) What are the social reasons? @gustavopinto Benefits and Challenges (Q14) What are the benefits of using CI? (Q15) What are the challenges of using CI?
  11. 36.

    ! ! ! ! ! ! ! ! ! !

    ! ! ! ! ! ! ! ! @gustavopinto ! ! ! ! ! !
  12. 40.

    Experience with CI: Reasons for adoption Quick feedback on produced

    development for adding business value running *all* tests locally is taking to long (10+ mins) Speed up software dev. Improve software quality Catch regressions/bugs Enforce automation Credibility @gustavopinto
  13. 41.

    Experience with CI: Reasons for adoption ensuring both, mine and

    other people changes, do not break anything Speed up software dev. Improve software quality Catch regressions/bugs Enforce automation Credibility @gustavopinto
  14. 42.

    Experience with CI: Reasons for adoption Much easier regression testing

    in complex projects Prevent to release a bug software into production environment Speed up software dev. Improve software quality Catch regressions/bugs Enforce automation Credibility @gustavopinto
  15. 43.

    Experience with CI: Reasons for adoption To make sure tests

    are always run, since humans can be inconsistent on that Speed up software dev. Improve software quality Catch regressions/bugs Enforce automation Credibility @gustavopinto
  16. 44.

    Experience with CI: Reasons for adoption it makes the project

    look more legit Speed up software dev. Improve software quality Catch regressions/bugs Enforce automation Credibility @gustavopinto
  17. 45.

    Technical reasons for build breakage @gustavopinto Inadequate Testing Version changes

    Dependency Management Complex code base Git usage Badly written that fail with minor bugfixes not enough tests
  18. 46.

    Technical reasons for build breakage @gustavopinto Inadequate Testing Version changes

    Dependency Management Complex code base Git usage the version of a language component is different, and the change made to the language cause breakage
  19. 47.

    Technical reasons for build breakage @gustavopinto Inadequate Testing Version changes

    Dependency management Complex code base Git usage people sometimes don’t update dependencies, so the Continuous Integration server detects errors that do not happen locally
  20. 48.

    Technical reasons for build breakage @gustavopinto Inadequate Testing Version changes

    Dependency Management Complex code base Git usage unfamiliarity with the architecture of the code and overall module interactions
  21. 49.

    Technical reasons for build breakage @gustavopinto Inadequate Testing Version changes

    Dependency Management Complex code base Git usage git rebase and left the work on hold for too long
  22. 50.

    Social reasons for build breakage just be happy I’m commiting

    to the project, somebody else can test if what I did works Carelessness Lack of testing culture Overconfidence Lack of communication It’s just fine @gustavopinto
  23. 51.

    Social reasons for build breakage Carelessness Lack of testing culture

    Overconfidence Lack of communication It’s just fine @gustavopinto people not running the tests and build on their machines before pushing the changes
  24. 52.

    Social reasons for build breakage Carelessness Lack of testing culture

    Overconfidence Lack of communication It’s just fine @gustavopinto this is only a small fix, it should not break anything
  25. 53.

    Social reasons for build breakage Carelessness Lack of testing culture

    Overconfidence Lack of communication It’s just fine @gustavopinto not knowing who to ask for help
  26. 54.

    Social reasons for build breakage @gustavopinto Carelessness Lack of testing

    culture Overconfidence Lack of communication It’s just fine CI is there for you not to be afraid for broken builds in a branch
  27. 55.

    Social reasons for build breakage @gustavopinto Carelessness Lack of testing

    culture Overconfidence Lack of communication It’s just fine CI is there for you not to be afraid for broken builds in a branch Only if it is not in the master branch
  28. 56.

    Benefits of CI @gustavopinto Catch problems early Cross-platform testing Confidence

    Automation Software quality Being aware of when/where breakage occurs greatly accelerates solution
  29. 57.

    Benefits of CI @gustavopinto Catch problems early Cross-platform testing Confidence

    Automation Software quality compiling for multiple different targets (x86, arm, windows, linux etc)
  30. 58.

    Benefits of CI @gustavopinto Catch problems early Cross-platform testing Confidence

    Automation Software quality compiling for multiple different targets (x86, arm, windows, linux etc) this is not feasible or cost-effective to do manually
  31. 59.

    Benefits of CI @gustavopinto Catch problems early Cross-platform testing Confidence

    Automation Software quality Some sort of confidence that introduced changes don’t break the current behavior But depends on the coverage
  32. 60.

    Benefits of CI @gustavopinto Catch problems early Cross-platform testing Confidence

    Automation Software quality performing a wide range of manual steps that a human would not normally be bothered to check
  33. 61.

    Benefits of CI @gustavopinto Catch problems early Cross-platform testing Confidence

    Automation Software quality When you use CI, you have a good health check in your code base […] It is a warrant of quality of your code
  34. 62.

    Challenges of CI @gustavopinto Set up environment False sense of

    confidence Discipline Monetary costs Flaky tests I found it hard to setup some distributed/multicomponent tests. This partially can be resolved by the containers
  35. 63.

    Challenges of CI @gustavopinto Set up environment False sense of

    confidence Discipline Monetary costs Flaky tests Over-reliance on a passing build result can encourage a reviewer to merge code without a thorough review.
  36. 64.

    Challenges of CI @gustavopinto Set up environment False sense of

    confidence Discipline Monetary costs Flaky tests instead of only focusing on the code of your application, you also focus on the code of your build system
  37. 65.

    Challenges of CI @gustavopinto Set up environment False sense of

    confidence Discipline Monetary costs Flaky tests Hosted services like Travis CI are not free if you want to use them at a bigger scale or you need to maintain by yourself
  38. 66.

    Challenges of CI @gustavopinto Set up environment False sense of

    confidence Discipline Monetary costs Flaky tests Tests can sometimes be flaky, which is frustrating
  39. 68.

    @gustavopinto Inadequate testing Dependency management Carelessness Overconfidence Why build breaks

    Speed up software dev. Catch bugs/ regressions Credibility Transparency Reason for adoption
  40. 69.

    @gustavopinto Catch problems Cross-platform testing Confidence Automation Benefits Inadequate testing

    Dependency management Carelessness Overconfidence Why build breaks Speed up software dev. Catch bugs/ regressions Credibility Transparency Reason for adoption
  41. 70.

    @gustavopinto Catch problems Cross-platform testing Confidence Automation Benefits Set up

    environment False sense of confidence Discipline Monetary costs Challenges Inadequate testing Dependency management Carelessness Overconfidence Why build breaks Speed up software dev. Catch bugs/ regressions Credibility Transparency Reason for adoption
  42. 77.

    Some non-software projects use CI Devs that broken the build

    are more experienced Why? Really? @gustavopinto
  43. 78.

    Some non-software projects use CI Devs that broken the build

    are more experienced Some PLs have higher proportion of broken builds Why? Really? Why? @gustavopinto
  44. 79.

    Some non-software projects use CI Devs that broken the build

    are more experienced Some PLs have higher proportion of broken builds Some commits that lead to broken builds are not broken Why? Really? Why? How? @gustavopinto
  45. 80.

    Are casual contributors more prone to create a failing build?

    Research Question MSR’17 @gustavopinto
  46. 81.

    Methodology TravisTorrent TravisCI CI Build Data Commiter Data Dataset Dataset

    without duplicated users User Disambiguation Technique Data Cleaning Dataset with 1,074 curated projects Data Processing Data Statistical Tests @gustavopinto
  47. 83.

    # Users 0 5000 10000 15000 20000 Builds 1 2

    3 4 5+ 0 150000 300000 450000 600000 Casual Non-Casual 1,074 projects 619,370 builds # Builds 35,360 contributors @gustavopinto
  48. 84.

    Being a casual contributor is not a strong indicator for

    creating failing builds 0 250 500 750 1000 No difference Higher Casual Success Lower Casual Success # Number of Projects @gustavopinto
  49. 85.

    0 22.5 45 67.5 90 0 1.15 2.3 3.45 4.6

    Casual contributions are smaller, both in modified source-code lines and modified files Median of Modified LoC Median of Modified Files
  50. 86.

    0 1.15 2.3 3.45 4.6 No difference Higher Casual Success

    Lower Casual Success Projects in which casuals fail more than non-casuals run more jobs per build Median of jobs per build @gustavopinto