KEEPING DEVELOPERS HAPPY WITH A FAST CI

When talking about performance, most developers think application speed, faster algorithms or better data structures. But what about your test suite? CI time is developer waiting time!

At Shopify we have more than 170,000 Ruby tests and we add 30,000 more annually. The sheer amount of tests and their growth requires some aggressive methods. We will illustrate some of our techniques including monitoring, test selection, timeouts and the 80/20 rule. If you have experience in writing tests and want to learn tricks on how to speed up your test suite, this talk is for you!

Transcript

1. Keeping 
 Developers 
 Happy with a 
 Fast CI
2. Hello! I am 
 Christian Bruckmayer 
 
 @bruckmayer 
 bruckmayer.net
3. Data Driven Development Overview Test Optimisation Test Selection Test Prioritisation
5. Test 
 Infrastructure
6. Quality 
 Assurance
7. Ensure that our 
 CI Systems are 
 Scalable, Robust, and Usable with an emphasis on our Monolith
8. Team?
9. ~2.8M Lines Of Code 
 > 1500 Engineers ~210k Ruby Tests 
 ~40 Hours to Execute Sequentially 
 >1000 Builds per Day 
 ~100Mil Test Runs per Day
10. Growth
12. Data Driven Development Overview Test Selection Test Optimisation Test Budget
13. Data driven means progress in an activity is compelled by data, rather than by intuition or by personal experience.
15. Command Build Job
17. Data Warehouse
18. p95 Build Time Distribution
19. There Is a Problem 
 vs. 
 Where Is the Problem
20. Command
22. Data Warehouse
25. Data Driven Development Overview Test Optimisation Test Selection Test Prioritisation
26. Number of Tests ~210k
27. p95 Build Time Distribution
28. Pattern
30. 30 min 
 Timeout
35. p95 Build Time Distribution
36. p95 Build Time Distribution
38. Test 
 Suite 
 Optimisation
39. 80/20
40. 99/1
41. Data Driven Development Overview Test Optimisation Test Selection Test Prioritisation
42. Test 
 Selection Test 
 Suite Only Relevant 
 to Pull Request Changes
43. ~300k Number of Tests
44. Stability
45. 1. Map App Files to Test Files 
 
 2. Use Map to Run Tests Related to PR Changes
46. Strategy A Test Selection tests/item_test.rb Strategy B app/models/item.rb
47. Ruby Files 
 Fixtures 
 Translations 
 Con fi gurations 
 Java Script 
 …. 

48. Dynamic 
 Language
49. Meta 
 Programming
50. Dynamic 
 Analysis
51. 1.
52. 2.
53. 3.
54. 3.
55. 3.
56. Output
57. Reverse Mapping
58. Test Selection
59. Ruby Files 
 Fixtures 
 Translations 
 Con fi gurations 
 Java Script 
 …. 

60. ActiveSupport 
 Noti fi cations
61. redirect_to.action_controller halted_callback.action_controller unpermitted_parameters.action_controller process_middleware.action_dispatch render_template.action_view render_partial.action_view render_collection.action_view sql.active_record instantiation.active_record … write_fragment.action_controller read_fragment.action_controller expire_fragment.action_controller exist_fragment?.action_controller write_page.action_controller expire_page.action_controller start_processing.action_controller process_action.action_controller send_ fi le.action_controller send_data.action_controller 51 Hooks
63. 40%
64. 210k 
 vs 
 85k
65. Data Driven Development Overview Test Optimisation Test Selection Test Prioritisation
66. Expectation
67. 100% Con fi dence?
68. Test Value
69. Prioritisation
70. Prioritisation
71. Random 
 Failure Rate 
 Complexity 
 Churn 
 Coverage 
 Average Duration
72. How to compare?
73. Continuous Integration Minitest … … …
74. Prioritisation Pipeline Random Complexity Churn …
75. Metric Random = 0.4 Complexity = 0.8 Churn = 0.5 …
76. Time To First Failure Duration Churn Complexity Coverage Random Failure Rate 12 min 10 min 8 min 6 min 3 min
77. Time To First Failure Duration Churn Complexity Coverage Random Failure Rate 12 min 10 min 8 min 6 min 3 min Median
78. Time To First Failure Duration Churn Complexity Coverage Random Failure Rate 12 min 10 min 8 min 6 min 3 min Min
79. Time To First Failure Duration Churn Complexity Coverage Random Failure Rate 12 min 10 min 8 min 6 min 3 min Max
80. Time To First Failure Duration Churn Complexity Coverage Random Failure Rate 12 min 10 min 8 min 6 min 3 min Outliers
81. Prioritisation
82. Tests Executed Faults Detected 10% 10% 100% 100% Random Failure Rate Median C.R. Fang, Z.Y. Chen, B.W. Xu, Comparing logic coverage criteria on test case prioritization, Sci. China Inf. Sci., 55 (2012), pp. 2826-2840
83. Tests Executed Faults Detected 10% 10% 100% 100% Random Failure Rate p95 C.R. Fang, Z.Y. Chen, B.W. Xu, Comparing logic coverage criteria on test case prioritization, Sci. China Inf. Sci., 55 (2012), pp. 2826-2840
84. One Failure 
 vs. 
 All Failures
85. Experiment
86. Data Driven Development Overview Test Optimisation Test Selection Test Prioritisation
87. Rome 
 Wasn’t 
 Built In a Day
88. Continuous Integration 
 vs. 
 Production
89. Experiments
90. Load Testing
91. Data Driven Development
93. Questions?
94. Thank You! 
 
 @bruckmayer 
 christian@bruckmayer.net