Mobile Continuous Integration at SoundCloud

In the past two years at SoundCloud, we've shipped a brand new iPhone and iPad app and completely transformed the way we work. The team scaled from 3 to 10 developers, we've moved away from pull-requests and rely heavily on automated testing. As SoundCloud grew, its mobile application got more complex and the number of contributors expanded as well. Maintaining consistency and quality across teams is key to SoundCloud's continuing success. That's why we're building custom continuous integration solutions for our mobile apps.

Transcript

2. Continuous Integration at SoundCloud Istanbul Tech Talks 2015 Vincent Garrigues @garriguv
3. Why invest in a mobile CI pipeline?
4. • move fast • keep codebase healthy • confidence • ship reliable apps
5. • move fast • keep codebase healthy • confidence • ship reliable apps
8. • started from scratch • months in development • millions of users
9. iOS Crash Complaints (avg per Week) 0 35 70 105 140 April May June July August 2014 SoundCloud community team
10. iOS Crash Complaints (avg per Week) 0 35 70 105 140 April 2014 July 2014 October 2014 January 2015 2015 SoundCloud community team
12. How do we work?
13. • work on master • little to no branches • pairing • feature flags
14. - (id<AnalyticsProviderInterface>)soundcloudInternalProvider { if ([FlipTheSwitch isDevEventgatewayEnabled]) { return [self eventGatewayProvider]; } else { return [self eventLoggerProvider]; } } Feature flagging github.com/michaelengland/fliptheswitch
15. Feature flagging
16. Timeline
17. 2012 2013 2014 2015
18. started very simple
20. ⛔
22. • unit tests • α and β builds • app store build
23. Overview of the current system
24. build
 analysis
 unit tests acceptance tests AppStore
 AdHoc
 α and β
25. build
 analysis
 unit tests acceptance tests AppStore
 AdHoc
 α and β ~ 5min 30sec ~ 7min ~ 3min ~ 15min
26. Let's have a look at the details
27. build
 analysis
 unit tests acceptance tests AppStore
 AdHoc
 α and β
28. build unit tests acceptance tests build linter dependencies local libraries
 unit tests i18n push
29. build local libraries
 unit tests i18n push unit tests acceptance tests build linter dependencies
30. build unit tests acceptance tests build linter dependencies ~2000 local libraries
 unit tests i18n push
31. build unit tests acceptance tests build linter dependencies ~2000 ~5000 local libraries
 unit tests i18n push
32. build unit tests acceptance tests build linter dependencies ~2000 ~5000 local libraries
 unit tests i18n push
33. build unit tests acceptance tests build linter dependencies ~5000 ~2000 local libraries
 unit tests i18n push
34. build unit tests acceptance tests build linter dependencies ~5000 ~2000 local libraries
 unit tests i18n push
35. build
 analysis
 unit tests acceptance tests AppStore
 AdHoc
 α and β
36. • frank • cucumber http://www.testingwithfrank.com
 https://cukes.info
38. • flexible platform & OS • bandwidth and reachability settings • launch arguments
39. stubby4j ✅ ⛔ datasets
40. mobile api proxy The proxy can change: • response status code • response body • record and undo actions (like, repost…)
41. ~360 acceptance tests
42. ~360 acceptance tests ~250 min
43. Sharding
44. distributed cucumber
45. distributed cucumber
46. x20 VM
47. build
 analysis
 unit tests acceptance tests AppStore
 AdHoc
 α and β ~ 5min 30sec ~ 7min ~ 3min ~ 15min
48. acceptance tests • iOS version • iPhone (4S, 5, 6, 6+) • iPad (retina and non retina) • feature flag configurations
49. What's the value of an acceptance test?
50. avoid regressions on hard to test flows/screens
53. keeping the build green
56. pre-ci
57. https://www.youtube.com/watch?v=BhMSzC1crr0 SpaceX
58. Flaky tests
59. Flaky tests • test driver is flaky • backend is flaky • app is unpredictable
60. Flaky tests • identify • isolate • fix
61. flakyrazor
62. flakyrazor 1. Take failing test out of the test pool 2. Run the test multiple times (flaky or failing?) 3. Assign it to the author/ committer
63. flakyrazor 4. Assess test value 5. Act on test duration changes 6. Show statistics on why the test failed
64. Taking care of our build machines
65. ❤ ❤ ❤ ❤ ❤ ❤ ❤ ❤
67. • we have images for all the machines • CLIs to provision the machines remotely • OS X server stores the images and controls the imaging process
68. What about Android?
69. Thank you! Istanbul Tech Talks 2015 Vincent Garrigues @garriguv