Mobile Continuous Integration at SoundCloud

Mobile Continuous Integration at SoundCloud

In the past two years at SoundCloud, we've shipped a brand new iPhone and iPad app and completely transformed the way we work. The team scaled from 3 to 10 developers, we've moved away from pull-requests and rely heavily on automated testing. As SoundCloud grew, its mobile application got more complex and the number of contributors expanded as well. Maintaining consistency and quality across teams is key to SoundCloud's continuing success. That's why we're building custom continuous integration solutions for our mobile apps.

E76dc85e3486993280ae6b4d2810f670?s=128

Vincent Garrigues

April 27, 2015
Tweet

Transcript

  1. None
  2. Continuous Integration at SoundCloud Istanbul Tech Talks 2015 Vincent Garrigues

    @garriguv
  3. Why invest in a mobile CI pipeline?

  4. • move fast • keep codebase healthy • confidence •

    ship reliable apps
  5. • move fast • keep codebase healthy • confidence •

    ship reliable apps
  6. None
  7. None
  8. • started from scratch • months in development • millions

    of users
  9. iOS Crash Complaints (avg per Week) 0 35 70 105

    140 April May June July August 2014 SoundCloud community team
  10. iOS Crash Complaints (avg per Week) 0 35 70 105

    140 April 2014 July 2014 October 2014 January 2015 2015 SoundCloud community team
  11. None
  12. How do we work?

  13. • work on master • little to no branches •

    pairing • feature flags
  14. - (id<AnalyticsProviderInterface>)soundcloudInternalProvider { if ([FlipTheSwitch isDevEventgatewayEnabled]) { return [self eventGatewayProvider];

    } else { return [self eventLoggerProvider]; } } Feature flagging github.com/michaelengland/fliptheswitch
  15. Feature flagging

  16. Timeline

  17. 2012 2013 2014 2015

  18. started very simple

  19. None
  20. None
  21. • unit tests • α and β builds • app

    store build
  22. Overview of the current system

  23. build
 analysis
 unit tests acceptance tests AppStore
 AdHoc
 α and

    β
  24. build
 analysis
 unit tests acceptance tests AppStore
 AdHoc
 α and

    β ~ 5min 30sec ~ 7min ~ 3min ~ 15min
  25. Let's have a look at the details

  26. build
 analysis
 unit tests acceptance tests AppStore
 AdHoc
 α and

    β
  27. build unit tests acceptance tests build linter dependencies local libraries


    unit tests i18n push
  28. build local libraries
 unit tests i18n push unit tests acceptance

    tests build linter dependencies
  29. build unit tests acceptance tests build linter dependencies ~2000 local

    libraries
 unit tests i18n push
  30. build unit tests acceptance tests build linter dependencies ~2000 ~5000

    local libraries
 unit tests i18n push
  31. build unit tests acceptance tests build linter dependencies ~2000 ~5000

    local libraries
 unit tests i18n push
  32. build unit tests acceptance tests build linter dependencies ~5000 ~2000

    local libraries
 unit tests i18n push
  33. build unit tests acceptance tests build linter dependencies ~5000 ~2000

    local libraries
 unit tests i18n push
  34. build
 analysis
 unit tests acceptance tests AppStore
 AdHoc
 α and

    β
  35. • frank • cucumber http://www.testingwithfrank.com
 https://cukes.info

  36. None
  37. • flexible platform & OS • bandwidth and reachability settings

    • launch arguments
  38. stubby4j ✅ ⛔ datasets

  39. mobile api proxy The proxy can change: • response status

    code • response body • record and undo actions (like, repost…)
  40. ~360 acceptance tests

  41. ~360 acceptance tests ~250 min

  42. Sharding

  43. distributed cucumber

  44. distributed cucumber

  45. x20 VM

  46. build
 analysis
 unit tests acceptance tests AppStore
 AdHoc
 α and

    β ~ 5min 30sec ~ 7min ~ 3min ~ 15min
  47. acceptance tests • iOS version • iPhone (4S, 5, 6,

    6+) • iPad (retina and non retina) • feature flag configurations
  48. What's the value of an acceptance test?

  49. avoid regressions on hard to test flows/screens

  50. None
  51. None
  52. keeping the build green

  53. None
  54. None
  55. pre-ci

  56. https://www.youtube.com/watch?v=BhMSzC1crr0 SpaceX

  57. Flaky tests

  58. Flaky tests • test driver is flaky • backend is

    flaky • app is unpredictable
  59. Flaky tests • identify • isolate • fix

  60. flakyrazor

  61. flakyrazor 1. Take failing test out of the test pool

    2. Run the test multiple times (flaky or failing?) 3. Assign it to the author/ committer
  62. flakyrazor 4. Assess test value 5. Act on test duration

    changes 6. Show statistics on why the test failed
  63. Taking care of our build machines

  64. ❤ ❤ ❤ ❤ ❤ ❤ ❤ ❤

  65. None
  66. • we have images for all the machines • CLIs

    to provision the machines remotely • OS X server stores the images and controls the imaging process
  67. What about Android?

  68. Thank you! Istanbul Tech Talks 2015 Vincent Garrigues @garriguv