Mobile Continuous Integration at SoundCloud

In the past two years at SoundCloud, we've shipped a brand new iPhone and iPad app and completely transformed the way we work. The team scaled from 3 to 10 developers, we've moved away from pull-requests and rely heavily on automated testing. As SoundCloud grew, its mobile application got more complex and the number of contributors expanded as well. Maintaining consistency and quality across teams is key to SoundCloud's continuing success. That's why we're building custom continuous integration solutions for our mobile apps.

Transcript

1. View Slide

2. Continuous Integration at
   SoundCloud
   Istanbul Tech Talks 2015
   Vincent Garrigues
   @garriguv
   

   View Slide

3. Why invest in
   a mobile CI
   pipeline?
   

   View Slide

4. • move fast
   • keep codebase healthy
   • confidence
   • ship reliable apps
   

   View Slide

5. • move fast
   • keep codebase healthy
   • confidence
   • ship reliable apps
   

   View Slide

6. View Slide

7. View Slide

8. • started from scratch
   • months in development
   • millions of users
   

   View Slide

9. iOS Crash Complaints (avg per Week)
   0
   35
   70
   105
   140
   April May June July August
   2014 SoundCloud community team
   

   View Slide

10. iOS Crash Complaints (avg per Week)
    0
    35
    70
    105
    140
    April 2014 July 2014 October 2014 January 2015
    2015 SoundCloud community team
    

    View Slide

11. View Slide

12. How do we
    work?
    

    View Slide

13. • work on master
    • little to no branches
    • pairing
    • feature flags
    

    View Slide

14. - (id)soundcloudInternalProvider
    {
    if ([FlipTheSwitch isDevEventgatewayEnabled]) {
    return [self eventGatewayProvider];
    } else {
    return [self eventLoggerProvider];
    }
    }
    Feature flagging
    github.com/michaelengland/fliptheswitch
    

    View Slide

15. Feature flagging
    

    View Slide

16. Timeline
    

    View Slide

17. 2012 2013 2014 2015
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    

    View Slide

18. started very
    simple
    

    View Slide

19. View Slide

20. ⛔
    

    View Slide

21. View Slide

22. • unit tests
    • α and β builds
    • app store build
    

    View Slide

23. Overview of
    the current
    system
    

    View Slide

24. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β
    
    

    View Slide

25. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β
    ~ 5min 30sec ~ 7min ~ 3min
    ~ 15min
    
    

    View Slide

26. Let's have a
    look at the
    details
    

    View Slide

27. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β
    
    

    View Slide

28. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    local libraries

    unit tests
    i18n
    push
    

    View Slide

29. build
    local libraries

    unit tests
    i18n
    push
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    

    View Slide

30. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~2000
    local libraries

    unit tests
    i18n
    push
    

    View Slide

31. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~2000
    ~5000
    local libraries

    unit tests
    i18n
    push
    

    View Slide

32. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~2000
    ~5000
    local libraries

    unit tests
    i18n
    push
    

    View Slide

33. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~5000
    ~2000
    local libraries

    unit tests
    i18n
    push
    

    View Slide

34. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~5000
    ~2000
    local libraries

    unit tests
    i18n
    push
    

    View Slide

35. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β
    
    

    View Slide

36. • frank
    • cucumber
    http://www.testingwithfrank.com

    https://cukes.info
    

    View Slide

37. View Slide

38. • flexible platform & OS
    • bandwidth and reachability
    settings
    • launch arguments
    

    View Slide

39. stubby4j
    ✅
    ⛔
    datasets
    

    View Slide

40. mobile api
    proxy
    The proxy can change:
    • response status code
    • response body
    • record and undo actions
    (like, repost…)
    

    View Slide

41. ~360
    acceptance
    tests
    

    View Slide

42. ~360
    acceptance
    tests
    ~250 min
    

    View Slide

43. Sharding
    

    View Slide

44. distributed
    cucumber
    

    View Slide

45. distributed cucumber
    

    View Slide

46. x20
    VM
    

    View Slide

47. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β
    
    ~ 5min 30sec ~ 7min ~ 3min
    ~ 15min
    

    View Slide

48. acceptance
    tests
    • iOS version
    • iPhone (4S, 5, 6, 6+)
    • iPad (retina and non retina)
    • feature flag configurations
    

    View Slide

49. What's the value
    of an acceptance
    test?
    

    View Slide

50. avoid
    regressions on
    hard to test
    flows/screens
    

    View Slide

51. View Slide

52. View Slide

53. keeping the
    build green
    

    View Slide

54. View Slide

55. View Slide

56. pre-ci
    

    View Slide

57. https://www.youtube.com/watch?v=BhMSzC1crr0 SpaceX
    

    View Slide

58. Flaky tests
    

    View Slide

59. Flaky tests
    • test driver is flaky
    • backend is flaky
    • app is unpredictable
    

    View Slide

60. Flaky tests
    • identify
    • isolate
    • fix
    

    View Slide

61. flakyrazor
    

    View Slide

62. flakyrazor
    1. Take failing test out of the
    test pool
    2. Run the test multiple times
    (flaky or failing?)
    3. Assign it to the author/
    committer
    

    View Slide

63. flakyrazor
    4. Assess test value
    5. Act on test duration changes
    6. Show statistics on why the
    test failed
    

    View Slide

64. Taking care of
    our build
    machines
    

    View Slide

65. ❤
    ❤
    ❤
    ❤
    ❤
    ❤
    ❤ ❤
    

    View Slide

66. View Slide

67. • we have images for all the
    machines
    • CLIs to provision the
    machines remotely
    • OS X server stores the
    images and controls the
    imaging process
    

    View Slide

68. What about
    Android?
    

    View Slide

69. Thank you!
    Istanbul Tech Talks 2015
    Vincent Garrigues
    @garriguv
    

    View Slide