Mobile Continuous Integration at SoundCloud

In the past two years at SoundCloud, we've shipped a brand new iPhone and iPad app and completely transformed the way we work. The team scaled from 3 to 10 developers, we've moved away from pull-requests and rely heavily on automated testing. As SoundCloud grew, its backend infrastructure got more complex with the addition of many microservices. Ensuring our apps to work with these microservices in production is key to SoundCloud's continuing success. That's why we're building custom continuous integration solutions for our mobile apps.

https://www.youtube.com/watch?v=Rq721qtKKNk

Transcript

1. View Slide

2. Mobile Continuous Integration
   at SoundCloud
   NSSpain

   2015-09-16

   Vincent Garrigues
   @garriguv
   

   View Slide

3. Why invest in
   a mobile CI
   pipeline?
   

   View Slide

4. • move fast
   • keep codebase healthy
   • confidence
   • ship reliable apps
   

   View Slide

5. • move fast
   • keep codebase healthy
   • confidence
   • ship reliable apps
   

   View Slide

6. View Slide

7. View Slide

8. • started from scratch
   • months in development
   • millions of users
   

   View Slide

9. iOS Crash Complaints (avg per Week)
   0
   35
   70
   105
   140
   April May June July August
   2014 SoundCloud community team
   

   View Slide

10. iOS Crash Complaints (avg per Week)
    0
    35
    70
    105
    140
    April 2014 August 2014 December 2014 April 2015 August 2015
    2015 SoundCloud community team
    

    View Slide

11. View Slide

12. How do we
    work?
    

    View Slide

13. 
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    Theory
    

    View Slide

14. 
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    In practice
    

    View Slide

15. • pairing
    • short lived branches
    • feature flags
    

    View Slide

16. - (id)soundcloudInternalProvider
    {
    if ([FlipTheSwitch isDevEventgatewayEnabled]) {
    return [self eventGatewayProvider];
    } else {
    return [self eventLoggerProvider];
    }
    }
    Feature flagging
    github.com/michaelengland/fliptheswitch
    

    View Slide

17. Feature flagging
    github.com/michaelengland/fliptheswitch
    

    View Slide

18. Timeline
    

    View Slide

19. 2012 2013 2014 2015
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    

    View Slide

20. started very
    simple
    

    View Slide

21. View Slide

22. ⛔
    

    View Slide

23. View Slide

24. • unit tests
    • α and β builds
    • app store build
    

    View Slide

25. Overview of
    the current
    system
    

    View Slide

26. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β
    
    

    View Slide

27. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β
    ~ 10min ~ 10min ~ 3min
    ~ 20min
    
    

    View Slide

28. Let's have a
    look at the
    details
    

    View Slide

29. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β
    
    

    View Slide

30. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    local libraries

    unit tests
    i18n
    push
    

    View Slide

31. build
    local libraries

    unit tests
    i18n
    push
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    

    View Slide

32. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~3000
    local libraries

    unit tests
    i18n
    push
    

    View Slide

33. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~3000
    ~7000
    local libraries

    unit tests
    i18n
    push
    

    View Slide

34. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~3000
    local libraries

    unit tests
    i18n
    push
    ~7000
    

    View Slide

35. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~3000
    local libraries

    unit tests
    i18n
    push
    ~7000
    

    View Slide

36. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β
    
    

    View Slide

37. • frank
    • cucumber
    http://www.testingwithfrank.com

    https://cukes.info
    

    View Slide

38. View Slide

39. • flexible platform & OS
    • bandwidth and reachability
    settings
    • launch arguments
    

    View Slide

40. mobile api
    proxy
    The proxy can change:
    • response status code
    • response body
    • record and undo actions
    (like, repost…)
    

    View Slide

41. ~350
    acceptance
    tests
    

    View Slide

42. ~350
    acceptance
    tests
    ~200 min
    

    View Slide

43. distributed
    cucumber
    

    View Slide

44. distributed cucumber
    

    View Slide

45. x20
    VM
    VM
    

    View Slide

46. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β
    
    ~ 10min ~ 10min ~ 3min
    ~ 20min
    

    View Slide

47. acceptance
    tests
    • iOS version
    • iPhone (4S, 5, 6, 6+)
    • iPad (retina and non retina)
    • feature flag configurations
    

    View Slide

48. What's the value
    of an acceptance
    test?
    

    View Slide

49. avoid
    regressions on
    hard to test
    flows/screens
    

    View Slide

50. View Slide

51. View Slide

52. keeping the
    build green
    

    View Slide

53. View Slide

54. View Slide

55. pre-ci
    

    View Slide

56. View Slide

57. https://www.youtube.com/watch?v=BhMSzC1crr0 SpaceX
    

    View Slide

58. Flaky tests
    

    View Slide

59. Flaky tests
    • test driver is flaky
    • backend is flaky
    • app is unpredictable
    

    View Slide

60. Flaky tests
    • identify
    • isolate
    • fix
    

    View Slide

61. flakyrazor
    

    View Slide

62. flakyrazor
    1. Take failing test out of the
    test pool
    2. Run the test multiple times
    (flaky or failing?)
    3. Assign it to the author/
    committer
    

    View Slide

63. flakyrazor
    4. Assess test value
    5. Act on test duration changes
    6. Show statistics on why the
    test failed
    

    View Slide

64. Taking care of
    our build
    machines
    

    View Slide

65. ❤
    ❤
    ❤
    ❤
    ❤
    ❤
    ❤ ❤
    

    View Slide

66. View Slide

67. • we have images for all the
    machines
    • CLIs to provision the
    machines remotely
    • OS X server stores the
    images and controls the
    imaging process
    

    View Slide

68. Thank you!
    NSSpain

    2015-09-16

    Vincent Garrigues
    @garriguv
    

    View Slide