Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mobile Continuous Integration at SoundCloud

Mobile Continuous Integration at SoundCloud

In the past two years at SoundCloud, we've shipped a brand new iPhone and iPad app and completely transformed the way we work. The team scaled from 3 to 10 developers, we've moved away from pull-requests and rely heavily on automated testing. As SoundCloud grew, its mobile application got more complex and the number of contributors expanded as well. Maintaining consistency and quality across teams is key to SoundCloud's continuing success. That's why we're building custom continuous integration solutions for our mobile apps.

Vincent Garrigues

April 27, 2015
Tweet

More Decks by Vincent Garrigues

Other Decks in Programming

Transcript

  1. View Slide

  2. Continuous Integration at
    SoundCloud
    Istanbul Tech Talks 2015
    Vincent Garrigues
    @garriguv

    View Slide

  3. Why invest in
    a mobile CI
    pipeline?

    View Slide

  4. • move fast
    • keep codebase healthy
    • confidence
    • ship reliable apps

    View Slide

  5. • move fast
    • keep codebase healthy
    • confidence
    • ship reliable apps

    View Slide

  6. View Slide

  7. View Slide

  8. • started from scratch
    • months in development
    • millions of users

    View Slide

  9. iOS Crash Complaints (avg per Week)
    0
    35
    70
    105
    140
    April May June July August
    2014 SoundCloud community team

    View Slide

  10. iOS Crash Complaints (avg per Week)
    0
    35
    70
    105
    140
    April 2014 July 2014 October 2014 January 2015
    2015 SoundCloud community team

    View Slide

  11. View Slide

  12. How do we
    work?

    View Slide

  13. • work on master
    • little to no branches
    • pairing
    • feature flags

    View Slide

  14. - (id)soundcloudInternalProvider
    {
    if ([FlipTheSwitch isDevEventgatewayEnabled]) {
    return [self eventGatewayProvider];
    } else {
    return [self eventLoggerProvider];
    }
    }
    Feature flagging
    github.com/michaelengland/fliptheswitch

    View Slide

  15. Feature flagging

    View Slide

  16. Timeline

    View Slide

  17. 2012 2013 2014 2015


















    View Slide

  18. started very
    simple

    View Slide

  19. View Slide


  20. View Slide

  21. View Slide

  22. • unit tests
    • α and β builds
    • app store build

    View Slide

  23. Overview of
    the current
    system

    View Slide

  24. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β

    View Slide

  25. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β
    ~ 5min 30sec ~ 7min ~ 3min
    ~ 15min

    View Slide

  26. Let's have a
    look at the
    details

    View Slide

  27. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β

    View Slide

  28. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    local libraries

    unit tests
    i18n
    push

    View Slide

  29. build
    local libraries

    unit tests
    i18n
    push
    unit
    tests
    acceptance
    tests build
    linter
    dependencies

    View Slide

  30. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~2000
    local libraries

    unit tests
    i18n
    push

    View Slide

  31. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~2000
    ~5000
    local libraries

    unit tests
    i18n
    push

    View Slide

  32. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~2000
    ~5000
    local libraries

    unit tests
    i18n
    push

    View Slide

  33. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~5000
    ~2000
    local libraries

    unit tests
    i18n
    push

    View Slide

  34. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~5000
    ~2000
    local libraries

    unit tests
    i18n
    push

    View Slide

  35. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β

    View Slide

  36. • frank
    • cucumber
    http://www.testingwithfrank.com

    https://cukes.info

    View Slide

  37. View Slide

  38. • flexible platform & OS
    • bandwidth and reachability
    settings
    • launch arguments

    View Slide

  39. stubby4j


    datasets

    View Slide

  40. mobile api
    proxy
    The proxy can change:
    • response status code
    • response body
    • record and undo actions
    (like, repost…)

    View Slide

  41. ~360
    acceptance
    tests

    View Slide

  42. ~360
    acceptance
    tests
    ~250 min

    View Slide

  43. Sharding

    View Slide

  44. distributed
    cucumber

    View Slide

  45. distributed cucumber

    View Slide

  46. x20
    VM

    View Slide

  47. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β

    ~ 5min 30sec ~ 7min ~ 3min
    ~ 15min

    View Slide

  48. acceptance
    tests
    • iOS version
    • iPhone (4S, 5, 6, 6+)
    • iPad (retina and non retina)
    • feature flag configurations

    View Slide

  49. What's the value
    of an acceptance
    test?

    View Slide

  50. avoid
    regressions on
    hard to test
    flows/screens

    View Slide

  51. View Slide

  52. View Slide

  53. keeping the
    build green

    View Slide

  54. View Slide

  55. View Slide

  56. pre-ci

    View Slide

  57. https://www.youtube.com/watch?v=BhMSzC1crr0 SpaceX

    View Slide

  58. Flaky tests

    View Slide

  59. Flaky tests
    • test driver is flaky
    • backend is flaky
    • app is unpredictable

    View Slide

  60. Flaky tests
    • identify
    • isolate
    • fix

    View Slide

  61. flakyrazor

    View Slide

  62. flakyrazor
    1. Take failing test out of the
    test pool
    2. Run the test multiple times
    (flaky or failing?)
    3. Assign it to the author/
    committer

    View Slide

  63. flakyrazor
    4. Assess test value
    5. Act on test duration changes
    6. Show statistics on why the
    test failed

    View Slide

  64. Taking care of
    our build
    machines

    View Slide







  65. ❤ ❤

    View Slide

  66. View Slide

  67. • we have images for all the
    machines
    • CLIs to provision the
    machines remotely
    • OS X server stores the
    images and controls the
    imaging process

    View Slide

  68. What about
    Android?

    View Slide

  69. Thank you!
    Istanbul Tech Talks 2015
    Vincent Garrigues
    @garriguv

    View Slide