Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mobile Continuous Integration at SoundCloud

Mobile Continuous Integration at SoundCloud

In the past two years at SoundCloud, we've shipped a brand new iPhone and iPad app and completely transformed the way we work. The team scaled from 3 to 10 developers, we've moved away from pull-requests and rely heavily on automated testing. As SoundCloud grew, its backend infrastructure got more complex with the addition of many microservices. Ensuring our apps to work with these microservices in production is key to SoundCloud's continuing success. That's why we're building custom continuous integration solutions for our mobile apps.

https://www.youtube.com/watch?v=Rq721qtKKNk

Vincent Garrigues

September 16, 2015
Tweet

More Decks by Vincent Garrigues

Other Decks in Programming

Transcript

  1. View Slide

  2. Mobile Continuous Integration
    at SoundCloud
    NSSpain

    2015-09-16

    Vincent Garrigues
    @garriguv

    View Slide

  3. Why invest in
    a mobile CI
    pipeline?

    View Slide

  4. • move fast
    • keep codebase healthy
    • confidence
    • ship reliable apps

    View Slide

  5. • move fast
    • keep codebase healthy
    • confidence
    • ship reliable apps

    View Slide

  6. View Slide

  7. View Slide

  8. • started from scratch
    • months in development
    • millions of users

    View Slide

  9. iOS Crash Complaints (avg per Week)
    0
    35
    70
    105
    140
    April May June July August
    2014 SoundCloud community team

    View Slide

  10. iOS Crash Complaints (avg per Week)
    0
    35
    70
    105
    140
    April 2014 August 2014 December 2014 April 2015 August 2015
    2015 SoundCloud community team

    View Slide

  11. View Slide

  12. How do we
    work?

    View Slide


















  13. Theory

    View Slide























  14. In practice

    View Slide

  15. • pairing
    • short lived branches
    • feature flags

    View Slide

  16. - (id)soundcloudInternalProvider
    {
    if ([FlipTheSwitch isDevEventgatewayEnabled]) {
    return [self eventGatewayProvider];
    } else {
    return [self eventLoggerProvider];
    }
    }
    Feature flagging
    github.com/michaelengland/fliptheswitch

    View Slide

  17. Feature flagging
    github.com/michaelengland/fliptheswitch

    View Slide

  18. Timeline

    View Slide

  19. 2012 2013 2014 2015


















    View Slide

  20. started very
    simple

    View Slide

  21. View Slide


  22. View Slide

  23. View Slide

  24. • unit tests
    • α and β builds
    • app store build

    View Slide

  25. Overview of
    the current
    system

    View Slide

  26. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β

    View Slide

  27. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β
    ~ 10min ~ 10min ~ 3min
    ~ 20min

    View Slide

  28. Let's have a
    look at the
    details

    View Slide

  29. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β

    View Slide

  30. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    local libraries

    unit tests
    i18n
    push

    View Slide

  31. build
    local libraries

    unit tests
    i18n
    push
    unit
    tests
    acceptance
    tests build
    linter
    dependencies

    View Slide

  32. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~3000
    local libraries

    unit tests
    i18n
    push

    View Slide

  33. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~3000
    ~7000
    local libraries

    unit tests
    i18n
    push

    View Slide

  34. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~3000
    local libraries

    unit tests
    i18n
    push
    ~7000

    View Slide

  35. build
    unit
    tests
    acceptance
    tests build
    linter
    dependencies
    ~3000
    local libraries

    unit tests
    i18n
    push
    ~7000

    View Slide

  36. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β

    View Slide

  37. • frank
    • cucumber
    http://www.testingwithfrank.com

    https://cukes.info

    View Slide

  38. View Slide

  39. • flexible platform & OS
    • bandwidth and reachability
    settings
    • launch arguments

    View Slide

  40. mobile api
    proxy
    The proxy can change:
    • response status code
    • response body
    • record and undo actions
    (like, repost…)

    View Slide

  41. ~350
    acceptance
    tests

    View Slide

  42. ~350
    acceptance
    tests
    ~200 min

    View Slide

  43. distributed
    cucumber

    View Slide

  44. distributed cucumber

    View Slide

  45. x20
    VM
    VM

    View Slide

  46. build

    analysis

    unit tests
    acceptance
    tests
    AppStore

    AdHoc

    α and β

    ~ 10min ~ 10min ~ 3min
    ~ 20min

    View Slide

  47. acceptance
    tests
    • iOS version
    • iPhone (4S, 5, 6, 6+)
    • iPad (retina and non retina)
    • feature flag configurations

    View Slide

  48. What's the value
    of an acceptance
    test?

    View Slide

  49. avoid
    regressions on
    hard to test
    flows/screens

    View Slide

  50. View Slide

  51. View Slide

  52. keeping the
    build green

    View Slide

  53. View Slide

  54. View Slide

  55. pre-ci

    View Slide

  56. View Slide

  57. https://www.youtube.com/watch?v=BhMSzC1crr0 SpaceX

    View Slide

  58. Flaky tests

    View Slide

  59. Flaky tests
    • test driver is flaky
    • backend is flaky
    • app is unpredictable

    View Slide

  60. Flaky tests
    • identify
    • isolate
    • fix

    View Slide

  61. flakyrazor

    View Slide

  62. flakyrazor
    1. Take failing test out of the
    test pool
    2. Run the test multiple times
    (flaky or failing?)
    3. Assign it to the author/
    committer

    View Slide

  63. flakyrazor
    4. Assess test value
    5. Act on test duration changes
    6. Show statistics on why the
    test failed

    View Slide

  64. Taking care of
    our build
    machines

    View Slide







  65. ❤ ❤

    View Slide

  66. View Slide

  67. • we have images for all the
    machines
    • CLIs to provision the
    machines remotely
    • OS X server stores the
    images and controls the
    imaging process

    View Slide

  68. Thank you!
    NSSpain

    2015-09-16

    Vincent Garrigues
    @garriguv

    View Slide