From hundred (86) to zero (7). How we publish 67 Android apps in under 7 minutes.

Given at Mobile DevOps Summit 2023

Imagine being able to publish 67 Android white-labeled-apps in under 7 minutes. It may seem like an impossible feat, but our team has turned this into a reality.

Since 2017, we have committed ourselves to a rigorous publishing schedule, releasing our apps every two weeks without fail. This relentless dedication has taught us valuable lessons about the importance of investing in our release pipeline and continuously improving it.

In this talk, we will share the secrets behind our remarkable achievement and delve into the strategies and tools that have enabled us to streamline the publishing process.

3 key takeaways from this session:
1. Continuously investing into your release pipeline is critical for your success
2. Sometimes it is not obvious which changes require more attention
3. ...

Transcript

1. From hundred (86) to zero (7).
   How we publish 67 Android
   apps in under 7 minutes.
   

   View Slide

2. This is me
   Stefan May
   Live in Germany
   Android Developer since 2014
   Senior Android Developer @ioki since 2020
   StefMa.medium.com github.com/StefMa x.com/StefMa91 StefMa.guru
   

   View Slide

3. Preamble
   White labelling
   multiple apps
   

   View Slide

4. Preamble
   White labelling
   multiple apps
   Bi-Weekly release
   

   View Slide

5. Preamble
   White labelling
   multiple apps
   Bi-Weekly release
   Release time ~30 min
   

   View Slide

6. Story time!
   Penetration test (pentest) 2022
   

   View Slide

7. Story time!
   Penetration test (pentest) 2022
   Recommendation:
   Obfuscate source code (/w Proguard/R8)
   

   View Slide

8. 86 min
   

   View Slide

9. More fun
   Machine with less specs
   increase the build time to >90 min
   

   View Slide

10. More fun
    Machine with less specs
    increase the build time to >90 min
    CI build timeout of 90 minutes!
    

    View Slide

11. More fun
    Machine with less specs
    increase the build time to >90 min
    CI build timeout of 90 minutes!
    Solution:
    Disable Proguard/R8 again
    

    View Slide

12. More fun
    Machine with less specs
    increase the build time to >90 min
    CI build timeout of 90 minutes!
    Solution Hotfix:
    Disable Proguard/R8 again
    

    View Slide

13. Why?
    

    View Slide

14. Why?
    Multiple apps (white labelling)
    

    View Slide

15. Why?
    Multiple apps (white labelling)
    R8 runs sequential
    

    View Slide

16. Why?
    Multiple apps (white labelling)
    R8 runs sequential
    Single CI workflow to build them all
    

    View Slide

17. Why?
    Multiple apps (white labelling)
    R8 runs sequential
    Single CI workflow to build them all
    

    View Slide

18. Fun fun fun
    Hotfixes for single apps
    

    View Slide

19. Fun fun fun
    Hotfixes for single apps
    Google servers returns HTTP status code 500
    

    View Slide

20. Find a way out
    

    View Slide

21. Find a way out
    What we can not change: What we can change:
    

    View Slide

22. Find a way out
    What we can not change:
    Multiple apps
    What we can change:
    

    View Slide

23. Find a way out
    What we can not change:
    Multiple apps
    R8 in sequence
    What we can change:
    

    View Slide

24. Find a way out
    What we can not change:
    Multiple apps
    R8 in sequence
    Hotfix for single apps
    What we can change:
    

    View Slide

25. Find a way out
    What we can not change:
    Multiple apps
    R8 in sequence
    Hotfix for single apps
    Google servers HTTP status code 500
    What we can change:
    

    View Slide

26. Find a way out
    What we can not change:
    Multiple apps
    R8 in sequence
    Hotfix for single apps
    Google servers HTTP status code 500
    What we can change:
    Single CI workflow to build them all
    

    View Slide

27. Let’s change it
    

    View Slide

28. WLA0
    WLA1
    WLA2
    WLA3
    WLA4
    WLA5
    WLA6
    WLA7
    WLA8
    WLA9
    WLA10
    WLA11
    WLA12
    WLA13
    WLA14
    Single Workflow
    

    View Slide

29. orchestra
    workflow
    WLA1
    WF
    WLA2
    WF
    WLA3
    WF
    WLA4
    WF
    WLA5
    WF
    WLA6
    WF
    WLA7
    WF
    WLA8
    WF
    A0
    F
    WLA10
    WF
    WLA11
    WF
    WLA12
    WF
    WLA13
    WF
    WLA14
    WF
    WLA15
    WF
    WLA16
    WF
    W
    W
    WLA9
    WF
    WLA19
    WF
    WLA20
    WF
    WLA21
    WF
    WLA22
    WF
    WLA23
    WF
    WLA24
    WF
    WLA25
    WF
    WLA26
    WF
    A18
    F
    

    View Slide

30. Let’s change it – Fixes
    Hotfixes on single apps ✅
    

    View Slide

31. Let’s change it – Fixes
    Hotfixes on single apps ✅
    Google server HTTP status code 500 ✅
    

    View Slide

32. Let’s change it – Fixes
    Hotfixes on single apps ✅
    Google server HTTP status code 500 ✅
    Don’t exceed build timeout ✅
    

    View Slide

33. Let’s change it – Fixes
    Hotfixes on single apps ✅
    Google server HTTP status code 500 ✅
    Don’t exceed build timeout ✅
    Scales horizontally ✅
    

    View Slide

34. Half time show
    

    View Slide

35. Half time show
    Improved CI build time
    from 86 min to stable 7 min
    

    View Slide

36. Half time show
    Improved CI build time
    from 86 min to stable 7 min
    Replace single workflow with multiple ones
    

    View Slide

37. Half time show
    Improved CI build time
    from 86 min to stable 7 min
    Replace single workflow with multiple ones
    R8 enabled!
    

    View Slide

38. But!
    There is always a but, right?
    

    View Slide

39. But! – Architecture
    - app
    build.gradle
    | include("lib-wla")
    | include("lib-logger")
    | include("feature-registration")
    | include("feature-ride-creation")
    src/main/someKotlinClasses
    - lib-wla
    - lib-logger
    - feature-registration
    - feature-ride-creation
    

    View Slide

40. But! – Architecture
    - app
    build.gradle
    | include("lib-wla")
    | include("lib-logger")
    | include("feature-registration")
    | include("feature-ride-creation")
    src/main/someKotlinClasses
    - lib-wla
    - lib-logger
    - feature-registration
    - feature-ride-creation
    - ioki/app
    build.gradle
    | include(":ioki-library:lib-core")
    - lib-core
    build.gradle
    | include("lib-logger")
    | include("feature-registration")
    | include("feature-ride-creation")
    src/main/someKotlinClasses
    - lib-logger
    - feature-registration
    - feature-ride-creation
    

    View Slide

41. But! – Architecture
    - app
    build.gradle
    | include("lib-wla")
    | include("lib-logger")
    | include("feature-registration")
    | include("feature-ride-creation")
    src/main/someKotlinClasses
    - lib-wla
    - lib-logger
    - feature-registration
    - feature-ride-creation
    - ioki/app
    build.gradle
    | include(":ioki-library:lib-core")
    - lib-core
    build.gradle
    | include("lib-logger")
    | include("feature-registration")
    | include("feature-ride-creation")
    src/main/someKotlinClasses
    - lib-logger
    - feature-registration
    - feature-ride-creation
    

    View Slide

42. But! – Nexus
    Publish library to Nexus
    a maven repository
    

    View Slide

43. But! – Nexus
    Publish library to Nexus
    a maven repository
    Library lint & test happens on library release day
    

    View Slide

44. But! – Nexus
    Publish library to Nexus
    a maven repository
    Library lint & test happens on library release day
    Apps consume library on app release day
    

    View Slide

45. Summary
    

    View Slide

46. Summary
    Improved CI build time
    from 86 min to stable 7 min
    Replace single workflow with multiple ones
    R8 enabled!
    

    View Slide

47. Summary
    Improved CI build time
    from 86 min to stable 7 min
    Replace single workflow with multiple ones
    R8 enabled!
    Possible because:
    

    View Slide

48. Summary
    Improved CI build time
    from 86 min to stable 7 min
    Replace single workflow with multiple ones
    R8 enabled!
    Possible because:
    Architecture & Nexus
    

    View Slide

49. Thank you,
    any questions?
    

    View Slide