Mobile Release Engineering

Mobile app releases are a manual, tedious and error-prone process when done at scale.

It is not possible to follow a continuous delivery (CD) approach because binaries need to be submitted to the stores, and once published they can’t be rolled back. Moreover, mobile releases can take several days to complete.

In contrast, we deploy the web version of Shopify around 50 times a day. We identified this gap between web and mobile releases as a problem.

In this talk, I’ll explain how my team built a continuous integration (CI) system for our mobile apps (Android and iOS), and how we used this infrastructure as a foundation to build a Ruby on Rails application and other tools to automate and orchestrate releases to Google Play and the App Store, reducing the need for human interaction as much as possible.

We’ll see the challenges of standardizing the release process and the coordination with third-party services like GitHub and the store APIs, and how we made it easier for developers to test the apps.

Transcript

1. Alejandro Rodríguez Salamanca Mobile Release Engineering
4. This talk • How does the mobile release process work? • How is it different from web deployment? • How did we improve the mobile release process?
7. Problems • Each app had its own release process • Apps became more complex and so did their release processes • The number of apps increased
8. Is Continuous Deployment possible for mobile? Not really.
9. Merge → Build container → Run CI → Ship to production
11. Pick a commit → Increase build and version numbers → Run CI → Test the app → Fix bugs → Run CI → Test the app → Fix bugs → … → Update Screenshots → Update Release Notes → Ship to the store Wait for approval → Start again if the app is not approved
12. Mobile releases are a discrete process.
13. Mobile releases are a discrete process • The stores take care of the distribution • Users don’t have new versions of the app right away. They have to update it.
14. Mobile releases can’t be rolled back.
15. Mobile releases can’t be rolled back. • We need to submit a new binary to the store • App needs more exhaustive tests and QA processes
17. Shipit Mobile: Overview
18. A platform to create, view, and manage app releases
19. Easy to consult current and past releases
20. Easy to create a hotfix release Hard to create a release if CI fails
21. Easy to see who’s responsible for the release Easy to distribute to different channels Prevents the app from uploading when tests fail
23. What have we been optimizing for? • master CI is always green • Release with a single click of a button • Fast QA
24. CI Stability
25. Automation requires a stable CI.
26. A stable CI requires a resilient infrastructure.
27. Sources of flakiness • Unreliable networks • Flaky tests • Side effects from previous runs • Bugs in the tools
28. Build stability and flakiness • Reproducible build environments (Anka, Docker) • Vendor dependencies in repository • Auto-retry network failures, flaky tests
29. Auto-retries and master pass rates From our blog post The Unreasonable Effectiveness of Test Retries: An Android Monorepo Case Study
30. 92% iOS 87% Android Pass Rates
31. Shipit Mobile: Technical Decisions
37. platform: ios build_pipeline: ping-ios-release-build version_file_path: Ping/xcconfigs/Version.xcconfig channels: appstore: bundle_id: com.shopify.ping upload_metadata: true ci: require: - ping-ios slack: channels: - ping
38. platform: ios build_pipeline: ping-ios-release-build version_file_path: Ping/xcconfigs/Version.xcconfig channels: appstore: bundle_id: com.shopify.ping upload_metadata: true ci: require: - ping-ios slack: channels: - ping
39. Notify teams about the status of a release
40. Speeding up QA! Check out our blog post Mobile Tophatting at Shopify
42. Steps for mobile app testing 1. Clone repository 2. Switch to branch 3. Compile the app 4. Boot app in the device
44. Developer hours saved
45. • Working on infra doesn’t mean you don’t have customers • Understand your users’ problems • Productize your tools • Release early and validate your assumptions
46. Thanks! alejandro.salamanca@shopify.com @alexrs95
47. Questions?
48. Thanks! alejandro.salamanca@shopify.com @alexrs95
49. Bonus: Build Infrastructure
50. Quick overview of infrastructure iOS • 100 Mac minis in MacStadium across two regions • Hosts provisioned with Ansible • Each build runs in an isolated Anka virtual machine Android • Google Kubernetes Engine (GKE) • Google Firebase Test Lab for virtual device testing
51. Anka: disposable macOS environments • Native macOS hypervisor built on top of FreeBSD bhyve and xhyve and macOS Hypervisor.Framework • Anka Registry for distributing VM images • Docker-like interface • Instant boot time Check out our blog post Scaling iOS CI with Anka
52. anka create --ram-size 8G --cpu-count 4 --disk-size 80G --app installer.app my-vm
53. • anka clone [OPTIONS] VM_ID NEW_VM_NAME • anka run [OPTIONS] VM_NAME COMMAND [ARGS]... • anka registry pull VM_ID