Mobile release trains Luis González Valle @lgvalle

What is Travelperk? ⭐ Make business travelers happy ⭐ Help companies save money World’s largest inventory Flights, hotels, trains, accommodation All in one place, no more chasing receipts Search, book, pay, invoice, share and report — all on the same platform. Save money Cheaper options, low commissions, more data to optimise, Expensify integration

Mobile Release trains at Travelperk Days 1-7 Development phase Day 8 - 15.00 Code freeze QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint

Why? Squads at Travelperk 💛 With mobile capabilities in 2020

Why? Squads at Travelperk 💛 With mobile capabilities in 2021

Why? Squads at Travelperk 💛 With mobile capabilities in 2022?

Work process for mobile engineering ● One codebase per platform ● Branches ○ /release/x.x.x latest released code ○ /master should always be releasable ○ /feature/ work branches ● Small frequent PRs ● Code reviews, pair programming ● Feature flags

Old release process Feature-based release ✅ Works great for one squad team ❌ Hold a release until desired feature is completed (no bug fixing) ❌ Multi-squad problems: ○ Separate features block each other ○ Delays and uncertainty ○ Who owns QA for the single binary? is the feature done? Work on a feature Ship new app version Yes No

What is a release?

“A release is the process of making a version of your app available to users” - Abraham Lincoln

Stages of release processes Main branch where developers merge changes Merged changes are automatically tested There’s a CI server running those tests Continuous Integration Changes merged into the main branch are automatically deployed to a production-like environment Release to production is manual Continuous Delivery Changes merged into the main branch are automatically deployed to production No human intervention Continuous Deployment Source: https://www.atlassian.com/continuous-delivery/principles/continuous-integration-vs-delivery-vs-deployment

Stages of release processes (for mobile) Main branch where developers merge changes Merged changes are automatically tested There’s a CI server running those tests Continuous Integration Changes merged into the main branch are automatically deployed to a production-like environment Release to production is manual Continuous Delivery Changes merged into the main branch are automatically deployed to production No human intervention Continuous Deployment Source: https://www.atlassian.com/continuous-delivery/principles/continuous-integration-vs-delivery-vs-deployment

Why mobile releases are different from web/backend Distribution environment Updates are not instant: review process can take days and even reject the update Updates are not mandatory: users may never update to a new release Release as a single binary Apps are distributed as one single binary: we can’t update only parts of it To remove feature’s code you need to ship a new version of the whole project Rollback App stores do not allow rollbacks Any new bug means ● New release for the whole app needs to be done (single binary) ● It won’t be fixed today (we don’t control distribution environments)

Release trains

Fixed schedule Metrics Release train main features Release captain Feature flags QA

Days 1-7 Development phase Day 8 - 15.00 Code freeze QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Mobile release train

Notion page with things to do before starting a release train Mobile release train: Checklist [ ] Assign release captains for Android and iOS [ ] Check which feature flags will need enabling [ ] Force upgrade after the release? [ ] Test cases for new features are added to Test Checklist [ ] Check all translations are in Phrase [ ] Release notes created and translated [ ] Changelog updated [ ] Release train announced in Slack [ ] Clone Amplitude release dashboard

Days 1-7 Development phase Day 8 - 15.00 Code freeze Train “departs” at 15.00 o’clock strictly Release branch cut from master New changes to master not allowed into the train from this point QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Mobile release train: Departure

desc "Start a new Release train" lane :start_release do # Code freeze release_version = prompt(text: "Please provide a version number for the new release:") create_branch(from: "master", release_version: release_version) # Generate builds create_staging_build(branch_name: git_branch) create_production_build(branch_name: git_branch) UI.success "🍻 STAGING build and PRODUCTION build generated for current branch" end Mobile release train: Departure

Days 1-7 Development phase Day 8 - 15.00 Code freeze Release Captain drives this phase Manual* testing of production build. Core & feature tests shared in spreadsheet Feature flags checked here Only fixes accepted at this stage QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint * Working towards automation in Q1 2021 Mobile release train: QA

Mobile release train: QA checklist

Days 1-7 Development phase Day 8 - 15.00 Code freeze QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Allowed into the release branch if critical bugs are found during QA Non-critical bug fixes will board the next release train Needs agreement between Release Captain and PM Bug fix Mobile release train: bug fixing

Days 1-7 Development phase Day 8 - 15.00 Code freeze Semi-automated process Release notes handled by PM Validation process in stores can take up to 2-3 days QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Mobile release train: store submission

Mobile release train: store submission

Days 1-7 Development phase Day 8 - 15.00 Code freeze 1% > 2% > 5% > 20% > … > 100% to closely monitor adoption issues Staged rollout can be halted but never rolled back Hotfixes allowed for critical bugs -> 100% QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Hotfix Mobile release train: staged rollout

Mobile release train: staged rollout

Days 1-7 Development phase Day 8 - 15.00 Code freeze Rollout reaches 100% Release tagged on Github Release branch merged back to master QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Mobile release train: Release

desc "Finish release train: Tag version, create PR from release branch to master" lane :finish_release do # Get the release branch release_branch = sh("git branch -r | grep 'release' | tail -1 | tr -d ' ' | cut -d'/' -f2-").chomp # Check out release_branch and pull changes sh("git checkout #{release_branch}") # Get current version current_version = get_version_name # Tag release and push sh("git tag -a v#{current_version} -m '' && git push origin v#{current_version}") # Create Pull request open_pr_release(version: current_version) UI.success "🍻 Success!" end end Mobile release train: Finish a release

Tools & automation

Start & finish release trains Build release & submit to stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling

Start & finish release trains Build release & submit to stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices

Start & finish release trains Build release & submit to stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Triggered automatically from Bitrise at specific times ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices

Start & finish release trains Build release & submit to stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Triggered automatically from Bitrise at specific times ➔ Start and finish release trains at fixed times ➔ Automatically create new release builds when hotfixes are merged into release branches and submit them ➔ Automatically run SAST/DAST tool nightly ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices

Start & finish release trains Build release & submit to stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Triggered automatically from Bitrise at specific times ➔ Start and finish release trains at fixed times ➔ Automatically create new release builds when hotfixes are merged into release branches and submit them ➔ Automatically run SAST/DAST tool nightly ➔ Release dashboard is created automatically ➔ Publish release rollout status reports automatically on Slack ➔ Halt rollout if error rate goes over a threshold ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices

Start & finish release trains Build release & submit to stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Triggered automatically from Bitrise at specific times ➔ Start and finish release trains at fixed times ➔ Automatically create new release builds when hotfixes are merged into release branches and submit them ➔ Automatically run SAST/DAST tool nightly ➔ Release dashboard is created automatically ➔ Publish release rollout status reports automatically on Slack ➔ Halt rollout if error rate goes over a threshold ➔ All train status changes are announced automatically ➔ Capacity to query: what’s the rollout status? Error rate? ➔ Capacity to action: halt release, generate stage build,... ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices

3 Execute the process manually first 4 Find automation opportunities 2 Define a process that works for you 1 Identify what are the pain points for your company

https://www.applause.com/blog/release-planning-mobile-release-train https://developers.soundcloud.com/blog/quality-mobile-trains https://www.scaledagileframework.com/agile-release-train/ https://medium.com/@SkyscannerEng/the-present-and-future-of-app-release-at-skyscanner-b393c88972a8 https://glovoapp.zoom.us/rec/play/UN2cJPb6wUS5ehpxbPid_FpFBZgbnD3MhCru8jfXtkaQnZPAZsAoYLSK1G O3-swBXkXLEenqNeNxiFjZ.fNXbB5-sP6nTyaRs https://medium.com/trinity-mirror-digital/why-we-started-riding-release-trains-for-our-mobile-applications-4 1f344b3ac50 Further reading

We are hiring! https://www.travelperk.com/careers/

Thanks! Luis González Valle @lgvalle