Mobile release trains

- What is a release train
- What problems does it solve
- How to implement it for your mobile team

Transcript

1. Mobile release trains Luis González Valle @lgvalle
2. What is Travelperk? ⭐ Make business travelers happy ⭐ Help companies save money World’s largest inventory Flights, hotels, trains, accommodation All in one place, no more chasing receipts Search, book, pay, invoice, share and report — all on the same platform. Save money Cheaper options, low commissions, more data to optimise, Expensify integration
3. Mobile Release trains at Travelperk Days 1-7 Development phase Day 8 - 15.00 Code freeze QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint
4. Why? Squads at Travelperk 💛 With mobile capabilities in 2020
5. Why? Squads at Travelperk 💛 With mobile capabilities in 2021
6. Why? Squads at Travelperk 💛 With mobile capabilities in 2022?
7. Work process for mobile engineering • One codebase per platform • Branches ◦ /release/x.x.x latest released code ◦ /master should always be releasable ◦ /feature/<feature-id> work branches • Small frequent PRs • Code reviews, pair programming • Feature flags
8. Old release process Feature-based release ✅ Works great for one squad team ❌ Hold a release until desired feature is completed (no bug fixing) ❌ Multi-squad problems: ◦ Separate features block each other ◦ Delays and uncertainty ◦ Who owns QA for the single binary? is the feature done? Work on a feature Ship new app version Yes No
9. What is a release?
10. “A release is the process of making a version of your app available to users” - Abraham Lincoln
11. Stages of release processes Main branch where developers merge changes Merged changes are automatically tested There’s a CI server running those tests Continuous Integration Changes merged into the main branch are automatically deployed to a production-like environment Release to production is manual Continuous Delivery Changes merged into the main branch are automatically deployed to production No human intervention Continuous Deployment Source: https://www.atlassian.com/continuous-delivery/principles/continuous-integration-vs-delivery-vs-deployment
12. Stages of release processes (for mobile) Main branch where developers merge changes Merged changes are automatically tested There’s a CI server running those tests Continuous Integration Changes merged into the main branch are automatically deployed to a production-like environment Release to production is manual Continuous Delivery Changes merged into the main branch are automatically deployed to production No human intervention Continuous Deployment Source: https://www.atlassian.com/continuous-delivery/principles/continuous-integration-vs-delivery-vs-deployment
13. Why mobile releases are different from web/backend Distribution environment Updates are not instant: review process can take days and even reject the update Updates are not mandatory: users may never update to a new release Release as a single binary Apps are distributed as one single binary: we can’t update only parts of it To remove feature’s code you need to ship a new version of the whole project Rollback App stores do not allow rollbacks Any new bug means • New release for the whole app needs to be done (single binary) • It won’t be fixed today (we don’t control distribution environments)
14. Release trains
15. Fixed schedule Metrics Release train main features Release captain Feature flags QA
16. Days 1-7 Development phase Day 8 - 15.00 Code freeze QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Mobile release train
17. Notion page with things to do before starting a release train Mobile release train: Checklist [ ] Assign release captains for Android and iOS [ ] Check which feature flags will need enabling [ ] Force upgrade after the release? [ ] Test cases for new features are added to Test Checklist [ ] Check all translations are in Phrase [ ] Release notes created and translated [ ] Changelog updated [ ] Release train announced in Slack [ ] Clone Amplitude release dashboard
18. Days 1-7 Development phase Day 8 - 15.00 Code freeze Train “departs” at 15.00 o’clock strictly Release branch cut from master New changes to master not allowed into the train from this point QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Mobile release train: Departure
19. desc "Start a new Release train" lane :start_release do # Code freeze release_version = prompt(text: "Please provide a version number for the new release:") create_branch(from: "master", release_version: release_version) # Generate builds create_staging_build(branch_name: git_branch) create_production_build(branch_name: git_branch) UI.success "🍻 STAGING build and PRODUCTION build generated for current branch" end Mobile release train: Departure
20. Days 1-7 Development phase Day 8 - 15.00 Code freeze Release Captain drives this phase Manual* testing of production build. Core & feature tests shared in spreadsheet Feature flags checked here Only fixes accepted at this stage QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint * Working towards automation in Q1 2021 Mobile release train: QA
21. Mobile release train: QA checklist
22. Days 1-7 Development phase Day 8 - 15.00 Code freeze QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Allowed into the release branch if critical bugs are found during QA Non-critical bug fixes will board the next release train Needs agreement between Release Captain and PM Bug fix Mobile release train: bug fixing
23. Days 1-7 Development phase Day 8 - 15.00 Code freeze Semi-automated process Release notes handled by PM Validation process in stores can take up to 2-3 days QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Mobile release train: store submission
24. Mobile release train: store submission
25. Days 1-7 Development phase Day 8 - 15.00 Code freeze 1% > 2% > 5% > 20% > … > 100% to closely monitor adoption issues Staged rollout can be halted but never rolled back Hotfixes allowed for critical bugs -> 100% QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Hotfix Mobile release train: staged rollout
26. Mobile release train: staged rollout
27. Days 1-7 Development phase Day 8 - 15.00 Code freeze Rollout reaches 100% Release tagged on Github Release branch merged back to master QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Mobile release train: Release
28. desc "Finish release train: Tag version, create PR from release branch to master" lane :finish_release do # Get the release branch release_branch = sh("git branch -r | grep 'release' | tail -1 | tr -d ' ' | cut -d'/' -f2-").chomp # Check out release_branch and pull changes sh("git checkout #{release_branch}") # Get current version current_version = get_version_name # Tag release and push sh("git tag -a v#{current_version} -m '' && git push origin v#{current_version}") # Create Pull request open_pr_release(version: current_version) UI.success "🍻 Success!" end end Mobile release train: Finish a release
29. Tools & automation
30. Start & finish release trains Build release & submit to stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling
31. Start & finish release trains Build release & submit to stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices
32. Start & finish release trains Build release & submit to stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Triggered automatically from Bitrise at specific times ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices
33. Start & finish release trains Build release & submit to stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Triggered automatically from Bitrise at specific times ➔ Start and finish release trains at fixed times ➔ Automatically create new release builds when hotfixes are merged into release branches and submit them ➔ Automatically run SAST/DAST tool nightly ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices
34. Start & finish release trains Build release & submit to stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Triggered automatically from Bitrise at specific times ➔ Start and finish release trains at fixed times ➔ Automatically create new release builds when hotfixes are merged into release branches and submit them ➔ Automatically run SAST/DAST tool nightly ➔ Release dashboard is created automatically ➔ Publish release rollout status reports automatically on Slack ➔ Halt rollout if error rate goes over a threshold ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices
35. Start & finish release trains Build release & submit to stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Triggered automatically from Bitrise at specific times ➔ Start and finish release trains at fixed times ➔ Automatically create new release builds when hotfixes are merged into release branches and submit them ➔ Automatically run SAST/DAST tool nightly ➔ Release dashboard is created automatically ➔ Publish release rollout status reports automatically on Slack ➔ Halt rollout if error rate goes over a threshold ➔ All train status changes are announced automatically ➔ Capacity to query: what’s the rollout status? Error rate? ➔ Capacity to action: halt release, generate stage build,... ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices
36. 3 Execute the process manually first 4 Find automation opportunities 2 Define a process that works for you 1 Identify what are the pain points for your company
37. https://www.applause.com/blog/release-planning-mobile-release-train https://developers.soundcloud.com/blog/quality-mobile-trains https://www.scaledagileframework.com/agile-release-train/ https://medium.com/@SkyscannerEng/the-present-and-future-of-app-release-at-skyscanner-b393c88972a8 https://glovoapp.zoom.us/rec/play/UN2cJPb6wUS5ehpxbPid_FpFBZgbnD3MhCru8jfXtkaQnZPAZsAoYLSK1G O3-swBXkXLEenqNeNxiFjZ.fNXbB5-sP6nTyaRs https://medium.com/trinity-mirror-digital/why-we-started-riding-release-trains-for-our-mobile-applications-4 1f344b3ac50 Further reading
38. We are hiring! https://www.travelperk.com/careers/
39. Thanks! Luis González Valle @lgvalle