Slack's Journey to Faster and More Frequent Mobile Releases

A product team’s success highly dependent on its iteration speed—how quickly it can get changes in front of its users and respond to their feedback.

Web developers are used to shipping new versions of their work to customers multiple times per day. In mobile engineering, the process of getting a new version of the application into the hands of Android and iOS customers is much more often measured in days. Hence, most mobile shops use the concept of a release train to ensure that development can continue while a release is in progress.

How does the release train move? How frequently and when should it depart and reach its final destination? What are the constraints that slow it down and what levers can be pulled to enable it to arrive faster?

This is a story of Slack’s recent experiment with moving to a more frequent release train schedule—a process that involved listening to engineering teams and distilling their hopes and fears into success criteria, instrumenting the release process to gain a full understanding of bottlenecks, optimizing release automation to enable moving faster and, finally, tracking the success of the experiment.

Transcript

1. FASTER. HARDER. STRONGER FREQUENTER. ’S ROAD TO FASTER MOBILE PRODUCT ITERATIONS
3. WE ARE HERE TO HELP WITH SHIPPING FEATURES TO USERS
7. ~1 DAY MIN ITERATION CYCLE ON WEB https://www.independent.co.uk/travel/news-and-advice/japan-s-bullet-train-50-years-of-powering-ahead-9940290.html
8. https://www.bbc.com/travel/article/20230313-the-slowest-train-journey-in-india 40+ DAYS MIN ITERATION ON MOBILE* *Slack Apps in 2022
9. R4 R3 R2 R1 Time (in days) 0 7 14 21 28 35 42 49 56 Release CADENCE: 2 WEEKS DURATION: 11 DAYS
10. R4 R3 R2 R1 Time (in days) 0 7 14 21 28 35 42 49 56 Release Detect CADENCE: 2 WEEKS DURATION: 11 DAYS
11. R4 R3 R2 R1 Time (in days) 0 7 14 21 28 35 42 49 56 Release Detect Fix 46 DAYS CADENCE: 2 WEEKS DURATION: 11 DAYS
13. R4 R3 R2 R1 Time (in days) 0 7 14 21 28 35 42 49 56 Release Detect Fix 46 DAYS CADENCE: 2 WEEKS DURATION: 11 DAYS
14. R5 R4 R3 R2 R1 Time (in days) 0 7 14 21 28 35 42 49 56 Release Detect Fix 39 DAYS 46 DAYS CADENCE: 1 WEEK DURATION: 11 DAYS
15. R5 R4 R3 R2 R1 Time (in days) 0 7 14 21 28 35 42 49 56 Release Detect Fix 28 DAYS 46 DAYS 39 DAYS CADENCE: 1 WEEK DURATION: 7 DAYS
16. RELEASE MORE FREQUENTLY OR FASTER?
17. R5 R4 R3 R2 R1 Time (in days) 0 7 14 21 28 35 42 49 56 Release Detect Fix 46 DAYS 39 DAYS CADENCE: 1 WEEK DURATION: 3 DAYS 28 DAYS 15 DAYS
18. https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance
19. https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance
20. MOBILE IS DIFFERENT
21. MOBILE IS DIFFERENT ▸ App Store review takes days. Sometimes multiple rounds. ▸ No fast rollbacks ▸ Users not guaranteed to install your update right away or ever ▸ Cost of a mistake is high https://www.progressiverailroading.com/amtrak/article/Passenger-rail-Amtrak-employs-new-safety-program-to-identify-improve-workers39-behavior--26585
22. DIVING DEEPER
23. R7 R6 R5 R4 R3 R2 R1 Time (in days) 0 7 14 21 28 35 42 49 56 63 70 77 84 91
24. R1 Time (in days) 0 7 14 Sat Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Fri TESTING RELEASE NOTES STRING TRANSLATION APP STORE REVIEW ROLLOUT Slack Mobile Releases in 2022
25. R1 Time (in days) 0 7 14 Sat Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Fri TESTING VENDOR TESTING TRIAGE OF VENDOR RESULTS DOGFOOD EXTERNAL BETA INTERNAL QA/VALIDATION OF HOTFIXES
26. R1 Time (in days) 0 7 14 Sat Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Fri KEPT SAME VENDOR TEST COVERAGE OPTIMIZED TRIAGE OF VENDOR FINDINGS INCREASED DOGFOOD ADOPTION TO COMPENSATE FOR SHORTER BAKE TIME MOVED INTERNAL QA/VALIDATION OF HOTFIXES TO TEAMS TESTING VENDOR TESTING TRIAGE OF VENDOR RESULTS DOGFOOD EXTERNAL BETA RELEASED EXTERNAL BETA SOONER LOWER TOLERANCE FOR LATE HOTFIXES BUFFER FOR HOTFIXES
27. HOTFIXES
28. R2 R1 Time (in days) 0 7 14 21 28 Release Tasks App Store Review Rollout CADENCE: 2 WEEKS DURATION: 11 DAYS
29. R2 R1 Time (in days) 0 7 14 21 28 Release Tasks App Store Review Hot fi x App Store Review #2 Rollout 4 DAYS CADENCE: 1 WEEK DURATION: 7 DAYS
30. R2 R1 Time (in days) 0 7 14 21 28 Release Tasks App Store Review Hot fi x App Store Review #2 Rollout 2 DAYS CADENCE: 1 WEEK DURATION: 7 DAYS
31. R1 Time (in days) 0 7 14 Sat Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Fri TESTING RELEASE NOTES STRING TRANSLATION ROLLOUT APP STORE REVIEW PART OF APP STORE SUBMISSION. MUST BE COLLECTED AND TRANSLATED BY THEN.
32. R1 Time (in days) 0 7 14 Sat Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Fri TESTING RELEASE NOTES STRING TRANSLATION ROLLOUT APP STORE REVIEW COLLECTION STARTS EARLIER GOES WITH NEXT RELEASE IF DIDN’T MAKE IT IN TIME
33. R1 Time (in days) 0 7 14 Sat Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Fri TESTING RELEASE NOTES STRING TRANSLATION APP STORE REVIEW ROLLOUT
34. R1 Time (in days) 0 7 14 Sat Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Fri TESTING RELEASE NOTES APP STORE REVIEW ROLLOUT AUTOMATION MERGED STRING TRANSLATION ONLY INTO MAIN BRANCH STRING FREEZE 2 DAYS PRIOR TO CODE FREEZE
35. R1 Time (in days) 0 7 14 Sat Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Fri TESTING RELEASE NOTES STRING TRANSLATION APP STORE REVIEW ROLLOUT NO STRING FREEZE. YAY!
36. R1 Time (in days) 0 7 14 Sat Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Fri TESTING RELEASE NOTES STRING TRANSLATION APP STORE REVIEW ROLLOUT 1-2 DAYS! ~4 HOURS NICE TO USE WEEKEND VERY THOROUGH (5X REJECTION RATE OF ANDROID) LESS STRICT NO MORE THAN 1 SUBMISSION/ROLLOUT CONCURRENTLY
37. R1 Time (in days) 0 7 14 Sat Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Fri TESTING RELEASE NOTES STRING TRANSLATION APP STORE REVIEW ROLLOUT HAD TO MOVE TO WEEKDAY
38. R1 Time (in days) 0 7 14 Sat Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Fri TESTING RELEASE NOTES STRING TRANSLATION APP STORE REVIEW ROLLOUT OBSERVE CRASH RATES AND ERROR REPORTS 1% -> 50% ->100% CAN BE ACCELERATED TO 1 DAY, BUT PREFER 2 TO REDUCE RISK TAKES ANOTHER FEW DAYS FOR MAJORITY OF USERS TO INSTALL UPDATE
39. R5 R4 R3 R2 R1 Time (in days) 0 7 14 21 28 35 42 49 56 Release Detect Fix 46 DAYS 39 DAYS CADENCE: 1 WEEK DURATION: 3 DAYS 28 DAYS 15 DAYS
40. R5 R4 R3 R2 R1 Time (in days) 0 7 14 21 28 35 42 49 56 Release Detect Fix 28 DAYS 46 DAYS 39 DAYS CADENCE: 1 WEEK DURATION: ~7 DAYS
41. THE PROCESS OF CHANGE
42. THE PROCESS OF CHANGE TALK TO FOLKS SURVEY COMMUNICATE, COMMUNICATE, COMMUNICATE INSTRUMENT THE PROCESS MAKE THE CHANGE GATHER METRICS SURVEY LET IT BAKE ROLLBACK EVALUATE CELEBRATE MITIGATE RISK
43. ▸ Hopes ▸ Faster iterations ▸ More clarity on schedule ▸ Reduced pressure to catch the train ▸ Better stability ▸ Success Criteria ▸ Increased velocity ▸ Increased clarity ▸ Reduced pressure ▸ No regression in stability ▸ Not just developers THE PROCESS OF CHANGE ▸ Fears ▸ Lower stability with reduced test time ▸ Increased pressure to catch the train TALK TO FOLKS
44. THE PROCESS OF CHANGE TALK TO FOLKS SURVEY COMMUNICATE, COMMUNICATE, COMMUNICATE INSTRUMENT THE PROCESS MAKE THE CHANGE GATHER METRICS SURVEY LET IT BAKE ROLLBACK EVALUATE CELEBRATE MITIGATE RISK
45. THE PROCESS OF CHANGE ▸ Based on success criteria from interviews ▸ Baseline for human perception ▸ Chance to ▸ gather more info ▸ let everyone be heard SURVEY
46. THE PROCESS OF CHANGE SURVEY Pressure Schedule Clarity Velocity Safety
47. THE PROCESS OF CHANGE SURVEY Pressure Schedule Clarity Velocity Safety
48. “WE TRY AND SQUEEZE UPDATES AND IMPROVEMENTS IN THE UPCOMING RELEASE, BECAUSE THE NEXT ONE IS TOO FAR. BY THE TIME THE CUSTOMER SEES OUR WORK - IT'S 1 MONTH OUT. IT SLOWS DOWN PACE OF SHIPPING, EXPERIMENTATION AND LEARNING ON MOBILE PLATFORMS. DESKTOP IS ABLE TO SHIP TO GA IN A DAY (WHICH WOULD BE THE DREAM). BUT, I CAN ONLY HOPE WE CAN MAKE IT QUICKER THAN WHAT IT IS TODAY.” Designer at Slack ON PRESSURE TO CATCH THE RELEASE TRAIN AND SPEED SURVEY
49. THE PROCESS OF CHANGE SURVEY Pressure Schedule Clarity Velocity Safety
50. ON CLARITY AROUND SCHEDULE SURVEY “I HAVE NO IDEA — WHAT ARE EXTERNAL BETA USERS?” “THERE ARE SEVERAL CUT OFFS THAT I HAVE TO CONSISTENTLY ASK AGAIN OR SECOND GUESS JUST IN CASE I CONFUSE MYSELF OR CONFUSE OTHERS.” A DEVELOPER A PRODUCT MANAGER
51. THE PROCESS OF CHANGE SURVEY Pressure Schedule Clarity Velocity Safety
52. ON RELEASE SPEED SURVEY “OTHER PLACES I'VE BEEN THE WAIT HAS BEEN EVEN LONGER. I FIND SLACK'S CURRENT TIMING FINE.” A DEVELOPER “2 WEEKS IS A LONG TIME IN THE MODERN WORLD OF SOFTWARE” A DESIGNER
53. THE PROCESS OF CHANGE SURVEY Pressure Schedule Clarity Velocity Safety
54. ON RELEASING WITHOUT MANUAL QA SURVEY “WE NEED TO IMPROVE (A LOT) ON OUR AUTOMATE TESTING BEFORE I CAN COMFORTABLE WITH THIS.” A DEVELOPER “JUST BECAUSE THE BUILD IS "GREEN", IT DOESN'T MEAN THAT THERE ARE NO CRITICAL ISSUES. WE SHOULD ALWAYS HAVE MANUAL REGRESSION PASS BEFORE RELEASING TO EXTERNAL USERS” AN ENGINEERING MANAGER
55. THE PROCESS OF CHANGE TALK TO FOLKS SURVEY COMMUNICATE, COMMUNICATE, COMMUNICATE INSTRUMENT THE PROCESS MAKE THE CHANGE GATHER METRICS SURVEY LET IT BAKE ROLLBACK EVALUATE CELEBRATE MITIGATE RISK
56. THE PROCESS OF CHANGE INSTRUMENT THE PROCESS GITHUB APP/PLAY STORE TRACING CI HUMAN INPUT
57. THE PROCESS OF CHANGE INSTRUMENT THE PROCESS HUMAN INPUT
58. THE PROCESS OF CHANGE INSTRUMENT THE PROCESS SWITCH TO WEEKLY CADENCE FASTER RELEASE TRAINS
59. THE PROCESS OF CHANGE INSTRUMENT THE PROCESS SWITCH TO WEEKLY CADENCE FASTER RELEASE TRAINS
60. THE PROCESS OF CHANGE INSTRUMENT THE PROCESS SWITCH TO WEEKLY CADENCE FASTER RELEASE TRAINS
61. THE PROCESS OF CHANGE TALK TO FOLKS SURVEY COMMUNICATE, COMMUNICATE, COMMUNICATE INSTRUMENT THE PROCESS MAKE THE CHANGE GATHER METRICS SURVEY LET IT BAKE ROLLBACK EVALUATE CELEBRATE MITIGATE RISK
62. THE PROCESS OF CHANGE ▸ Make the change incrementally ▸ Stage 1: Increase release frequency ▸ Stage 2: Increase release speed MITIGATE RISK https://www.pocketmindfulness.com/baby-steps-approach/
63. THE PROCESS OF CHANGE ▸ Practice with a dry-run ▸ Go through the process, but skip rollout MITIGATE RISK https://www.vipav.com/blog/why-you-shouldn-t-skip-rehearsals
64. THE PROCESS OF CHANGE Pressure Schedule Clarity Velocity Safety
65. THE PROCESS OF CHANGE ▸ Feature fl ags ▸ Not a silver bullet ▸ Measure % or change behind fl ag? MITIGATE RISK
66. THE PROCESS OF CHANGE ▸ Dogfood (internal testing) ▸ Reduced time to catch errors ▸ Compensated by increase in usage MITIGATE RISK
67. THE PROCESS OF CHANGE MITIGATE RISK LIVE WITH RISK
68. THE PROCESS OF CHANGE TALK TO FOLKS SURVEY COMMUNICATE, COMMUNICATE, COMMUNICATE INSTRUMENT THE PROCESS MAKE THE CHANGE GATHER METRICS SURVEY LET IT BAKE ROLLBACK EVALUATE CELEBRATE MITIGATE RISK
69. THE PROCESS OF CHANGE ▸ Repetition ▸ Repetition ▸ Keep it Real ▸ Educate ▸ Emphasis: Experimentation and Success Criteria COMMUNICATE, COMMUNICATE, COMMUNICATE
70. THE PROCESS OF CHANGE TALK TO FOLKS SURVEY COMMUNICATE, COMMUNICATE, COMMUNICATE INSTRUMENT THE PROCESS MAKE THE CHANGE GATHER METRICS SURVEY LET IT BAKE ROLLBACK EVALUATE CELEBRATE MITIGATE RISK
71. R3 R2 R1 Time (in days) 0 7 14 21 28 35 Release Tasks App Store Review Rollout MAKE THE CHANGE
72. R3 R2 R1 Time (in days) 0 7 14 21 28 35 Release Tasks App Store Review Rollout MAKE THE CHANGE
73. R3 R2 R1 Time (in days) 0 7 14 21 28 35 Release Tasks App Store Review Rollout 💥 MAKE THE CHANGE
74. R3 R2 R1 Time (in days) 0 7 14 21 28 35 Release Tasks App Store Review Rollout MAKE THE CHANGE
75. THE PROCESS OF CHANGE TALK TO FOLKS SURVEY COMMUNICATE, COMMUNICATE, COMMUNICATE INSTRUMENT THE PROCESS MAKE THE CHANGE GATHER METRICS SURVEY LET IT BAKE ROLLBACK EVALUATE CELEBRATE MITIGATE RISK
76. THE PROCESS OF CHANGE GATHER METRICS LET IT BAKE
77. THE PROCESS OF CHANGE TALK TO FOLKS SURVEY COMMUNICATE, COMMUNICATE, COMMUNICATE INSTRUMENT THE PROCESS MAKE THE CHANGE GATHER METRICS SURVEY LET IT BAKE ROLLBACK EVALUATE CELEBRATE MITIGATE RISK
78. THE PROCESS OF CHANGE SURVEY AFTER SWITCH TO WEEKLY RELEASES THIS USED TO BE A STRESSFUL REALITY, BUT THANKS TO OUR WEEKLY RELEASE SCHEDULE (AND THE MOVING OF STRING A DEVELOPER A WEEK AND A HALF FEELS LIKE A LONG TIME. A DESIGNER WITHOUT PROJECT GROUNDHOG AND WEEKLY RELEASES, I FELT THE RELEASES WERE TOO INFREQUENT AND WE WOULD WAIT OR RESORT TO HOT FIXES MORE OFTEN. A DEVELOPER
79. THE PROCESS OF CHANGE SURVEY AFTER SWITCH TO FASTER TRAIN I WOULD SAY THE PRESSURE IS LESS THAN WHAT IT WAS WITH LONGER TRAIN A DEVELOPER THE 5-DAY SCHEDULE WORKS GREAT WHEN EVERYTHING GOES SMOOTHLY. IN PRACTICE, THERE ARE SEVERAL PAIN POINTS THAT MAKE THE RELEASE PROCESS HIGHER-RISK, AND LESS- FLEXIBLE… THE RELEASE MANAGER AS LONG AS WE CONTINUE TO FEEL CONFIDENT ABOUT OUR ABILITY TO TEST THE APP BEFORE IT GOES OUT TO EXTERNAL CUSTOMERS, THIS PACE IS FANTASTIC! A DEVELOPER
80. THE PROCESS OF CHANGE TALK TO FOLKS SURVEY COMMUNICATE, COMMUNICATE, COMMUNICATE INSTRUMENT THE PROCESS MAKE THE CHANGE GATHER METRICS SURVEY LET IT BAKE ROLLBACK EVALUATE CELEBRATE MITIGATE RISK
81. THE PROCESS OF CHANGE EVALUATE ▸ Success Criteria ▸ Increased velocity ▸ Increased clarity ▸ Reduced pressure ▸ No regression in stability
82. THE PROCESS OF CHANGE EVALUATE ▸ Success Criteria ▸ Increased velocity ▸ Increased clarity ▸ Reduced pressure ▸ No regression in stability
83. THE PROCESS OF CHANGE EVALUATE ▸ Success Criteria ▸ Increased velocity ▸ Increased clarity ▸ Reduced pressure ▸ No regression in stability
84. THE PROCESS OF CHANGE EVALUATE ▸ Success Criteria ▸ Increased velocity ▸ Increased clarity ▸ Reduced pressure ▸ No regression in stability
85. THE PROCESS OF CHANGE EVALUATE ▸ Success Criteria ▸ Increased velocity ▸ Increased clarity ▸ Reduced pressure ▸ No regression in stability
86. EVALUATE
87. THE PROCESS OF CHANGE TALK TO FOLKS SURVEY COMMUNICATE, COMMUNICATE, COMMUNICATE INSTRUMENT THE PROCESS MAKE THE CHANGE GATHER METRICS SURVEY LET IT BAKE ROLLBACK EVALUATE MITIGATE RISK CELEBRATE
88. THE PROCESS OF CHANGE TALK TO FOLKS SURVEY COMMUNICATE, COMMUNICATE, COMMUNICATE INSTRUMENT THE PROCESS MAKE THE CHANGE GATHER METRICS SURVEY LET IT BAKE EVALUATE MITIGATE RISK CELEBRATE CELEBRATE
89. FUTURE POSSIBILITIES
91. THE FUTURE OPPORTUNITIES ▸ Iterate with “closer” audiences ▸ Early prototypes to key stakeholders ▸ Internal users ▸ External beta users ▸ Shift Left ▸ Reduce risk of hot fi xes ▸ Get to releasing a green build from main ▸ AI ▸ Optimize time to detect and fi x
92. FINAL THOUGHTS
93. THE END
94. FINAL THOUGHTS ▸ Change triggers fear ▸ “Fear is the mind killer” ▸ It’s easy to get into a comfort zone ▸ With the right approach we can reach a new level of productivity Which, if you have read book 4 of the dune series, is something that we probably don’t want to do. But that philosophical talk for next year. If AI hasn’t become sentient and taken over by then. See you all in the future.