Integrating and Releasing code at Tuenti: an automated pipeline

Integrar, probar y distribuir el código en el que trabajan unos 30 desarrolladores supone un reto. Diseñando procesos flexibles, pero cuyo cumplimiento sea estricto, y automatizándolos, en Tuenti hemos aplicado al desarrollo de aplicaciones móviles las lecciones que nos permitieron incrementar el número de releases de código de nuestra web de dos a la semana a 2 al día, resultando en un proceso transparente al desarrollador, y con todas las características que se exigen de un proceso de release: seguridad, trazabilidad, repetibilidad, automatización y eficiente flujo de información de arriba a abajo y de abajo arriba.

Transcript

1. Integrating and Releasing code at Tuenti: an automated pipeline @oscarsj
2. Problem • Lots of developers writing code at the same time • Buggy code gets to main development branch • A lot of time wasted checking test results, performing builds, merging branches, publishing versions… • Information missing, lots of meetings required
3. Solution Create a process that is: • Well-defined • Repeteable • Trackable • Automated (as far as possible) How?
4. Agenda 1. Overview a. Dev/Release system architecture b. Flow 2. Process a. Development b. Integration c. Distribution d. Release e. Publishing
5. Dev/Release system architecture
6. Dev/Release system architecture
7. Dev/Release system architecture
8. Dev/Release system architecture
9. Dev/Release system architecture
10. Flow
11. Dev/Release system architecture
12. Mercurial (SCM) • Mercurial/Git: distributed scm ◦ every repo is a full copy, with history ◦ Developers work on their copy, and frequently synchronize with central repo (push/pull) • Protected branches: ◦ integration ◦ release-* ◦ hotfix-* ◦ live
13. Mercurial (SCM)
14. Mercurial (SCM) • Sanity check “hooks” ◦ Syntax checks ◦ No conflict markers ◦ All commits with ticket identifier ◦ Data formats validation • Tags: ◦ All versions published are uniquely identified by version ◦ This allow to track back errors to the code
15. Dev/Release system architecture
16. Jenkins mvn
17. Dev/Release system architecture
18. Hockeyapps • Development versions distribution system ◦ http://hockeyapp.net/ • Stores versioned apks, along with symbol tables • Provides an API to include in the app: ◦ Send crash reports to hockeyapps ◦ Notifies and installs new versions from server
19. Hockeyapps • Projects: ◦ alphas: loose versioning, teams manage it themselves ◦ preproduction: strict, production versioning. Only versions built from integration/release pipeline ◦ production: no distribution, only to store crashes
20. Hockeyapps • Development versions generated locally, by devs • Preproduction versions generated from integration and release branches ◦ Tagged, and version increased after successful publish • Versioning in integration and release branch ◦ Major.minor.build
21. Dev/Release system architecture
22. Jira • Issue tracker ◦ Rest API ◦ Hooks -> events • All info of project stored here ◦ Human generated: Content of branches, risks, effects on customers, etc ◦ Auto generated: Results of builds, times of deployment, etc • Process triggered from here ◦ Integrate branch ◦ Create releases ◦ Publish versions
23. Dev/Release system architecture
24. Release Manager • Controls processes • Checks tests status • Checks integrity of builds and branches • Checks for abnormal situations • Broadcast info for stakeholders • Operates scripts to build and deploy to different environments
25. Flow
26. Flow
27. Process • From development (my laptop) to production • As fast as possible: all code finished but not published gets old • Quality is integral part of the process: ◦ devs: automated testing ◦ product: ensure quality as a feature ◦ leads: provide means with scheduling and formation • Makes sure code in integration branch is always “good” (passes tests)
28. Process: Development 1. Developer branches from integration 2. Creates a INT (integration request) ticket 3. Develops, tests 4. Peer code reviews 5. Pull from integration branch 6. Translations 7. Ready (reviewed+tests passing+feature accepted): pull request 8. Waits for feedback
29. Process: Integration • All info to merge a branch in jira ticket: ◦ Changeset ◦ Author/Reviewer ◦ Contents ◦ Risks... • Create temporal branch with integration + feature branch • Test that temporal branch • Blue ball: integrated, otherwise: rejected • Feedback: ◦ Jira ◦ Mail
30. Process: Distribution • Integration latest changeset gets build in “production” mode (no debug, ofuscated, prod. key) • If build succeeds: ◦ Pushed to hockeyapps ◦ Version (build number) increased ◦ Tag in mercurial with version number ◦ Feedback via mail, with information, changelog and link to donwload • Periodically, at least once a week ◦ Too often may be counterproductive
31. Process: Release • Integration latest changeset get branched • Latest translations included • Latest hotfixes included • Product version set (major.minor) if needed • Build/distribution ◦ all company encouraged to install • Stabilization: ◦ 3 days ◦ devs must check and fix crashes related to release ◦ at the end of each day a new build is distributed
32. Process: Release • Release stable: agreement between developers and product owners • Google play submit ◦ progressive roll-out: ▪ 8 hours to 5% ▪ increased to 100% during next 8 hours • Release branch is merged to live branch ◦ hotfixes branched from here • Release branch is merged to integration ◦ via pull request
33. Process: Publishing • multiple architectures • Progressive roll-out ◦ 5% for 24 hours ◦ Progressive increase during another 8 hours • Manual process :(
34. QUESTIONS? @oscarsj