Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Integrating and Releasing code at Tuenti: an automated pipeline

March 08, 2014

Integrating and Releasing code at Tuenti: an automated pipeline

Integrar, probar y distribuir el código en el que trabajan unos 30 desarrolladores supone un reto. Diseñando procesos flexibles, pero cuyo cumplimiento sea estricto, y automatizándolos, en Tuenti hemos aplicado al desarrollo de aplicaciones móviles las lecciones que nos permitieron incrementar el número de releases de código de nuestra web de dos a la semana a 2 al día, resultando en un proceso transparente al desarrollador, y con todas las características que se exigen de un proceso de release: seguridad, trazabilidad, repetibilidad, automatización y eficiente flujo de información de arriba a abajo y de abajo arriba.


March 08, 2014

More Decks by Tuenti

Other Decks in Technology


  1. Problem • Lots of developers writing code at the same

    time • Buggy code gets to main development branch • A lot of time wasted checking test results, performing builds, merging branches, publishing versions… • Information missing, lots of meetings required
  2. Solution Create a process that is: • Well-defined • Repeteable

    • Trackable • Automated (as far as possible) How?
  3. Agenda 1. Overview a. Dev/Release system architecture b. Flow 2.

    Process a. Development b. Integration c. Distribution d. Release e. Publishing
  4. Mercurial (SCM) • Mercurial/Git: distributed scm ◦ every repo is

    a full copy, with history ◦ Developers work on their copy, and frequently synchronize with central repo (push/pull) • Protected branches: ◦ integration ◦ release-* ◦ hotfix-* ◦ live
  5. Mercurial (SCM) • Sanity check “hooks” ◦ Syntax checks ◦

    No conflict markers ◦ All commits with ticket identifier ◦ Data formats validation • Tags: ◦ All versions published are uniquely identified by version ◦ This allow to track back errors to the code
  6. Hockeyapps • Development versions distribution system ◦ http://hockeyapp.net/ • Stores

    versioned apks, along with symbol tables • Provides an API to include in the app: ◦ Send crash reports to hockeyapps ◦ Notifies and installs new versions from server
  7. Hockeyapps • Projects: ◦ alphas: loose versioning, teams manage it

    themselves ◦ preproduction: strict, production versioning. Only versions built from integration/release pipeline ◦ production: no distribution, only to store crashes
  8. Hockeyapps • Development versions generated locally, by devs • Preproduction

    versions generated from integration and release branches ◦ Tagged, and version increased after successful publish • Versioning in integration and release branch ◦ Major.minor.build
  9. Jira • Issue tracker ◦ Rest API ◦ Hooks ->

    events • All info of project stored here ◦ Human generated: Content of branches, risks, effects on customers, etc ◦ Auto generated: Results of builds, times of deployment, etc • Process triggered from here ◦ Integrate branch ◦ Create releases ◦ Publish versions
  10. Release Manager • Controls processes • Checks tests status •

    Checks integrity of builds and branches • Checks for abnormal situations • Broadcast info for stakeholders • Operates scripts to build and deploy to different environments
  11. Process • From development (my laptop) to production • As

    fast as possible: all code finished but not published gets old • Quality is integral part of the process: ◦ devs: automated testing ◦ product: ensure quality as a feature ◦ leads: provide means with scheduling and formation • Makes sure code in integration branch is always “good” (passes tests)
  12. Process: Development 1. Developer branches from integration 2. Creates a

    INT (integration request) ticket 3. Develops, tests 4. Peer code reviews 5. Pull from integration branch 6. Translations 7. Ready (reviewed+tests passing+feature accepted): pull request 8. Waits for feedback
  13. Process: Integration • All info to merge a branch in

    jira ticket: ◦ Changeset ◦ Author/Reviewer ◦ Contents ◦ Risks... • Create temporal branch with integration + feature branch • Test that temporal branch • Blue ball: integrated, otherwise: rejected • Feedback: ◦ Jira ◦ Mail
  14. Process: Distribution • Integration latest changeset gets build in “production”

    mode (no debug, ofuscated, prod. key) • If build succeeds: ◦ Pushed to hockeyapps ◦ Version (build number) increased ◦ Tag in mercurial with version number ◦ Feedback via mail, with information, changelog and link to donwload • Periodically, at least once a week ◦ Too often may be counterproductive
  15. Process: Release • Integration latest changeset get branched • Latest

    translations included • Latest hotfixes included • Product version set (major.minor) if needed • Build/distribution ◦ all company encouraged to install • Stabilization: ◦ 3 days ◦ devs must check and fix crashes related to release ◦ at the end of each day a new build is distributed
  16. Process: Release • Release stable: agreement between developers and product

    owners • Google play submit ◦ progressive roll-out: ▪ 8 hours to 5% ▪ increased to 100% during next 8 hours • Release branch is merged to live branch ◦ hotfixes branched from here • Release branch is merged to integration ◦ via pull request
  17. Process: Publishing • multiple architectures • Progressive roll-out ◦ 5%

    for 24 hours ◦ Progressive increase during another 8 hours • Manual process :(