Integrating and Releasing code at Tuenti: an automated pipeline

Integrar, probar y distribuir el código en el que trabajan unos 30 desarrolladores supone un reto. Diseñando procesos flexibles, pero cuyo cumplimiento sea estricto, y automatizándolos, en Tuenti hemos aplicado al desarrollo de aplicaciones móviles las lecciones que nos permitieron incrementar el número de releases de código de nuestra web de dos a la semana a 2 al día, resultando en un proceso transparente al desarrollador, y con todas las características que se exigen de un proceso de release: seguridad, trazabilidad, repetibilidad, automatización y eficiente flujo de información de arriba a abajo y de abajo arriba.

Transcript

1. Integrating and Releasing code at
   Tuenti: an automated pipeline
   @oscarsj
   

   View Slide

2. Problem
   ● Lots of developers writing code at the same
   time
   ● Buggy code gets to main development
   branch
   ● A lot of time wasted checking test results,
   performing builds, merging branches,
   publishing versions…
   ● Information missing, lots of meetings
   required
   

   View Slide

3. Solution
   Create a process that is:
   ● Well-defined
   ● Repeteable
   ● Trackable
   ● Automated (as far as possible)
   How?
   

   View Slide

4. Agenda
   1. Overview
   a. Dev/Release system architecture
   b. Flow
   2. Process
   a. Development
   b. Integration
   c. Distribution
   d. Release
   e. Publishing
   

   View Slide

5. Dev/Release system architecture
   

   View Slide

6. Dev/Release system architecture
   

   View Slide

7. Dev/Release system architecture
   

   View Slide

8. Dev/Release system architecture
   

   View Slide

9. Dev/Release system architecture
   

   View Slide

10. Flow
    

    View Slide

11. Dev/Release system architecture
    

    View Slide

12. Mercurial (SCM)
    ● Mercurial/Git: distributed scm
    ○ every repo is a full copy, with history
    ○ Developers work on their copy, and frequently
    synchronize with central repo (push/pull)
    ● Protected branches:
    ○ integration
    ○ release-*
    ○ hotfix-*
    ○ live
    

    View Slide

13. Mercurial (SCM)
    

    View Slide

14. Mercurial (SCM)
    ● Sanity check “hooks”
    ○ Syntax checks
    ○ No conflict markers
    ○ All commits with ticket identifier
    ○ Data formats validation
    ● Tags:
    ○ All versions published are uniquely identified by
    version
    ○ This allow to track back errors to the code
    

    View Slide

15. Dev/Release system architecture
    

    View Slide

16. Jenkins
    mvn
    

    View Slide

17. Dev/Release system architecture
    

    View Slide

18. Hockeyapps
    ● Development versions distribution system
    ○ http://hockeyapp.net/
    ● Stores versioned apks, along with symbol
    tables
    ● Provides an API to include in the app:
    ○ Send crash reports to hockeyapps
    ○ Notifies and installs new versions from server
    

    View Slide

19. Hockeyapps
    ● Projects:
    ○ alphas: loose versioning, teams manage it
    themselves
    ○ preproduction: strict, production versioning. Only
    versions built from integration/release pipeline
    ○ production: no distribution, only to store crashes
    

    View Slide

20. Hockeyapps
    ● Development versions generated locally, by
    devs
    ● Preproduction versions generated from
    integration and release branches
    ○ Tagged, and version increased after successful
    publish
    ● Versioning in integration and release branch
    ○ Major.minor.build
    

    View Slide

21. Dev/Release system architecture
    

    View Slide

22. Jira
    ● Issue tracker
    ○ Rest API
    ○ Hooks -> events
    ● All info of project stored here
    ○ Human generated: Content of branches, risks,
    effects on customers, etc
    ○ Auto generated: Results of builds, times of
    deployment, etc
    ● Process triggered from here
    ○ Integrate branch
    ○ Create releases
    ○ Publish versions
    

    View Slide

23. Dev/Release system architecture
    

    View Slide

24. Release Manager
    ● Controls processes
    ● Checks tests status
    ● Checks integrity of builds and branches
    ● Checks for abnormal situations
    ● Broadcast info for stakeholders
    ● Operates scripts to build and deploy to
    different environments
    

    View Slide

25. Flow
    

    View Slide

26. Flow
    

    View Slide

27. Process
    ● From development (my laptop) to production
    ● As fast as possible: all code finished but not
    published gets old
    ● Quality is integral part of the process:
    ○ devs: automated testing
    ○ product: ensure quality as a feature
    ○ leads: provide means with scheduling and formation
    ● Makes sure code in integration branch is
    always “good” (passes tests)
    

    View Slide

28. Process: Development
    1. Developer branches from integration
    2. Creates a INT (integration request) ticket
    3. Develops, tests
    4. Peer code reviews
    5. Pull from integration branch
    6. Translations
    7. Ready (reviewed+tests passing+feature
    accepted): pull request
    8. Waits for feedback
    

    View Slide

29. Process: Integration
    ● All info to merge a branch in jira ticket:
    ○ Changeset
    ○ Author/Reviewer
    ○ Contents
    ○ Risks...
    ● Create temporal branch with integration +
    feature branch
    ● Test that temporal branch
    ● Blue ball: integrated, otherwise: rejected
    ● Feedback:
    ○ Jira
    ○ Mail
    

    View Slide

30. Process: Distribution
    ● Integration latest changeset gets build in
    “production” mode (no debug, ofuscated,
    prod. key)
    ● If build succeeds:
    ○ Pushed to hockeyapps
    ○ Version (build number) increased
    ○ Tag in mercurial with version number
    ○ Feedback via mail, with information, changelog and
    link to donwload
    ● Periodically, at least once a week
    ○ Too often may be counterproductive
    

    View Slide

31. Process: Release
    ● Integration latest changeset get branched
    ● Latest translations included
    ● Latest hotfixes included
    ● Product version set (major.minor) if needed
    ● Build/distribution
    ○ all company encouraged to install
    ● Stabilization:
    ○ 3 days
    ○ devs must check and fix crashes related to release
    ○ at the end of each day a new build is distributed
    

    View Slide

32. Process: Release
    ● Release stable: agreement between
    developers and product owners
    ● Google play submit
    ○ progressive roll-out:
    ■ 8 hours to 5%
    ■ increased to 100% during next 8 hours
    ● Release branch is merged to live branch
    ○ hotfixes branched from here
    ● Release branch is merged to integration
    ○ via pull request
    

    View Slide

33. Process: Publishing
    ● multiple architectures
    ● Progressive roll-out
    ○ 5% for 24 hours
    ○ Progressive increase during another 8 hours
    ● Manual process :(
    

    View Slide

34. QUESTIONS?
    @oscarsj
    

    View Slide