Devops Road at Tuenti

At some point a startup needs to grow, without losing agileness. It needs well-know processes. It needs developers to know about the infrastructure. It needs tools that expose knowledge, not that hide it. It needs scalable wokflows, based on self service tools, so system admins and release managers are not a bottleneck for developers. It needs tools that enforce processes, not tons of documentation.

So we implemented and put to work some systems to allow developers to do that:

* Tuenti-in-a-box: Private, replicable, virtualized development and testing environment, with all the features production platform has. No more shared development environments.

* Configcop: Test your application config changes in your development environment, promote it to staging and then to production. Easy, safe, and logged.

* Flow: Development, integration and release management of code made easy. Promote your own code to main integration branch, build and deploy it, release it, all pressing a button from your preferred ticketing system. Of course transparent information, traceability, stats.

All these tools are developing devops culture inside the company, making easy for developers to deploy and test their code, providing information about all stages and ensuring higher code quality by automating testing and other tedious, risky tasks, such as code versioning, tagging, build and distribution to app markets, etc. We would like to share how this ideas and tools helped in Tuenti's development processed, and allowed us to go from 2 releases per week, with multiple hotfixing in a single project, to a multiple project, multiple releases per day, reducing workload and time that code spends ready but unreleased from some days to some minutes.

Transcript

1. DEVOPS ROAD AT TUENTI
   Óscar San José (@oscarsj)
   Víctor García (@v1kt)
   October 11th, 2013
   

   View Slide

2. The company
   Tuenti: social network
   ● 15 MM users, 500 MM chat messages sent/week, 2MM
   uploaded photos/day, ~180 engineers,...
   Tuenti Móvil: MVNO
   

   View Slide

3. Past
   Good
   ● Agile
   ● Information transparency
   Then we grew
   ● Number of devs growing fast
   ○ Chaotic
   ● Unstable, unreliable, untested
   ● Non repetitive
   ● Non traceable
   ● Stressful, lot of effort
   ● Bottlenecks:
   ○ Merges: tons of conflicts
   ○ Tests running time
   ○ Whole team focusing in release
   

   View Slide

4. Past
   Good
   ● Agile
   ● Information transparency
   Then we grew
   ● Number of devs growing fast
   ○ Chaotic
   ● Unstable, unreliable, untested
   ● Non repetitive
   ● Non traceable
   ● Stressful, lot of effort
   ● Bottlenecks:
   ○ Merges: tons of conflicts
   ○ Tests running time
   ○ Whole team focusing in release
   

   View Slide

5. The Road
   The team (us)
   ● Developers with some experience as sysadmins (most of us)
   ● Sysadmins with experience as developers
   ● Skills with people
   ● Release Management background
   The ideas
   ● Build a platform to allow developers to self-service envs and services
   ● Provide a toolchain so developers can build, test, deploy and run systems
   ● Coach teams to move to this model
   

   View Slide

6. The Road::Tuenti-in-a-box
   Goal:
   ● Private, replicable, virtualized development and testing environment, with
   all the features production platform has.
   Before
   ● Shared development environment:
   ○ bottlenecks
   ○ worse performance
   ○ wasted resources
   With Tuenti-in-a-box
   ● Self service
   ● Easy to set up and run
   ○ Just “vagrant up” to get a new environment
   

   View Slide

7. The Road::Tuenti-in-a-box
   

   View Slide

8. The Road::ConfigCop
   Goal:
   ● Builds, tests, and deploys configs to testing and production envs, self
   service
   Before
   ● Release engineers in charge
   ○ Bottleneck
   ○ Knowledge silo
   ○ Repetitive and boring task
   ○ Manual and error prone
   With ConfigCop
   ● Self service
   ● Fast
   ● Logs, Stats
   

   View Slide

9. The Road::TuentiDeployer
   Goal
   ● Perform any kind of deployment within Tuenti
   ○ Envs: Production, preproduction, staging,...
   ○ Services: tuenticom, adserver, backend router,...
   ● Extensible by the devs
   Before
   ● Many types of “homemade” deploys
   ● Not reliable nor trackable
   With TuentiDeployer
   ● Easy to extend and very flexible
   ● Devs acquire knowledge about environment, apps componentes and how
   is deployed: static servers, frontends, backends, …
   ● Logs, Stats
   

   View Slide

10. The Road::Flow
    Goal
    ● Tests, integrates, releases, builds and deploys - nicely, easily
    ○ Clicking a single button triggers the processes
    ● Radiate info to all stakeholders (transparency)
    ● Systems integration:
    ○ Jira, Jenkins, TuentiDeployer, Fisheye, GDocs, HGWeb, HockeyApps
    ● Single, central point of information for all the company
    Modules
    ● Integration pull requests
    ● Release management
    ● FlowDeployer
    ● Dashboards
    ● Stats and charts
    

    View Slide

11. The Road::Flow
    How:
    ● Ambitious project
    ● This is a product so development techniques were applied
    ○ Incremental, mVP
    ○ Evolved gradually
    ○ Did not involve ruptures in processes
    ● Joined small existent tools and created new ones
    

    View Slide

12. The Road::Flow
    

    View Slide

13. The Road::Flow
    

    View Slide

14. Present
    Figures:
    ● 7.16 production config changes per day
    ● 14.09 preproduction config changes per day
    ● 3.21 production deployments per day
    ● 67.64 preproduction deployments per day
    ● 7.19 pull requests per day
    Almost no manual tasks
    Developers throughput increased
    Ageing of finished code not released greatly reduced
    Fast reaction time to requirement changes or bugs
    Information transparency
    

    View Slide

15. Future
    We’d love to
    ● Open source tools
    ● Support multiple SCM in our tools
    ● Support all projects in tuenti, including Flow itself
    ● Detach tools from Tuenti specific processes
    ● Risk analysis and branch reordering
    ● Parallel probabilistic pull request testing
    ● Pure Continuous Delivery: deploy every changeset
    

    View Slide

16. THANKS!
    Óscar San José (@oscarsj)
    Víctor García (@v1kt)
    

    View Slide