Devops Road at Tuenti

At some point a startup needs to grow, without losing agileness. It needs well-know processes. It needs developers to know about the infrastructure. It needs tools that expose knowledge, not that hide it. It needs scalable wokflows, based on self service tools, so system admins and release managers are not a bottleneck for developers. It needs tools that enforce processes, not tons of documentation.

So we implemented and put to work some systems to allow developers to do that:

* Tuenti-in-a-box: Private, replicable, virtualized development and testing environment, with all the features production platform has. No more shared development environments.

* Configcop: Test your application config changes in your development environment, promote it to staging and then to production. Easy, safe, and logged.

* Flow: Development, integration and release management of code made easy. Promote your own code to main integration branch, build and deploy it, release it, all pressing a button from your preferred ticketing system. Of course transparent information, traceability, stats.

All these tools are developing devops culture inside the company, making easy for developers to deploy and test their code, providing information about all stages and ensuring higher code quality by automating testing and other tedious, risky tasks, such as code versioning, tagging, build and distribution to app markets, etc. We would like to share how this ideas and tools helped in Tuenti's development processed, and allowed us to go from 2 releases per week, with multiple hotfixing in a single project, to a multiple project, multiple releases per day, reducing workload and time that code spends ready but unreleased from some days to some minutes.

Transcript

1. DEVOPS ROAD AT TUENTI Óscar San José (@oscarsj) Víctor García (@v1kt) October 11th, 2013
2. The company Tuenti: social network • 15 MM users, 500 MM chat messages sent/week, 2MM uploaded photos/day, ~180 engineers,... Tuenti Móvil: MVNO
3. Past Good • Agile • Information transparency Then we grew • Number of devs growing fast ◦ Chaotic • Unstable, unreliable, untested • Non repetitive • Non traceable • Stressful, lot of effort • Bottlenecks: ◦ Merges: tons of conflicts ◦ Tests running time ◦ Whole team focusing in release
4. Past Good • Agile • Information transparency Then we grew • Number of devs growing fast ◦ Chaotic • Unstable, unreliable, untested • Non repetitive • Non traceable • Stressful, lot of effort • Bottlenecks: ◦ Merges: tons of conflicts ◦ Tests running time ◦ Whole team focusing in release
5. The Road The team (us) • Developers with some experience as sysadmins (most of us) • Sysadmins with experience as developers • Skills with people • Release Management background The ideas • Build a platform to allow developers to self-service envs and services • Provide a toolchain so developers can build, test, deploy and run systems • Coach teams to move to this model
6. The Road::Tuenti-in-a-box Goal: • Private, replicable, virtualized development and testing environment, with all the features production platform has. Before • Shared development environment: ◦ bottlenecks ◦ worse performance ◦ wasted resources With Tuenti-in-a-box • Self service • Easy to set up and run ◦ Just “vagrant up” to get a new environment
7. The Road::Tuenti-in-a-box
8. The Road::ConfigCop Goal: • Builds, tests, and deploys configs to testing and production envs, self service Before • Release engineers in charge ◦ Bottleneck ◦ Knowledge silo ◦ Repetitive and boring task ◦ Manual and error prone With ConfigCop • Self service • Fast • Logs, Stats
9. The Road::TuentiDeployer Goal • Perform any kind of deployment within Tuenti ◦ Envs: Production, preproduction, staging,... ◦ Services: tuenticom, adserver, backend router,... • Extensible by the devs Before • Many types of “homemade” deploys • Not reliable nor trackable With TuentiDeployer • Easy to extend and very flexible • Devs acquire knowledge about environment, apps componentes and how is deployed: static servers, frontends, backends, … • Logs, Stats
10. The Road::Flow Goal • Tests, integrates, releases, builds and deploys - nicely, easily ◦ Clicking a single button triggers the processes • Radiate info to all stakeholders (transparency) • Systems integration: ◦ Jira, Jenkins, TuentiDeployer, Fisheye, GDocs, HGWeb, HockeyApps • Single, central point of information for all the company Modules • Integration pull requests • Release management • FlowDeployer • Dashboards • Stats and charts
11. The Road::Flow How: • Ambitious project • This is a product so development techniques were applied ◦ Incremental, mVP ◦ Evolved gradually ◦ Did not involve ruptures in processes • Joined small existent tools and created new ones
12. The Road::Flow
13. The Road::Flow
14. Present Figures: • 7.16 production config changes per day • 14.09 preproduction config changes per day • 3.21 production deployments per day • 67.64 preproduction deployments per day • 7.19 pull requests per day Almost no manual tasks Developers throughput increased Ageing of finished code not released greatly reduced Fast reaction time to requirement changes or bugs Information transparency
15. Future We’d love to • Open source tools • Support multiple SCM in our tools • Support all projects in tuenti, including Flow itself • Detach tools from Tuenti specific processes • Risk analysis and branch reordering • Parallel probabilistic pull request testing • Pure Continuous Delivery: deploy every changeset
16. THANKS! Óscar San José (@oscarsj) Víctor García (@v1kt)