Tuenti Release Workflow

How Tuenti code lifetime is handled, releases are done and which opensource tools we use.

Transcript

1. Release Workflow
   Diego Muñoz
   diego@tuenti.com
   http://twitter.com/Kartones
   V1.1
   

   View Slide

2. Agenda
   • Numbers
   • Release workflow
   • Tools
   

   View Slide

3. Numbers
   • +13M users (~10,4M active users dec 2011)
   • +100 usage minutes / day (avg)
   • +400M chat messages / day
   • +4M photos uploaded / day (peaks)
   • +41,000M page views / month
   • +35K requests / sec (peaks)
   • +1,3K servers
   • +250 employees (~60% techies)
   • +15K files in the repositories
   • +10K Tests
   

   View Slide

4. Release Workflow
   Branch Code Test Integrate Release Stabilize
   

   View Slide

5. Release Workflow: Branch
   Branch Code Test Integrate Release Stabilize
   • Avg. 15 branches per release
   • Current record: 29 branches
   • Repository per functional area (be, fe, stats, …)
   • Avg. lines modified per release: 63K
   

   View Slide

6. Release Workflow: Code + Test
   Branch Code Test Integrate Release Stabilize
   • Scrum (or at least Agile)
   • As TDD as possible
   • Labs
   • A/B Testing
   • PoCs
   • Dark launch
   

   View Slide

7. Release Workflow: Integrate
   Branch Code Test Integrate Release Stabilize
   • Repo always available
   • Specific release date given by devops
   – Merge & wait for target
   • Only merge if 100% tests ok or specific approval
   • QA Regression & manual tests
   • Fix possible integration problems ASAP
   

   View Slide

8. Release Workflow: Release
   Branch Code Test Integrate Release Stabilize
   • 3 releases per week
   – DevOps goal: All weekdays
   • Latest stable changeset from Integration taken
   previous working day morning
   • Release doc, pre-release meetings
   • Staging servers to test with live data
   

   View Slide

9. Release Workflow: Stabilize
   Branch Code Test Integrate Release Stabilize
   • First code push: 8 AM
   – DevOps Goal: single push + release closed
   • Release window: 1-2 h
   – DevOps goal: < 30 minutes
   • Error stabilization or release rollback
   • Representatives from all involved teams
   

   View Slide

10. Tools
    

    View Slide

11. DVCS: Mercurial
    • http://mercurial.selenic.com/
    • Syntax similar to SVN (our old system)
    • Easy API to plug our plugins and hooks
    • Cross-platform
    • Tuenti Addons:
    – Commit hooks to check syntax, push ticket #...
    • Problems:
    – Push/pulls through VPN are slow
    – Handling multiple repos still slow
    – Only one level of rollback!
    

    View Slide

12. Issue Tracking: Trac
    • http://trac.edgewall.org/
    • User Stories tasks + Bugs
    • Wiki (now also internal Google Sites)
    • Plugins and extensible
    • Tuenti Addons:
    – Master/Slave architecture
    – Tons of tweaks and source code integration hooks
    • Problems:
    – Slow, limited, code viewing sucks
    • Migration to JIRA planned
    

    View Slide

13. Testing: PHPUnit
    • http://www.phpunit.de
    • Some caveats
    – Mocking just ‘works’
    – PHP process spawning PHP tests
    • Tuenti Addons:
    – Vastly improved mocking framework
    – Shell scripts that isolate test batteries
    – Better integration with Selenium
    • Problems:
    – Our current FEFW does not cope perfectly with
    PHPUnit/Selenium
    

    View Slide

14. Testing: Selenium
    • http://seleniumhq.org/
    • Running browser tests in FFox and IE
    • Tuenti Addons:
    –Custom build with some fixes
    • Problems:
    –Javascript handling/detection not perfect
    –AJAX far from optimal
    –IE runner is an iframe
    • Planned migration to Webdriver
    

    View Slide

15. CI: Jenkins
    • http://jenkins-ci.org/
    • Previously Hudson too
    • Specialized farm (master + 22 nodes)
    • Tuenti Addons:
    • Parallelization (up to 6 nodes)
    • Special reports
    • “Smart” runs (try first last failed tests, etc.)
    • Problems:
    • Browser tests slow (due to Selenium)
    • Unstable (mainly due to Selenium)
    

    View Slide

16. Storage: MySQL
    • http://www.mysql.com/ | http://www.percona.com
    • Live site storage
    • Dev. env. storage
    – 1 DB per user (to run tests)
    – 1 shared DB (common faked data)
    • Clusters of master/slave DBs
    • Problems:
    – Slow when running tests
    – Shared dev DB has old-time inconsistencies
    

    View Slide

17. Storage: Hadoop
    • http://hadoop.apache.org/
    • Dedicated cluster
    • Pig scripts: Stats, other non-realtime data
    • HBase: Async. data storage
    • Hive: SQL-like querying
    • Problems:
    –Complex configuration for newcomers
    

    View Slide

18. Caching: Memcached
    • http://memcached.org/
    • Avg. DB querys/pageview: 0.3
    • Dev. Behaviour == live behaviour
    • Tuenti Addons (https://github.com/tuenti):
    – UDP + multi-ports
    • Problems:
    – 32GB RAM / machine practical limit
    – Remember to warm-up data or MC will kill the DB!
    

    View Slide

19. Configuration: Puppet
    • http://puppetlabs.com/
    • Production machines
    • Jenkins nodes
    • VM management / Dev web servers config
    • Problems:
    – Wipes user config if not puppetized
    

    View Slide

20. Search: Sphinx
    • http://sphinxsearch.com/
    • Non-realtime (index based)
    • Very fast
    • Problems
    – Index re-generation on dev & test env.
    – Could be more friendly to add new data
    

    View Slide

21. Build: Our build script
    • http://ant.apache.org/
    • Localization
    • Minification + Bundling + Versioning
    • Statics deployment to CDNs
    • Fast: 2-3 minutes full build
    – Multithreading + parallelization
    • Allows partial, component based builds
    • Problems:
    – Under heavy CPU load, build time goes up :(
    

    View Slide

22. Build: RSync
    • http://rsync.samba.org/
    • Deployment of code (live & dev)
    • Sends deltas/diffs
    • Really fast
    

    View Slide

23. Statics bundling: YUI
    • Less files == faster download & deploy
    • Big text file == better HTTP Gzip
    • HTML/JS/CSS Minification
    • Ultra fast build: ~4 seconds in Dev.
    • On demand JS loading!
    • Nice typical framework features
    • Wonderful & simple events system
    

    View Slide

24. Statics bundling: YUI (II)
    • Tuenti Addons:
    – Caching builds
    – Line breaks each # characters (easier IE debugging)
    – CDNs handling
    • Problems:
    – Change in JS requires rebuilding even in dev.
    – Requires small migrations/changes in existing JS
    

    View Slide

25. Chat Server: Ejabberd
    • http://www.ejabberd.im/
    • Erlang XMPP (Jabber) server
    • 400M msgs/day, 1M concurrent users peak,…
    • 20 machines, ~5 instances per machine
    • Tuenti Addons:
    – Ejabberd codebase tweaked (3,5x faster)
    – Protocol tweaks to optimize for our architecture
    • Problems:
    – Same behaviour dev/live is critical
    

    View Slide

26. Compilation: HipHop
    • Migrating old code to fully support HipHop
    – With PHP 5.3
    • Obvious speed improvements
    • Also nice for static code analysis
    

    View Slide

27. Sounds interesting?
    http://jobs.tuenti.com
    exit(0);
    

    View Slide