Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous Delivery at Wix, Yaniv Even Haim

DevopsCon
January 28, 2013

Continuous Delivery at Wix, Yaniv Even Haim

We will go over the motivations for wix.com R&D to move to a CI/CD/TDD model, how the model was implemented and the impact on Wix R&D. We will cover the tools used (developed in-house and 3rd party), change in methodologies, what we have learned during the transformation and the unexpected change in working with product and the rest of the company.

Presented in the Continuous Delivery track at DevOps Con Israel 2013

DevopsCon

January 28, 2013
Tweet

More Decks by DevopsCon

Other Decks in Technology

Transcript

  1. Wix in numbers • 29,000,000 registered users from 233 countries

    • ~1,200,000 new users every month • ~ 30,000 new websites every day • 26,000,000 web sites • 800,000 mobile sites • 1Tbyte of media uploaded every day • More than 1 Billion users media files • 250+ servers in 3 data centers • 350 employees • 100 R&D people
  2. CD is business concern Traditional “Economies of scale” threaten by

    cloud technology Be faster than small startups CD “Economies of Agility “ = Velocity X Size 2010-8 2010-9 2010-… 2010-… 2010-… 2011-1 2011-2 2011-3 2011-4 2011-5 2011-6 2011-7 2011-8 2011-9 2011-… 2011-… 2011-… 2012-1 2012-2 2012-3 2012-4 Revenue per month
  3. CD is culture & mindset • Trust the developers –

    Empower developers to change production – Developer knows his system best
  4. CD is culture & mindset • Automation as a default

    choice – no more “is it worth to automate ? ” – Everything should be automated
  5. • Move fast - Less formality – Corridor - IN

    , Meeting Room - Out Welcome to the twilight zone
  6. Get out of thought land • The Law of Failure

    – Most new “its” will fail even if they are flawlessly executed • Invest less, in-touch less , better ability to admit it fail – Data beats opinions - Let the market decide make sure you building the right it before build it right
  7. Some Best Practices • Everyone develops on the Trunk/Master –

    Release frequent – small pieces of functionality – It’s the developer responsibility not to break anything • Code can get to production at anytime – TDD & integration test • Use Feature Toggle – Unbaked new code can go to production – no harm done – New code goes with a guard – use new or old code
  8. Some Best Practices • Backwards and Forwards compatible – Each

    component has to function with latest, next or prior version of other components (including DBs) • Gradual Deployment & Self Test – Deploy new version to one server and perform self test. If it passes, continue deployment to other servers • A/B Testing – Open a new feature to a percent of your users. Is it better? – Deployment is an engineering decision
  9. CD – prepare to invest….. • Dev infrastructure - Refactor

    , Refactor, Refactor • Testing infrastructure & know how • Deployment infrastructure & tools • Automation , Automation , Automation • Monitoring (business and technical) – hundreds of aspects – thresholds use is a Must – Monitor business KPIs – Internal & external – Endless Tuning & learning
  10. • Deployment complexity – More services – More dependencies –

    Multi data centers • Development complexity – Very hard to work on master/trunk with 30+ dev on same code base – backward and forward comptability is not easy • Wider set of technologies ,tools & languages – Higher level of infrastructure investments – Harder to have all around players – More communication/sync required Scaling challenges
  11. Scaling challenges – Quality – Testing coverage challenge • Use

    cases increases • Browsers wild west • Tablets and mobile variety – Feature toggle combinations • Dozens of parallel feature toggles – Production frequent changes • Services are updated daily • Toggles switch on/off on daily basis
  12. Scaling challenges - Availability • Service Availability • Cost of

    fault/downtime increasing as business grow • Service downtime during deployment is unacceptable • Human mistakes are inventible when changing production daily
  13. Scaling challenges – Monitoring • Monitoring in multi technology env

    • Unified dashboard require major effort • Reduce alert noises resulted by deployments • Identify root cause is not simple – Some faults does not have immediate impacts – Cascading effects • Reliable business KPI without false positive –Marketing changes –Special events (holidays, sport event etc)
  14. Scaling challenges – Product • Product MVP practices – Does

    MVP meet your product standards? • What about tooltip,help,first time ux, etc.. ? – How to define a product that can be developed in a day ? – And that can win in a/b test … To Be Implemented
  15. Scaling challenges – Org • Complex Matrix Management - People

    moving between teams frequently - One can have several managers on different tasks - Hard both to managers and employees - Everyone should master the matrix management skills