Strategies In Continuous Delivery

Find out some of the strategies we use in order to deploy 3000 time a year

Transcript

1. Strategies In ContinuousDelivery Learn the “magic” behind the scenes Aviran Mordo @aviranm http://www.linkedin.com/in/aviran 07:41
2. About Wix 20:37
3. Wixin Numbers • Over 35,000,000 users – Adding over 1,000,000 new users each month • Static storage is over 150TB of data – Adding over 1TB of files every day • 3 Data centers + 2 Clouds (Google AE, Amazon) – Around 300 servers • Over 40,000,000 Server API calls per day • ~400 people work at wix – Over 100 people in R&D 20:37
4. 20:37 From Jan’ 2013 –May 2013 • 1500 Deployments • 470 A/B Tests • 200 Feature Toggles
5. Is Doing 10s Of Deployments A Day Takes Guts ? 20:37
6. Continuous Delivery at Wix • Abandon “VERSION” paradigm – move feature centric life • Make small and frequent release as soon as possible • Automate everything – TDD/CI/CD • Measure Everything –A/B test every new feature –Monitor real KPIs (business, not CPU) 20:37
7. Continuous Delivery at Wix • Empower the developer –The developer is responsible from product idea to 1M active users – Remove every obstacle in the developer’s path – Big cultural change from waterfall – affects the whole company 20:40
8. Test Driven Development • No new code is pushed to Git without being fully tested • Before fixing a bug first write a test to reproduce the bug • Cover legacy (untested) systems with Integration tests 20:37
9. Test Driven Development • What people think is the impact on development –TDD slows down development – With TDD we write more code (product + test code). • Current Test Count (U-Tests + IT-Tests) – over 10,000 20:42
10. Test Driven Development • Actual impact on development –We develop products faster – Removes fear of change – Easier to enter some one else’s project –Do we really need QA? (Yes, they code tests) – Writing a feature is 10-30% slower, 45-90% less bugs – 50% faster to reach production. –Considerably faster time to fix bugs 20:37
11. Guidelines for successful TDD • Tests should run on project checkout to a random computer. • Tests that cannot be debugged on a developer machine will never consistently run for any period of time • Tests should run fast • Tests have to be readable – They are the project’s specs • Fixture is evil! 20:47
12. Feature Toggles 20:37
13. Feature Toggles • Everyone develops on the Trunk • Every piece of code can get to production at anytime 20:37
14. Feature Toggle to the rescue • Unused new code can go to production – no harm done • Operational new code goes with a guard – use new or old code by feature toggle 20:37
15. Usage example Simple “if” statement in your code 20:37
16. Feature Toggle Strategies (gradual expose users) • Company employees • Specific users or group of users • Percentage of traffic • By GEO • By Language • By user-agent • User Profile based • By context (site id or some kind of hash on site id) 20:37
17. Feature Toggle Override • By specific server – Used to test system load – New database flows/migration – Refactoring that may affect performance and memory usage • By Url parameter – Enable internal testing – Product acceptance – Faking GEO • By FT cookie value – Testing – When working with API on a single page application 20:37
18. Feature Toggles Management 20:37
19. A/B Tests 20:37
20. A/B Test • Every new feature is A/B tested • We open the new feature to a % of users – Define KPIs to check if the new feature is better or worse – If it is better, we keep it – If worse, we check why and improve – If we find flaws, the impact is just for % of our users (kind of Feature Toggle) • An interesting site effect on product • How many times did you have the conversion “what is better”? – Put the menu on top / on the side • Well, how about building both and A/B Testing? 20:37
21. Marking users with toss value in a cookie • Anonymous user – Toss is randomly determined – Can not guarantee persistent experience if changing browser • Registered User – Toss is determined by the user ID – Guarantee toss persistency across browsers – Allows setting additional tossing criteria (for example new users only) – Only use this for sections that a user has to be authenticated 20:37
22. • Do not mix anonymous and registered tests • AB test parentage of users with optional filters – New Users Only (Registered users only) – By language – By GEO – By Browser – user-agent – OS – Any other criteria you have on your users 20:37
23. A/B Test Features • A/B Test Override – Allows setting a value of a test for validation – Helps support experience what users experiencing • Override methods – Via URL parameter – Via cookie • Pause tests • Start/Stop Test • Bots always get “A” 20:37
24. Manage your tests 20:37
25. Refactoring 20:37
26. Refactoring • Before refactoring make sure everything is covered with tests – Legacy code usually covered by IT tests • Refactor from inside out – Small iterations with tests – Refactor small methods - make sure the tests don’t break – Deploy often • Re-write from the outside in – Write from scratch – Code duplication sometimes needed (temporary) – Protected by Feature Toggle 20:37
27. Code branch 20:37 New Code Old Code FT Opened Yes No
28. DB Schema Changes Without Downtime • Adding columns – Use another table link by primary key – Use blob field for schema flexibility • Removing fields – Stop using. Do not do any DB schema changes 20:37
29. New DB schema • Plan a lazy migration path controlled by feature toggle 1. Write to old / Read from old 2. Write to both / Read from old 3. Write to both / Read from new, fallback to old 4. Write to new / Read from new, fallback to old 5. Eagerly migrate data in the background 6. Write to new / Read from new 20:37
30. Gradual Deployment 20:37 • Assume two components • We shutdown one and install on it the new version. It is not active yet • Do self test • Activate the new server it is passes self test • Continue deploying the other servers, a few at a time, checking each one with self test A 1.1 B 1.1 A 1.1 B 1.2 A 1.1 A 1.1 B 1.1 B 1.1 A 1.1 A 1.1 B 1.1 B 1.2 A 1.1 B 1.2 A 1.1 A 1.1 B 1.1 B 1.2 A 1.1 B 1.1 A 1.1 A 1.1 B 1.1 B 1.2
31. Self Test / Post Deployment Test After each server deployment run a self test before deploying the next server. • Checking server configuration and topology – Make sure database is accessible (DB connection string) – Is the schema the one I expect – Access required local resources (data files, other config files, templates, etc’) – Access remote resources – RPC / REST endpoints reachable and operational • Server will refuse requests unless it passes the self test • Allow a way to skip self test (and continue deployment) 20:37
32. Tools -App-info 20:37
33. Backward and Forward compatible • Assume two components • We release a new version of one • Now Rollback the other… 20:37 A 1.1 B 1.2 A 1.2 B 1.1 A 1.1 A 1.1 B 1.1 B 1.2 A 1.2 A 1.1 B 1.1 B 1.1 A 1.1 B 1.1 A 1.1 A 1.1 B 1.1 B 1.1 A 1.0 A 1.2 A 1.1 B 1.2 B 1.1 B 1.2 A 1.2 A 1.2 A 1.1 B 1.2 B 1.1 B 1.0
34. Production Visibility 20:37 Prepare for a rainy day
35. How does it work –CD Practices • Test driven development • Small Development Iterations • Backwards and Forwards compatible • Gradual Deployment & Self-Test • Feature Toggle • A/B Testing • Exception Classification • Production visibility 20:37
36. Where are we today? • We have re-written our flash editor product as an HTML 5 editor – In just 4 months • Introduced Wix 3rd party applications (developers API) – In just 6 weeks • We are easily replacing significant parts of our infrastructure • And we are doing ~10 releases a day! 20:37 0 5 10 15 20 25 30 35 1/1/2013 1/4/2013 1/8/2013 1/11/2013 1/15/2013 1/18/2013 1/22/2013 1/27/2013 1/30/2013 2/3/2013 2/6/2013 2/10/2013 2/13/2013 2/18/2013 2/21/2013 2/25/2013 2/28/2013 3/4/2013 3/7/2013 3/12/2013 3/15/2013 3/19/2013 3/24/2013 4/3/2013 4/7/2013 4/10/2013 4/15/2013 4/19/2013 4/23/2013 4/28/2013 5/1/2013 5/6/2013 5/9/2013 5/13/2013 5/19/2013 5/22/2013 5/26/2013 5/29/2013 6/1/2013 6/4/2013 6/7/2013 6/11/2013 6/16/2013 (blank) Total Total
37. 20:37
38. 07:41 Aviran Mordo @aviranm http://www.linkedin.com/in/aviran Read more: The Road To Continuous Delivery: http://goo.gl/K6zEK http://www.slideshare.net/aviranwix/strategies-in-continuous-delivery