ReBuild 19 - Lessons learned on DevOps

Transcript

1. Azure Engineering @cmaneu
2. Increase flow of value Shorten cycle times Continuously Improve
4. Sprint 1 August 2010 Sprint 29 June 2012 VSTS Preview Sprint 64 April 2014 VSTS GA Sprint 67 June 2014 1ES Sprint 92 Nov 2015 Extensions Sprint 102 June 2016 GVFS Sprint 124 Sep 2016
5. Build what customers actually want Empower individuals and teams Innovate at a Cloud Cadence More features in the 2016 calendar year (262 features) than the previous 4 years Combined (256 features). And the growth continues! 22 58 65 111 262 435 2012 2013 2014 2015 2016 2017
6. Team Autonomy + Enterprise Alignment Shift Left Quality Customer Focused Production First Mindset Infrastructure as Flexible Resource
7. Team Autonomy + Enterprise Alignment Shift Left Quality Customer Focused Production First Mindset Infrastructure as Flexible Resource
8. Alignment Autonomy “Let’s try to give our teams three things…. Autonomy, Mastery, Purpose”
12. Teams  Physical team rooms  Cross discipline  10-12 people  Self managing  Clear charter and goals  Intact for 12-18 months  Own features in production  Own deployment of features
13. UI API Data
14. UI API Data
15. Employee choice, not manager driven Typically <20% change, but 100% get to make a choice Cross-pollinate talent and micro-culture Sticky Note Exercise - Self Forming Teams
16. Sprint 3 weeks 1 Plan 3 sprints 3 Season 6 months 6 Scenario 18 months Teams are responsible for the detail Leadership is responsible for driving the big picture
17. Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 18 month scenario 6 month plan CY17 H1
18. Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 18 month scenario 6 month plan
19. Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 18 month scenario 6 month plan
20. Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 18 month scenario 6 month plan
21. Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 18 month scenario 6 month plan
22. 3 weeks Spring Fall Spring Fall
23. What we accomplished Progressive Deployment Week 1 Week 2 Week 3 Week 1 Week 2 Week 3 Week 2 Week 3 Sprint 98 Sprint 97 Sprint 99 The sprint plan
26. Spring Fall Spring Fall 3 sprints
29. Work Management Information Tools Coding Tools Testing Tools Build Tools Debugging Tools Developer Time Source: Internal Microsoft Telemetry analysis of 4700 engineers in WDG
30. Team Autonomy + Enterprise Alignment Shift Left Quality Customer Focused Production First Mindset Infrastructure as Flexible Resource
32. Code Test & Stabilize Code Test & Stabilize Code Complete Planning
34. engineers on your team # 5 ? x =
35. Rule: If your bug count exceeds your bug cap… stop working on new features until you’re back under the cap. 5 50 x = 10
36. • Tests that anyone can run anywhere (inc production) • Shifted to unit tests from automated functional tests • Core tests run before pull request • Fast and 100% reliable build and test is critical • Rolling tests run after commit
38. Team Autonomy + Enterprise Alignment Shift Left Quality Customer Focused Production First Mindset Infrastructure as Flexible Resource
42. • Original estimate • Completed hours • Lines of Code • Team capacity • Team burndown • Team velocity • # of bugs found Things we don’t watch • Acquisition • Engagement • Satisfaction • Churn • Feature Usage Usage • Time to Detect • Time to Communicate • Time to Mitigate • Customer Impact • Incident Prevention Items • Aging Live Site Problems • SLA per Customer • Customer Support Metrics Live Site Health • Time to Build • Time to Self Test • Time to Deploy • Time to Learn Velocity
43. Found one of the top customers with low availability. Proactively reached out and resolved their issue.
47. Team Autonomy + Enterprise Alignment Shift Left Quality Customer Focused Production First Mindset Infrastructure as Flexible Resource
48. • • • • • • •
50.    
51. 1 2 3 4 5
52. https://github.com/Netflix/Hystrix/wiki
53. • • • • • • • • •
54.  Double blind test  Full disclosure at or near end vs.  Share tactics & lessons learned  Continued evolution Assume Breach - Use War Games to the learn attacks and practice response
55. Team Autonomy + Enterprise Alignment Shift Left Quality Customer Focused Production First Mindset Infrastructure as Flexible Resource
57. Shared abstraction layer Single master branch, multiple release branches Update 1 Update 2 Update N Visual Studio Team Services Team Foundation Server One code base with multiple delivery streams
58. Shared Platform Services (SPS) North Central TFS SU1 North Central AT AT AT JA JA JA Blob TFS SU7 Australia TFS SU0 West Central Containerized Services
59. • • • • • • If you are starting out today and cloud native, consider PaaS, Service Fabric + Azure Functions • We need to ship same code to on-prem & cloud
60. After • 3-week sprints • Vertical teams • Team rooms • Continual Planning & Learning • PM & Engineering • Continual customer engagement • Everyone in master • 8-12 person teams • Publicly shared roadmap • Zero debt • Mockups in PPT • Inner source • Flattened organization hierarchy • User satisfaction determines success • Features shipped every sprint Before • 4-6 month milestones • Horizontal teams • Personal offices • Long planning cycles • PM, Dev, Test • Yearly customer engagement • Feature branches • 20+ person teams • Secret roadmap • Bug debt • 100 page spec documents • Private repositories • Deep organizational hierarchy • Success is a measure of install numbers • Features shipped once a year