ReBuild 19 - Lessons learned on DevOps

Transcript

1. Azure Engineering
   @cmaneu
   

   View Slide

2. Increase flow of value Shorten cycle times Continuously Improve
   

   View Slide

3. View Slide

4. Sprint 1
   August 2010
   Sprint 29
   June 2012
   VSTS Preview
   Sprint 64
   April 2014
   VSTS GA
   Sprint 67
   June 2014
   1ES
   Sprint 92
   Nov 2015
   Extensions
   Sprint 102
   June 2016
   GVFS
   Sprint 124
   Sep 2016
   

   View Slide

5. Build what
   customers
   actually want
   Empower
   individuals and
   teams
   Innovate at a
   Cloud
   Cadence
   More features in the 2016
   calendar year (262 features)
   than the previous 4 years
   Combined (256 features).
   And the growth continues!
   22
   58 65
   111
   262
   435
   2012 2013 2014 2015 2016 2017
   

   View Slide

6. Team Autonomy + Enterprise Alignment
   Shift Left Quality
   Customer Focused
   Production First Mindset
   Infrastructure as Flexible Resource
   

   View Slide

7. Team Autonomy + Enterprise Alignment
   Shift Left Quality
   Customer Focused
   Production First Mindset
   Infrastructure as Flexible Resource
   

   View Slide

8. Alignment
   Autonomy
   “Let’s try to give our teams three things….
   Autonomy, Mastery, Purpose”
   

   View Slide

9. View Slide

10. View Slide

11. View Slide

12. Teams
     Physical team rooms
     Cross discipline
     10-12 people
     Self managing
     Clear charter and goals
     Intact for 12-18 months
     Own features in production
     Own deployment of features
    

    View Slide

13. UI
    API
    Data
    

    View Slide

14. UI
    API
    Data
    

    View Slide

15. Employee choice, not
    manager driven
    Typically <20%
    change, but 100% get
    to make a choice
    Cross-pollinate talent
    and micro-culture
    Sticky Note Exercise - Self Forming Teams
    

    View Slide

16. Sprint
    3 weeks
    1
    Plan
    3 sprints
    3
    Season
    6 months
    6
    Scenario
    18 months
    Teams are responsible for the detail
    Leadership is responsible
    for driving the big picture
    

    View Slide

17. Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
    18 month scenario
    6 month plan
    CY17 H1
    

    View Slide

18. Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
    18 month scenario
    6 month plan
    

    View Slide

19. Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
    18 month scenario
    6 month plan
    

    View Slide

20. Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
    18 month scenario
    6 month plan
    

    View Slide

21. Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
    18 month scenario
    6 month plan
    

    View Slide

22. 3 weeks
    Spring
    Fall
    Spring Fall
    

    View Slide

23. What we accomplished
    Progressive
    Deployment
    Week 1 Week 2 Week 3
    Week 1 Week 2 Week 3
    Week 2 Week 3
    Sprint 98
    Sprint 97 Sprint 99
    The sprint plan
    

    View Slide

24. View Slide

25. View Slide

26. Spring
    Fall
    Spring Fall
    3 sprints
    

    View Slide

27. View Slide

28. View Slide

29. Work Management
    Information Tools
    Coding Tools
    Testing Tools
    Build Tools
    Debugging Tools
    Developer
    Time
    Source: Internal Microsoft Telemetry analysis of 4700 engineers in WDG
    

    View Slide

30. Team Autonomy + Enterprise Alignment
    Shift Left Quality
    Customer Focused
    Production First Mindset
    Infrastructure as Flexible Resource
    

    View Slide

31. View Slide

32. Code Test & Stabilize Code Test & Stabilize
    Code
    Complete
    Planning
    

    View Slide

33. View Slide

34. engineers on
    your team
    # 5 ?
    x =
    

    View Slide

35. Rule: If your bug count exceeds your bug cap… stop working
    on new features until you’re back under the cap.
    5 50
    x =
    10
    

    View Slide

36. • Tests that anyone can run anywhere (inc production)
    • Shifted to unit tests from automated functional tests
    • Core tests run before pull request
    • Fast and 100% reliable build and test is critical
    • Rolling tests run after commit
    

    View Slide

37. View Slide

38. Team Autonomy + Enterprise Alignment
    Shift Left Quality
    Customer Focused
    Production First Mindset
    Infrastructure as Flexible Resource
    

    View Slide

39. View Slide

40. View Slide

41. View Slide

42. • Original estimate
    • Completed hours
    • Lines of Code
    • Team capacity
    • Team burndown
    • Team velocity
    • # of bugs found
    Things we don’t watch
    • Acquisition
    • Engagement
    • Satisfaction
    • Churn
    • Feature Usage
    Usage
    • Time to Detect
    • Time to Communicate
    • Time to Mitigate
    • Customer Impact
    • Incident Prevention Items
    • Aging Live Site Problems
    • SLA per Customer
    • Customer Support Metrics
    Live Site Health
    • Time to Build
    • Time to Self Test
    • Time to Deploy
    • Time to Learn
    Velocity
    

    View Slide

43. Found one of the top customers with low availability.
    Proactively reached out and resolved their issue.
    

    View Slide

44. View Slide

45. View Slide

46. View Slide

47. Team Autonomy + Enterprise Alignment
    Shift Left Quality
    Customer Focused
    Production First Mindset
    Infrastructure as Flexible Resource
    

    View Slide

48. •
    •
    •
    •
    •
    •
    •
    

    View Slide

49. View Slide

50. 
    
    
    
    

    View Slide

51. 1 2 3 4 5
    

    View Slide

52. https://github.com/Netflix/Hystrix/wiki
    

    View Slide

53. •
    •
    •
    •
    •
    •
    •
    •
    •
    

    View Slide

54.  Double blind test
     Full disclosure at or near end
    vs.
     Share tactics & lessons learned
     Continued evolution
    Assume Breach - Use War Games to the learn attacks and practice response
    

    View Slide

55. Team Autonomy + Enterprise Alignment
    Shift Left Quality
    Customer Focused
    Production First Mindset
    Infrastructure as Flexible Resource
    

    View Slide

56. View Slide

57. Shared abstraction layer
    Single master branch, multiple release branches
    Update 1
    Update 2
    Update N
    Visual Studio Team Services
    Team Foundation Server
    One code base with multiple delivery streams
    

    View Slide

58. Shared Platform Services (SPS)
    North Central
    TFS SU1
    North Central
    AT
    AT
    AT
    JA
    JA
    JA
    Blob
    TFS SU7
    Australia
    TFS SU0
    West Central
    Containerized Services
    

    View Slide

59. •
    •
    •
    •
    •
    • If you are starting out today and cloud native, consider
    PaaS, Service Fabric + Azure Functions
    • We need to ship same code to on-prem & cloud
    

    View Slide

60. After
    • 3-week sprints
    • Vertical teams
    • Team rooms
    • Continual Planning & Learning
    • PM & Engineering
    • Continual customer engagement
    • Everyone in master
    • 8-12 person teams
    • Publicly shared roadmap
    • Zero debt
    • Mockups in PPT
    • Inner source
    • Flattened organization hierarchy
    • User satisfaction determines success
    • Features shipped every sprint
    Before
    • 4-6 month milestones
    • Horizontal teams
    • Personal offices
    • Long planning cycles
    • PM, Dev, Test
    • Yearly customer engagement
    • Feature branches
    • 20+ person teams
    • Secret roadmap
    • Bug debt
    • 100 page spec documents
    • Private repositories
    • Deep organizational hierarchy
    • Success is a measure of install numbers
    • Features shipped once a year
    

    View Slide

61. View Slide