DevOpsDaysPortugal 2019 - Allegra Cooper - Applying Shift Left Test Strategy for Higher Quality

Shift Left is a mind shift to test earlier in the development life cycle. A Shift Left strategy, enables teams to realize outcomes such as higher quality applications, creating faster feedback loops and more confidence making changes to their application. It’s about evolving the culture within the agile team which will lead to success in achieving continuous delivery.

Transcript

1. Applying Shift Left Test Strategy for Higher Quality Allegra B. Cooper allegra_b_cooper@vanguard.com Technical Lead Chief Technology Office @ Vanguard
2. It’s nice to meet you… • Currently Technical Lead • Life at Vanguard: 6 years, 3 divisions, 4 Teams, 5 projects • Background: Mathematics with Computer Science, MS Applied Mathematics, MBA with Analytics and Information Systems • Movies, Travel 2
3. About Vanguard • Financial services industry - $5.2 T in global AUM* • Founded May 1, 1975 • IT has 10 subdivisions with ~283 IT teams which support different ‘business lines’* • Testing Center of Excellence is a sub department under the Chief Technology Office subdivision * As of December 31, 2018 3
4. Our Agile Journey 4 ~2015 ~2015 - 2018 ~2015 - present ~ 2008 Product Centered Waterfall to Agile Team Org Changes Shifting Left CI/CD, Products, LEAN, BV@SS
5. What does it mean to Shift Left? 5 ➢ Test Early, Test Often, Test Fast approach ➢ Quality Continuous Testing ➢ Single Source of Truth for System Behavior ➢ On Demand Prod like Environment ➢ Shift Right – Smoke Testing Production ➢ Control and Ownership of Data
6. Testing Center of Excellence is formed Testing Cycle Time Speed to Market Software Quality Software Testing Costs 6 Testing Center of Excellence is a group of people who are passionate about software engineering excellence and have been selected to help the enterprise change the way we build software. Shift Left & Test Transformation Performance Engineering Services Regression & Automation Services
7. Getting the enterprise started with Shift Left 7 Guidance Phase • Develop support teams across IT • “Give It a Try” 13 pilot teams • Measure and collect feedback, refine model, right skills and infrastructure • Finalize model, implementation and role out strategy
8. How has our software process evolved as a result of Shift Left? 8 • Culture Change • Role sharing/cross functional teams • Assembly Line vs. Product Ownership People • Requirement Sessions vs. Collaboration Sessions • Single source of truth • Build the right thing, build the thing right Process • New Tools for Automating • Infrastructure Changes Technology • Lower Level Testing • API Testing • Performance Testing Product Test Engineer & Automation Engineer Behavior Acceptance Test Test Driven Development
9. Change management & community support Training & Dojos Coaching Office Hours Shift Left Forums & Brown Bags Open Space Conferences Shift Left Champions & Practitioners Shift Left SharePoint Community & Site 9
10. Case Study: Big Bucks We are updating our existing legacy application with a new UI feature in AngularJS. We are currently using Selenium and want to switch to Protractor, a new test automation tool. Team Dynamics • 1 Project Manager • 1 Product Owner • 1 Business System Analyst • 1 Tester • 5 Midtier Developer • 1 UI Developer Application Code Statistics • 79,696 Lines of code • 349 Code violations • 65 Days of technical debt Perceived Problem Product Dynamics UI DB External Dependencies MT Big Bucks App Testing Dynamics Health of Tests • Line Coverage – 62% • Branch Coverage – 51.1% • Mutation Testing – 41% 6% 0% 94% Additional Context • Automation flakiness • Hand-offs between team members Total Automated Tests – 4149 • 250 End to End Tests– 125 failing • 0 Integration Tests • 3899 Unit Tests Regression Suite • 530 Manual Tests @ 4wks • 90 Defects
11. Check list for a shift left strategy ✓Desired outcomes ✓Measuring progress (metrics) 11 Testing Cycle Time Speed to Market Software Quality Software Testing Costs By 20% By 20%
12. Check list for a shift left strategy ✓Desired outcomes ✓Measuring progress (metrics) ✓Identifying your system under test ✓Code design/quality 12 UI DB External Dependencies MT Big Bucks App
13. Check list for a shift left strategy ✓Desired outcomes ✓Measuring progress (metrics) ✓Identifying your system under test ✓Code design/quality ✓Automation is an investment ✓Maintenance 13
14. Check list for a shift left strategy ✓Desired outcomes ✓Measuring progress (metrics) ✓Identifying your system under test ✓Code design/quality ✓Automation is an investment ✓Maintenance ✓Test design & techniques 14 Performance Usability
15. Check list for a shift left strategy ✓Desired outcomes ✓Measuring progress (metrics) ✓Identifying your system under test ✓Code design/quality ✓Automation is an investment ✓Maintenance ✓Test design & techniques ✓Development process 15
16. Check list for a shift left strategy ✓Desired outcomes ✓Measuring progress (metrics) ✓Identifying your system under test ✓Code design/quality ✓Automation is an investment ✓Maintenance ✓Test design & techniques ✓Development process ✓Shift Left tools in CI/CD 16 Planning Development Testing Release
17. Shift Left Tools for CI/CD 17
18. Case Study: Big Bucks 6 Months Later… • Increase Software Quality by 20% • Reduce Testing Cycle Time by 20% • Increase Speed to Market • Decrease Software testing Costs Application Code Statistics • Lines of code increased by 15% • Code violations decreased by 10% • Days of technical debt deceased by 10% Outcomes Shift Left Results Total Automated Tests – 5,438 • 105 End to End Tests • 210 Integration Tests • 5123 Unit Tests 2% 4% 94% Action Plan • Team Collaboration, no more hand offs • Analyzed & developed action plan for existing defects and technical debt • Incorporated pair programming to practice testable code techniques • Created more lower level automated tests • Analyzed existing UI tests until stable and reliable • Clear understanding of CI/CD to pick the right tool Health of Tests • Line Coverage – 75%, increase by 19% • Branch Coverage – 65%, increase by 24% • Mutation Testing – 50%, increase by 19% Regression Suite Manual Tests – 430, decrease by 21% Production Defects – 13, decrease by 85.55% Testing Cycle Time – 3 weeks, decrease by 29% Automated Testing Pyramid: Total Tests – 5,438
19. Why should you practice Shift Left? 19
20. Where are we now? 20 Shift left competencies are part of IT’s Business Value at Start Up Speed Fast IT Framework ➢ Total 283 teams ➢ NGA 170 teams ➢ In 2018, there were 2439 NGA deployments at a rate of approx. 2 deploys per month Overall defects have decreased from thousands to hundreds Outcome focused and caring more about the why and what, than the how
21. Questions? Thank You Allegra B. Cooper allegra_b_cooper@vanguard.com Technical Lead Chief Technology Office @ Vanguard