From manual test shop to fully automated test coverage: A How-To session to speed up your journey

Scores of applications. Hundreds of production deployments per week. Test automation is a must. But how does a QA department go from a manual test shop to fully automated test coverage within a continuous integration and deployment environment?
These are our early steps towards full automation, walking through what worked (and didn’t work) for us.
Presented at Agile & Beyond 2017 (#aab17)

Transcript

1. From manual test shop to fully automated test coverage: A How-To session to speed up your journey JAYSHREE BHAKTA ITHAKA/JSTOR
2. The Journey Applications 1 app 268 apps Deployments 1 / month 234 / week Automation % 0% 100% QA per team 3 / team 1 / team Testing cycle 3-6 weeks 20 - 60 mins
3. How?
4. Vision and Goals: Define BUSINESS • Faster response to business needs • Enhance business risk mitigations • Product deficiency prevention • Reduction in overall cost of achieving quality TECHNICAL • Continuous delivery and testing • Efficient, scalable and sustainable products and technology • Immediate feedback loops • Reduction of overall cycle time
5. Vision and Goals: Share Engagement with partnership • Leadership • Teams Team ownership of quality • Value-add for everyone • Value-add for users
6. Vision and Goals: Milestones Iterative process Iterative process Plan Act Evaluate
7. Implementation
8. Implementation: Tool Selection TOOL SELECTION TOOL EVALUATION SKILL TRAINING CODING QA TEAM CULTURE TALENT/ HIRING LESSONS LEARNED • Acceptance Tests Driven Development / Behavior Driven Development • Language to write tests in business language • Easier language to learn • Open source • Lower cost • Dynamic and faster
9. Implementation: Tool Evaluation TOOL SELECTION TOOL EVALUATION SKILL TRAINING CODING QA TEAM CULTURE TALENT/ HIRING LESSONS LEARNED Cucumber/Ruby • Business language / Easy to learn GitHub • Code version control Jenkins • Run tests remotely • Schedule test runs • Results – Storage and Notifications QTP • Expensive • Not very efficient Domain Specific Language • Developed without QA engagement • No internal staff knowledge • Maintenance and upkeep • Hiring challenge
10. Implementation: Skills Training TEAM • Introduce the idea/plan • Team training • LeanDog • Smaller group workshops • Wiki creation • 1-1 training TOOL SELECTION TOOL EVALUATION SKILL TRAINING CODING QA TEAM CULTURE TALENT/ HIRING LESSONS LEARNED LEADS / TASK FORCE • Develop core framework • Learn the tools • Set up the pipeline • Coach the team • Adapt the training process
11. Implementation: Coding • Prioritized business workflows • Coding standards • Tagging • Test coverage • Code review standards TOOL SELECTION TOOL EVALUATION SKILL TRAINING CODING QA TEAM CULTURE TALENT/ HIRING LESSONS LEARNED • Sprint planning– Write cucumber • Write ruby code • Run and analyze - Jenkins • Run times • Maintainability • 0 test errors • Updates
12. Implementation: QA Team Culture DURING TRANSITION • Shared learning • Support • QA and ruby - Slack • Automation retrospectives TOOL SELECTION TOOL EVALUATION SKILL TRAINING CODING QA TEAM CULTURE TALENT/ HIRING LESSONS LEARNED ON-GOING • Cross team collaboration • QA and ruby - Slack • Shared learning • Support • On-going skill development
13. Implementation: Talent Hiring HIRING • Skills needed • Technical aptitude • Coding language knowledge • Culture fit • Leadership TOOL SELECTION TOOL EVALUATION SKILL TRAINING CODING QA TEAM CULTURE TALENT/ HIRING LESSONS LEARNED ON-BOARDING • Couple of weeks • Domain • Technical • Team • Dedicated time from leads
14. Implementation: Lessons Learned Ø Big user stories = longer to automate Ø Too much coverage = Slow run times Ø End to end automation = brittle Ø Custom built Domain Specific Language Ø Duplication of step definitions Ø Monitoring should be done using monitoring tools – not tests TOOL SELECTION TOOL EVALUATION SKILL TRAINING CODING QA TEAM CULTURE TALENT/ HIRING LESSONS LEARNED
15. Our Process Pipeline Sprint Planning Story: Pick Up Story: Preview Story: Code and Test Story: Validation Story: Deployment Sprint Retrospectives
16. SPRINT/STORY: PLANNING Product Detailing • Requirement • Design • Success criteria – First pass Technical Detailing • Code approach • Testing approach • Success criteria – Refine PLAN STORY PREVIEW TEST + CODE VALIDATE DEPLOY RETRO
17. STORY: PICK UP PLAN STORY PREVIEW TEST + CODE VALIDATE DEPLOY RETRO
18. STORY: PREVIEW In depth discussion of the specific story between • Dev • QA • UX PLAN STORY PREVIEW TEST + CODE VALIDATE DEPLOY RETRO
19. STORY: TESTING AND CODING PLAN STORY PREVIEW TEST + CODE VALIDATE DEPLOY RETRO
20. STORY: VALIDATION PLAN STORY PREVIEW TEST + CODE VALIDATE DEPLOY RETRO
21. STORY: DEPLOYMENT PLAN STORY PREVIEW TEST + CODE VALIDATE DEPLOY RETRO Feature deployment to Prod One story corresponds to one deployment.
22. SPRINT: RETROSPECTIVES PLAN STORY PREVIEW TEST + CODE VALIDATE DEPLOY RETRO
23. What did we learn?
24. Takeaways ü Quality strategy with partnership ü Select tools that fit your requirement ü Dedicated task force ü Set coding standards ü Develop team culture ü Iterative process
25. Questions? JAYSHREE BHAKTA QA Engineer at ITHAKA/JSTOR jayshree2901@gmail.com @jayshreebhakta