Using Test Automation for Continuous Integration, Delivery and Quality

by Brandon Weaver
DevOps Pro Vilnius 2016

Transcript

1. Using Test Automation for Continuous Integration, Delivery and Quality Brandon Weaver Virtustream Inc.
2. Automation Drivers
3. Automation is a work multiplier and quality check • Tests are a series of repeatable tasks that are time intensive when done manually • Automaton allows you to test more scenarios in less time • Automation through the entire stack saves more the further in the process you get • Automated tests help you to fail early
4. Repetition • 7 tests – 5 Minutes to write – 1 minute to execute • Manual execute 2 mins • 5 run return on effort
5. Exponential Cost of Bugs
6. Fail Early • Failing tests should stop the process and flow of your software • Failing tests are an opportunity for development and QA to communicate and work together • Failures after release are more urgent and disruptive. i.e. catching a failure early likely means a more thorough fix
7. Automation Throughout the Process
8. DevOps Process and Testing • Build automation enables test automation • Test automation execution • Automated Test result checks – Pass/Fail • Automated Environment setup • Automated Deployment Build Automation Test Run Automation
9. DevOps Process and Testing • Promotion automation and testing • Automated tests for Deployment code/process • Automated tests for Environment code/process Automated Environment Build Automated Deployment
10. Development Process and Testing • Unit Tests - Isolated tests at the code level – First chance to identify failures – Identifying bugs at the lowest level – Measurable coverage of the code • Integration Tests – Test integrations with Databases, Third party Systems – Test integrations with other teams Unit Tests Integration Tests
11. QA/QE Process and Testing • End to End – Test the software and its’ dependencies API or back- end • UI Tests – User interface tests if applicable • Performance tests – Stress the system, try to break it E2E Test Automation UI Test Automation Performance Test Automation
12. Security Process and Testing • Automate Scanning • Automate where possible and flag for failure Automated Security Scanning
13. The Progression
14. DevOps • Standardize on a build tool that supports running tests (Gradle, Maven, MSBuild) • Use a Continuous Integration tool (Jenkins, TeamCity, Bamboo, TFS) • Enforce standards for Failed and Skipped tests – 0 Failed tests < 10% skipped • Implement automated runs for all automated tests
15. DevOps • Standardize packaging to make installation automation capable • Implement environment automation for testing • Automate deployment – zero touch • Report test coverage metrics DevOps
16. Development • Test Failures take precedence • Start Small - All new code has corresponding unit tests written and reviewed • Every defect resolution gets a corresponding unit test where possible • When refactoring code produce unit tests • Drive toward 100% code coverage
17. Development • Integration tests come after Unit tests – Still important but QA E2E tests that are being worked on in parallel should surface some of the same issues – Still important more coverage is better Development
18. QA/QE • Implement automated tests for new functionality as it happens, where appropriate – Work with development starting with design – Write your test cases – Write your automated tests and revise when you have code
19. QA/QE • Write tests for existing test cases as time allows • Identify the key tests to automate first – Areas with frequent bugs – Areas with the broadest exposure across many parts of the software QA
20. Security • Identify and implement tools that can be automated if possible • Identify the thresholds or criteria to trigger automation failures Security
21. Full stack
22. CI Overview
23. CI • Test your tests with tools like lint • Test your environments with whatever tools are available to make sure you built what you think you built. • Test your deployment every time • Automate your upgrades and test your upgrades
24. CD Overview
25. CD • Test your performance in a “real” environment • Test your security in a “real” environment • UAT in staging to get as close to real as possible, benchmark for releasing to “production”