Using Test Automation for Continuous Integration, Delivery and Quality

Using Test Automation for Continuous Integration, Delivery and Quality

by Brandon Weaver
DevOps Pro Vilnius 2016

6d46284ef16436cb154adf4963e236f0?s=128

DevOps Pro

June 01, 2016
Tweet

Transcript

  1. Using Test Automation for Continuous Integration, Delivery and Quality Brandon

    Weaver Virtustream Inc.
  2. Automation Drivers

  3. Automation is a work multiplier and quality check • Tests

    are a series of repeatable tasks that are time intensive when done manually • Automaton allows you to test more scenarios in less time • Automation through the entire stack saves more the further in the process you get • Automated tests help you to fail early
  4. Repetition • 7 tests – 5 Minutes to write –

    1 minute to execute • Manual execute 2 mins • 5 run return on effort
  5. Exponential Cost of Bugs

  6. Fail Early • Failing tests should stop the process and

    flow of your software • Failing tests are an opportunity for development and QA to communicate and work together • Failures after release are more urgent and disruptive. i.e. catching a failure early likely means a more thorough fix
  7. Automation Throughout the Process

  8. DevOps Process and Testing • Build automation enables test automation

    • Test automation execution • Automated Test result checks – Pass/Fail • Automated Environment setup • Automated Deployment Build Automation Test Run Automation
  9. DevOps Process and Testing • Promotion automation and testing •

    Automated tests for Deployment code/process • Automated tests for Environment code/process Automated Environment Build Automated Deployment
  10. Development Process and Testing • Unit Tests - Isolated tests

    at the code level – First chance to identify failures – Identifying bugs at the lowest level – Measurable coverage of the code • Integration Tests – Test integrations with Databases, Third party Systems – Test integrations with other teams Unit Tests Integration Tests
  11. QA/QE Process and Testing • End to End – Test

    the software and its’ dependencies API or back- end • UI Tests – User interface tests if applicable • Performance tests – Stress the system, try to break it E2E Test Automation UI Test Automation Performance Test Automation
  12. Security Process and Testing • Automate Scanning • Automate where

    possible and flag for failure Automated Security Scanning
  13. The Progression

  14. DevOps • Standardize on a build tool that supports running

    tests (Gradle, Maven, MSBuild) • Use a Continuous Integration tool (Jenkins, TeamCity, Bamboo, TFS) • Enforce standards for Failed and Skipped tests – 0 Failed tests < 10% skipped • Implement automated runs for all automated tests
  15. DevOps • Standardize packaging to make installation automation capable •

    Implement environment automation for testing • Automate deployment – zero touch • Report test coverage metrics DevOps
  16. Development • Test Failures take precedence • Start Small -

    All new code has corresponding unit tests written and reviewed • Every defect resolution gets a corresponding unit test where possible • When refactoring code produce unit tests • Drive toward 100% code coverage
  17. Development • Integration tests come after Unit tests – Still

    important but QA E2E tests that are being worked on in parallel should surface some of the same issues – Still important more coverage is better Development
  18. QA/QE • Implement automated tests for new functionality as it

    happens, where appropriate – Work with development starting with design – Write your test cases – Write your automated tests and revise when you have code
  19. QA/QE • Write tests for existing test cases as time

    allows • Identify the key tests to automate first – Areas with frequent bugs – Areas with the broadest exposure across many parts of the software QA
  20. Security • Identify and implement tools that can be automated

    if possible • Identify the thresholds or criteria to trigger automation failures Security
  21. Full stack

  22. CI Overview

  23. CI • Test your tests with tools like lint •

    Test your environments with whatever tools are available to make sure you built what you think you built. • Test your deployment every time • Automate your upgrades and test your upgrades
  24. CD Overview

  25. CD • Test your performance in a “real” environment •

    Test your security in a “real” environment • UAT in staging to get as close to real as possible, benchmark for releasing to “production”
  26. None