Using Test Automation for Continuous Integration, Delivery and Quality

Using Test Automation for Continuous Integration, Delivery and Quality

by Brandon Weaver
DevOps Pro Vilnius 2016


DevOps Pro

June 01, 2016


  1. Using Test Automation for Continuous Integration, Delivery and Quality Brandon

    Weaver Virtustream Inc.
  2. Automation Drivers

  3. Automation is a work multiplier and quality check • Tests

    are a series of repeatable tasks that are time intensive when done manually • Automaton allows you to test more scenarios in less time • Automation through the entire stack saves more the further in the process you get • Automated tests help you to fail early
  4. Repetition • 7 tests – 5 Minutes to write –

    1 minute to execute • Manual execute 2 mins • 5 run return on effort
  5. Exponential Cost of Bugs

  6. Fail Early • Failing tests should stop the process and

    flow of your software • Failing tests are an opportunity for development and QA to communicate and work together • Failures after release are more urgent and disruptive. i.e. catching a failure early likely means a more thorough fix
  7. Automation Throughout the Process

  8. DevOps Process and Testing • Build automation enables test automation

    • Test automation execution • Automated Test result checks – Pass/Fail • Automated Environment setup • Automated Deployment Build Automation Test Run Automation
  9. DevOps Process and Testing • Promotion automation and testing •

    Automated tests for Deployment code/process • Automated tests for Environment code/process Automated Environment Build Automated Deployment
  10. Development Process and Testing • Unit Tests - Isolated tests

    at the code level – First chance to identify failures – Identifying bugs at the lowest level – Measurable coverage of the code • Integration Tests – Test integrations with Databases, Third party Systems – Test integrations with other teams Unit Tests Integration Tests
  11. QA/QE Process and Testing • End to End – Test

    the software and its’ dependencies API or back- end • UI Tests – User interface tests if applicable • Performance tests – Stress the system, try to break it E2E Test Automation UI Test Automation Performance Test Automation
  12. Security Process and Testing • Automate Scanning • Automate where

    possible and flag for failure Automated Security Scanning
  13. The Progression

  14. DevOps • Standardize on a build tool that supports running

    tests (Gradle, Maven, MSBuild) • Use a Continuous Integration tool (Jenkins, TeamCity, Bamboo, TFS) • Enforce standards for Failed and Skipped tests – 0 Failed tests < 10% skipped • Implement automated runs for all automated tests
  15. DevOps • Standardize packaging to make installation automation capable •

    Implement environment automation for testing • Automate deployment – zero touch • Report test coverage metrics DevOps
  16. Development • Test Failures take precedence • Start Small -

    All new code has corresponding unit tests written and reviewed • Every defect resolution gets a corresponding unit test where possible • When refactoring code produce unit tests • Drive toward 100% code coverage
  17. Development • Integration tests come after Unit tests – Still

    important but QA E2E tests that are being worked on in parallel should surface some of the same issues – Still important more coverage is better Development
  18. QA/QE • Implement automated tests for new functionality as it

    happens, where appropriate – Work with development starting with design – Write your test cases – Write your automated tests and revise when you have code
  19. QA/QE • Write tests for existing test cases as time

    allows • Identify the key tests to automate first – Areas with frequent bugs – Areas with the broadest exposure across many parts of the software QA
  20. Security • Identify and implement tools that can be automated

    if possible • Identify the thresholds or criteria to trigger automation failures Security
  21. Full stack

  22. CI Overview

  23. CI • Test your tests with tools like lint •

    Test your environments with whatever tools are available to make sure you built what you think you built. • Test your deployment every time • Automate your upgrades and test your upgrades
  24. CD Overview

  25. CD • Test your performance in a “real” environment •

    Test your security in a “real” environment • UAT in staging to get as close to real as possible, benchmark for releasing to “production”
  26. None