DevOpsPorto Meetup16: Testing travel guide to Continuous Delivery by André Carmo

Transcript

1. Testing travel guide to Continuous Delivery André Carmo

2. Who am I? André Carmo Started as software developer (J2EE)

   Worked at CRITICAL Software, Banco BPI and Farfetch Worked in software testing for 5+ years Test Automation Lead @ Farfetch Senior Infrastructure Engineer @ Farfetch SeleniumConf UK 2016

3. Quality vs speed Quality Speed of delivery

4. Phases of test automation - 1st • Unit tests •

   Manual tests • Very little test automation Problems you start experiencing • Releases are becoming difficult • More and more bugs

5. 2nd phase Backlog User Story Task - development Task -

   test automation User Story (...) User Story - Technical Debt - Test Automation Automation becomes part of your development culture

6. 3rd phase • Test suite becomes too big and unstable

   • Safari Mac, Chrome Windows, Safari iPhone, Safari iPad, Chrome Android • Team motivation goes down • Pipelines are difficult to maintain and environments are unstable • Continuous delivery… yeah right

7. 4th phase - let’s rethink automation

8. Rethink automation - Microservices • Independent deployable services and applications

   • Spread the pain across smaller applications • Easier, faster and cheaper to test and deploy By @alvaro_sanchez (twitter) as referred in http://www.wwwlicious.com/2016/05/03/servicestack- microservices-1/

9. Rethink automation - Framework for test automation • Different apps

   and service have common needs • Teams don’t have to deal with browser setup and teardown (cache, cookies) • Common page elements (dropdowns, textboxes, …) • Make automation easier • Framework that does magic and science

10. Rethink automation - Testing pyramid Unit Service/API UI Cost Complexity

    Execution time Flakiness Coverage Number of tests

11. Testing pyramid - Unit Tests Business logic and algorithms Each

    “unit” should be tested independently Use mocks and fakes • Easy to develop and maintain • Execute very fast • Miss integrations aspects

12. Testing pyramid - Service / API Tests Integration between multiple

    layers (services, SQLServer, Cassandra, ... ) Simulate business flows without touching UI • Connectivity • Fast enough • Stability • Not as fast and precise as unit tests • Difficult to implement if not using modular or microservices architecture

13. Testing pyramid - UI Tests Interaction with the browser (Chrome,

    Firefox, Safari, Edge, …) Test what the end user will see Cover happy paths • Allow end to end testing • Test the system as a whole: with all its dependencies • Expensive • Slow to run • Not so stable • Temptation to only have UI tests - because they actually allow to test everything

14. Rethink automation - Test levels Component Mock System Shift left

15. Rethink automation - Test levels Similar to unit tests Integration

    between units Code is not running in an application server Very fast and easy to maintain Component 1 Unit 1 Unit 2 Component

16. Example of tool: http:// www.mock-server.com/ Mock Isolation - test each

    service/app independently Simulate connectivity issues Simulate bad responses (HTTP 4xx, 5xx) Rethink automation - Test levels

17. Mock test types - API Contract tests Contract based testing

    Test if the contract established between a service and an application is still being respected Backwards compatible in new versions Example of tools: PACT tests Provider Consumer Consumer Consumer Contract

18. Rethink automation - Test levels Test the system with all

    its dependencies (UI, services, database, …) UI Tests (Selenium, Protractor, Robot Framework, …) API tests (SoapUI, MS Tests, JUnit, ….) System

19. Rethink automation - Cascade testing Isolation: App A tests itself

    Integration: App A tests integration with App B Isolation: App B tests itself Integration: App B tests integration with App C Contract: App B tests if it respects the contract with App A App A App B App C

20. Rethink automation - Test optimization Run full test suite during

    the night Run tests in parallel Have powerful and reliable test targets SeleniumGrid with horizontal scaling with Docker and Kubernetes Jenkins slaves created on demand with Kubernetes

21. Rethink automation - Scalable SeleniumGrid Why Docker instead of VMs?

    • When a VM gets “sick”, you call 112 / 911 so that some one fixes them • SeleniumGrid in VMs is hard to scale • When a container gets “sick”, you just dispose it and recreate a new one • Easy to scale using Kubernetes • ReplicationController assures SeleniumGrid is healthy • Bonus: for a scalable Jenkins with Kubernetes, check out the Jenkins X project

22. Rethink automation - Metrics Code coverage Areas of code with

    the most defects % automated tests Go live time Pipeline time Common test errors found Number of defects found

23. 5th phase - Fixing pipelines and environments Manual CI/CD configuration

    becomes difficult and error prone It’s difficult to keep track of changes More and more test instability due to not reliable environments Not enough CI/CD resources Solution based on 3 principles • Easy to use CI/CD Infrastructure as code (pipelines & environments as code) • Create, dispose, recreate • Autonomy for teams

24. Fixing pipelines • Have a standard for pipelines to work

    across your teams • Give teams autonomy to configure their own pipelines • Abstract pipelines DSL / scripting languages complexity • Provide pipeline code reuse - avoid copy-paste code • Pipeline code should be simple and standard

25. Fixing pipelines - build file Pipeline file: commands for test

    execution in the pipeline Pipeline configuration as code in source control Pipeline is built on the fly based on top level keywords and commands

26. Application + pipeline file • Demo

27. Test environments • Rely on Docker containers + Kubernetes •

    Your environments are code! • Dispose environment after using it and recreate it before next run

28. Fixing Pipelines + Environments = <3 Ultimate goals: Based on

    the build file, the pipeline creates the whole infrastructure on the fly, then disposes it, and then recreates it. And all of this is scripted by code • Some techs you can use: CI/CD (Jenkins), Configuration Management (SaltStack, Chef, Puppet), Cloud Infrastructure Orchestration (Terraform, SaltCloud), Cloud APIs (AWS, MS Azure, Google Cloud Platform), Docker, Kubernetes

29. Continuous Delivery “Continuous Delivery is a software development discipline where

    you build software in such a way that the software can be released to production at any time.” Martin Fowler “Safely and quickly in a sustainable way.” Jezz Humble

30. Are you ready for the Test Automation road trip? Big

    pain -> smaller pains Choose good testing tools - choose levels and types of tests wisely Optimize your tests and your infrastructure Metrics, metrics, metrics Provide product teams powerful, easy to use pipelines and stable environments Automation is not “one approach fits all”. Choose what's best for you. Thank you! André Carmo andretmcarmo@gmail.com github.com/atmcarmo @atmcarmo andrecarmo