DevOpsPorto Meetup16: Testing travel guide to Continuous Delivery by André Carmo

Talk delivered by André Carmo

Transcript

1. Testing travel guide to Continuous Delivery André Carmo
2. Who am I? André Carmo Started as software developer (J2EE) Worked at CRITICAL Software, Banco BPI and Farfetch Worked in software testing for 5+ years Test Automation Lead @ Farfetch Senior Infrastructure Engineer @ Farfetch SeleniumConf UK 2016
3. Quality vs speed Quality Speed of delivery
4. Phases of test automation - 1st • Unit tests • Manual tests • Very little test automation Problems you start experiencing • Releases are becoming difficult • More and more bugs
5. 2nd phase Backlog User Story Task - development Task - test automation User Story (...) User Story - Technical Debt - Test Automation Automation becomes part of your development culture
6. 3rd phase • Test suite becomes too big and unstable • Safari Mac, Chrome Windows, Safari iPhone, Safari iPad, Chrome Android • Team motivation goes down • Pipelines are difficult to maintain and environments are unstable • Continuous delivery… yeah right
7. 4th phase - let’s rethink automation
8. Rethink automation - Microservices • Independent deployable services and applications • Spread the pain across smaller applications • Easier, faster and cheaper to test and deploy By @alvaro_sanchez (twitter) as referred in http://www.wwwlicious.com/2016/05/03/servicestack- microservices-1/
9. Rethink automation - Framework for test automation • Different apps and service have common needs • Teams don’t have to deal with browser setup and teardown (cache, cookies) • Common page elements (dropdowns, textboxes, …) • Make automation easier • Framework that does magic and science
10. Rethink automation - Testing pyramid Unit Service/API UI Cost Complexity Execution time Flakiness Coverage Number of tests
11. Testing pyramid - Unit Tests Business logic and algorithms Each “unit” should be tested independently Use mocks and fakes • Easy to develop and maintain • Execute very fast • Miss integrations aspects
12. Testing pyramid - Service / API Tests Integration between multiple layers (services, SQLServer, Cassandra, ... ) Simulate business flows without touching UI • Connectivity • Fast enough • Stability • Not as fast and precise as unit tests • Difficult to implement if not using modular or microservices architecture
13. Testing pyramid - UI Tests Interaction with the browser (Chrome, Firefox, Safari, Edge, …) Test what the end user will see Cover happy paths • Allow end to end testing • Test the system as a whole: with all its dependencies • Expensive • Slow to run • Not so stable • Temptation to only have UI tests - because they actually allow to test everything
14. Rethink automation - Test levels Component Mock System Shift left
15. Rethink automation - Test levels Similar to unit tests Integration between units Code is not running in an application server Very fast and easy to maintain Component 1 Unit 1 Unit 2 Component
16. Example of tool: http:// www.mock-server.com/ Mock Isolation - test each service/app independently Simulate connectivity issues Simulate bad responses (HTTP 4xx, 5xx) Rethink automation - Test levels
17. Mock test types - API Contract tests Contract based testing Test if the contract established between a service and an application is still being respected Backwards compatible in new versions Example of tools: PACT tests Provider Consumer Consumer Consumer Contract
18. Rethink automation - Test levels Test the system with all its dependencies (UI, services, database, …) UI Tests (Selenium, Protractor, Robot Framework, …) API tests (SoapUI, MS Tests, JUnit, ….) System
19. Rethink automation - Cascade testing Isolation: App A tests itself Integration: App A tests integration with App B Isolation: App B tests itself Integration: App B tests integration with App C Contract: App B tests if it respects the contract with App A App A App B App C
20. Rethink automation - Test optimization Run full test suite during the night Run tests in parallel Have powerful and reliable test targets SeleniumGrid with horizontal scaling with Docker and Kubernetes Jenkins slaves created on demand with Kubernetes
21. Rethink automation - Scalable SeleniumGrid Why Docker instead of VMs? • When a VM gets “sick”, you call 112 / 911 so that some one fixes them • SeleniumGrid in VMs is hard to scale • When a container gets “sick”, you just dispose it and recreate a new one • Easy to scale using Kubernetes • ReplicationController assures SeleniumGrid is healthy • Bonus: for a scalable Jenkins with Kubernetes, check out the Jenkins X project
22. Rethink automation - Metrics Code coverage Areas of code with the most defects % automated tests Go live time Pipeline time Common test errors found Number of defects found
23. 5th phase - Fixing pipelines and environments Manual CI/CD configuration becomes difficult and error prone It’s difficult to keep track of changes More and more test instability due to not reliable environments Not enough CI/CD resources Solution based on 3 principles • Easy to use CI/CD Infrastructure as code (pipelines & environments as code) • Create, dispose, recreate • Autonomy for teams
24. Fixing pipelines • Have a standard for pipelines to work across your teams • Give teams autonomy to configure their own pipelines • Abstract pipelines DSL / scripting languages complexity • Provide pipeline code reuse - avoid copy-paste code • Pipeline code should be simple and standard
25. Fixing pipelines - build file Pipeline file: commands for test execution in the pipeline Pipeline configuration as code in source control Pipeline is built on the fly based on top level keywords and commands
26. Application + pipeline file • Demo
27. Test environments • Rely on Docker containers + Kubernetes • Your environments are code! • Dispose environment after using it and recreate it before next run
28. Fixing Pipelines + Environments = <3 Ultimate goals: Based on the build file, the pipeline creates the whole infrastructure on the fly, then disposes it, and then recreates it. And all of this is scripted by code • Some techs you can use: CI/CD (Jenkins), Configuration Management (SaltStack, Chef, Puppet), Cloud Infrastructure Orchestration (Terraform, SaltCloud), Cloud APIs (AWS, MS Azure, Google Cloud Platform), Docker, Kubernetes
29. Continuous Delivery “Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.” Martin Fowler “Safely and quickly in a sustainable way.” Jezz Humble
30. Are you ready for the Test Automation road trip? Big pain -> smaller pains Choose good testing tools - choose levels and types of tests wisely Optimize your tests and your infrastructure Metrics, metrics, metrics Provide product teams powerful, easy to use pipelines and stable environments Automation is not “one approach fits all”. Choose what's best for you. Thank you! André Carmo andretmcarmo@gmail.com github.com/atmcarmo @atmcarmo andrecarmo