Who am I? André Carmo Started as software developer (J2EE) Worked at CRITICAL Software, Banco BPI and Farfetch Worked in software testing for 5+ years Test Automation Lead @ Farfetch Senior Infrastructure Engineer @ Farfetch SeleniumConf UK 2016
Phases of test automation - 1st • Unit tests • Manual tests • Very little test automation Problems you start experiencing • Releases are becoming difficult • More and more bugs
2nd phase Backlog User Story Task - development Task - test automation User Story (...) User Story - Technical Debt - Test Automation Automation becomes part of your development culture
3rd phase • Test suite becomes too big and unstable • Safari Mac, Chrome Windows, Safari iPhone, Safari iPad, Chrome Android • Team motivation goes down • Pipelines are difficult to maintain and environments are unstable • Continuous delivery… yeah right
Rethink automation - Microservices • Independent deployable services and applications • Spread the pain across smaller applications • Easier, faster and cheaper to test and deploy By @alvaro_sanchez (twitter) as referred in http://www.wwwlicious.com/2016/05/03/servicestack- microservices-1/
Rethink automation - Framework for test automation • Different apps and service have common needs • Teams don’t have to deal with browser setup and teardown (cache, cookies) • Common page elements (dropdowns, textboxes, …) • Make automation easier • Framework that does magic and science
Testing pyramid - Unit Tests Business logic and algorithms Each “unit” should be tested independently Use mocks and fakes • Easy to develop and maintain • Execute very fast • Miss integrations aspects
Testing pyramid - Service / API Tests Integration between multiple layers (services, SQLServer, Cassandra, ... ) Simulate business flows without touching UI • Connectivity • Fast enough • Stability • Not as fast and precise as unit tests • Difficult to implement if not using modular or microservices architecture
Testing pyramid - UI Tests Interaction with the browser (Chrome, Firefox, Safari, Edge, …) Test what the end user will see Cover happy paths • Allow end to end testing • Test the system as a whole: with all its dependencies • Expensive • Slow to run • Not so stable • Temptation to only have UI tests - because they actually allow to test everything
Rethink automation - Test levels Similar to unit tests Integration between units Code is not running in an application server Very fast and easy to maintain Component 1 Unit 1 Unit 2 Component
Example of tool: http:// www.mock-server.com/ Mock Isolation - test each service/app independently Simulate connectivity issues Simulate bad responses (HTTP 4xx, 5xx) Rethink automation - Test levels
Mock test types - API Contract tests Contract based testing Test if the contract established between a service and an application is still being respected Backwards compatible in new versions Example of tools: PACT tests Provider Consumer Consumer Consumer Contract
Rethink automation - Test levels Test the system with all its dependencies (UI, services, database, …) UI Tests (Selenium, Protractor, Robot Framework, …) API tests (SoapUI, MS Tests, JUnit, ….) System
Rethink automation - Cascade testing Isolation: App A tests itself Integration: App A tests integration with App B Isolation: App B tests itself Integration: App B tests integration with App C Contract: App B tests if it respects the contract with App A App A App B App C
Rethink automation - Test optimization Run full test suite during the night Run tests in parallel Have powerful and reliable test targets SeleniumGrid with horizontal scaling with Docker and Kubernetes Jenkins slaves created on demand with Kubernetes
Rethink automation - Scalable SeleniumGrid Why Docker instead of VMs? • When a VM gets “sick”, you call 112 / 911 so that some one fixes them • SeleniumGrid in VMs is hard to scale • When a container gets “sick”, you just dispose it and recreate a new one • Easy to scale using Kubernetes • ReplicationController assures SeleniumGrid is healthy • Bonus: for a scalable Jenkins with Kubernetes, check out the Jenkins X project
Rethink automation - Metrics Code coverage Areas of code with the most defects % automated tests Go live time Pipeline time Common test errors found Number of defects found
5th phase - Fixing pipelines and environments Manual CI/CD configuration becomes difficult and error prone It’s difficult to keep track of changes More and more test instability due to not reliable environments Not enough CI/CD resources Solution based on 3 principles • Easy to use CI/CD Infrastructure as code (pipelines & environments as code) • Create, dispose, recreate • Autonomy for teams
Fixing pipelines • Have a standard for pipelines to work across your teams • Give teams autonomy to configure their own pipelines • Abstract pipelines DSL / scripting languages complexity • Provide pipeline code reuse - avoid copy-paste code • Pipeline code should be simple and standard
Fixing pipelines - build file Pipeline file: commands for test execution in the pipeline Pipeline configuration as code in source control Pipeline is built on the fly based on top level keywords and commands
Test environments • Rely on Docker containers + Kubernetes • Your environments are code! • Dispose environment after using it and recreate it before next run
Fixing Pipelines + Environments = <3 Ultimate goals: Based on the build file, the pipeline creates the whole infrastructure on the fly, then disposes it, and then recreates it. And all of this is scripted by code • Some techs you can use: CI/CD (Jenkins), Configuration Management (SaltStack, Chef, Puppet), Cloud Infrastructure Orchestration (Terraform, SaltCloud), Cloud APIs (AWS, MS Azure, Google Cloud Platform), Docker, Kubernetes
Continuous Delivery “Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.” Martin Fowler “Safely and quickly in a sustainable way.” Jezz Humble
Are you ready for the Test Automation road trip? Big pain -> smaller pains Choose good testing tools - choose levels and types of tests wisely Optimize your tests and your infrastructure Metrics, metrics, metrics Provide product teams powerful, easy to use pipelines and stable environments Automation is not “one approach fits all”. Choose what's best for you. Thank you! André Carmo [email protected] github.com/atmcarmo @atmcarmo andrecarmo
devopsporto
hosimesi11
trycycle
mode86
okada_fuutass
mercari
skrb
line_developers
nozomiito
suzukahr
sasakitomohiro
soracom
yuyakakizaki08
jasonvnalue
carmenintech
zenorocha
sonatard
denniskardys
quramy
sugarenia
lara
erikaheidi
danielanewman
jnunemaker
matthewcrist