Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DevOpsPorto Meetup16: Testing travel guide to Continuous Delivery by André Carmo

DevOpsPorto Meetup16: Testing travel guide to Continuous Delivery by André Carmo

Talk delivered by André Carmo

DevOpsPorto

May 16, 2018
Tweet

More Decks by DevOpsPorto

Other Decks in Technology

Transcript

  1. Testing travel guide to
    Continuous Delivery
    André Carmo

    View full-size slide

  2. Who am I?
    André Carmo
    Started as software developer (J2EE)
    Worked at CRITICAL Software,
    Banco BPI and Farfetch
    Worked in software testing for 5+
    years
    Test Automation Lead @ Farfetch
    Senior Infrastructure Engineer @
    Farfetch
    SeleniumConf UK 2016

    View full-size slide

  3. Quality vs speed
    Quality
    Speed of
    delivery

    View full-size slide

  4. Phases of test automation - 1st
    • Unit tests
    • Manual tests
    • Very little test automation
    Problems you start experiencing
    • Releases are becoming difficult
    • More and more bugs

    View full-size slide

  5. 2nd phase
    Backlog
    User Story
    Task - development
    Task - test automation
    User Story (...)
    User Story -
    Technical Debt -
    Test Automation
    Automation becomes part of your
    development culture

    View full-size slide

  6. 3rd phase
    • Test suite becomes too big and unstable
    • Safari Mac, Chrome Windows, Safari iPhone, Safari iPad,
    Chrome Android
    • Team motivation goes down
    • Pipelines are difficult to maintain and environments are unstable
    • Continuous delivery… yeah right

    View full-size slide

  7. 4th phase - let’s rethink automation

    View full-size slide

  8. Rethink automation - Microservices
    • Independent deployable services
    and applications
    • Spread the pain across smaller
    applications
    • Easier, faster and cheaper to test
    and deploy
    By @alvaro_sanchez (twitter) as referred in http://www.wwwlicious.com/2016/05/03/servicestack-
    microservices-1/

    View full-size slide

  9. Rethink automation - Framework for test automation
    • Different apps and service have common
    needs
    • Teams don’t have to deal with browser setup
    and teardown (cache, cookies)
    • Common page elements (dropdowns,
    textboxes, …)
    • Make automation easier
    • Framework that does magic and science

    View full-size slide

  10. Rethink automation - Testing pyramid
    Unit
    Service/API
    UI
    Cost
    Complexity
    Execution time
    Flakiness
    Coverage
    Number of tests

    View full-size slide

  11. Testing pyramid - Unit Tests
    Business logic and algorithms
    Each “unit” should be tested
    independently
    Use mocks and fakes
    • Easy to develop and maintain
    • Execute very fast
    • Miss integrations aspects

    View full-size slide

  12. Testing pyramid - Service / API Tests
    Integration between multiple layers (services, SQLServer, Cassandra, ... )
    Simulate business flows without touching UI
    • Connectivity
    • Fast enough
    • Stability
    • Not as fast and precise as unit tests
    • Difficult to implement if not using modular or microservices architecture

    View full-size slide

  13. Testing pyramid - UI Tests
    Interaction with the browser (Chrome, Firefox, Safari, Edge, …)
    Test what the end user will see
    Cover happy paths
    • Allow end to end testing
    • Test the system as a whole: with all its dependencies
    • Expensive
    • Slow to run
    • Not so stable
    • Temptation to only have UI tests - because they actually allow to test
    everything

    View full-size slide

  14. Rethink automation - Test levels
    Component Mock System
    Shift left

    View full-size slide

  15. Rethink automation - Test levels
    Similar to unit tests
    Integration between units
    Code is not running in an
    application server
    Very fast and easy to
    maintain
    Component 1
    Unit 1
    Unit 2
    Component

    View full-size slide

  16. Example of tool: http://
    www.mock-server.com/
    Mock
    Isolation - test each service/app independently
    Simulate connectivity issues
    Simulate bad responses (HTTP 4xx, 5xx)
    Rethink automation - Test levels

    View full-size slide

  17. Mock test types - API Contract tests
    Contract based testing
    Test if the contract established between a service and an application is still being respected
    Backwards compatible in new versions
    Example of tools: PACT tests
    Provider
    Consumer
    Consumer
    Consumer
    Contract

    View full-size slide

  18. Rethink automation - Test levels
    Test the system with all its dependencies (UI, services,
    database, …)
    UI Tests (Selenium, Protractor, Robot Framework, …)
    API tests (SoapUI, MS Tests, JUnit, ….)
    System

    View full-size slide

  19. Rethink automation - Cascade testing
    Isolation: App A tests itself
    Integration: App A tests integration with App B
    Isolation: App B tests itself
    Integration: App B tests integration with App C
    Contract: App B tests if it respects the contract with App A
    App A
    App B
    App C

    View full-size slide

  20. Rethink automation - Test optimization
    Run full test suite during the night
    Run tests in parallel
    Have powerful and reliable test targets
    SeleniumGrid with horizontal scaling with Docker
    and Kubernetes
    Jenkins slaves created on demand with
    Kubernetes

    View full-size slide

  21. Rethink automation - Scalable SeleniumGrid
    Why Docker instead of VMs?
    • When a VM gets “sick”, you call 112 / 911 so that some
    one fixes them
    • SeleniumGrid in VMs is hard to scale
    • When a container gets “sick”, you just dispose it and
    recreate a new one
    • Easy to scale using Kubernetes
    • ReplicationController assures SeleniumGrid is healthy
    • Bonus: for a scalable Jenkins with Kubernetes, check
    out the Jenkins X project

    View full-size slide

  22. Rethink automation - Metrics
    Code coverage
    Areas of code with the most defects
    % automated tests
    Go live time
    Pipeline time
    Common test errors found
    Number of defects found

    View full-size slide

  23. 5th phase - Fixing pipelines and environments
    Manual CI/CD configuration becomes difficult and error prone
    It’s difficult to keep track of changes
    More and more test instability due to not reliable environments
    Not enough CI/CD resources
    Solution based on 3 principles
    • Easy to use CI/CD Infrastructure as code (pipelines &
    environments as code)
    • Create, dispose, recreate
    • Autonomy for teams

    View full-size slide

  24. Fixing pipelines
    • Have a standard for pipelines to work across your teams
    • Give teams autonomy to configure their own pipelines
    • Abstract pipelines DSL / scripting languages complexity
    • Provide pipeline code reuse - avoid copy-paste code
    • Pipeline code should be simple and standard

    View full-size slide

  25. Fixing pipelines - build file
    Pipeline file: commands for test execution
    in the pipeline
    Pipeline configuration as code in source
    control
    Pipeline is built on the fly based on top
    level keywords and commands

    View full-size slide

  26. Application + pipeline file
    • Demo

    View full-size slide

  27. Test environments
    • Rely on Docker containers + Kubernetes
    • Your environments are code!
    • Dispose environment after using it and
    recreate it before next run

    View full-size slide

  28. Fixing Pipelines + Environments = <3
    Ultimate goals:
    Based on the build file, the pipeline creates the whole infrastructure on the fly,
    then disposes it, and then recreates it.
    And all of this is scripted by code
    • Some techs you can use: CI/CD (Jenkins), Configuration Management (SaltStack, Chef, Puppet), Cloud
    Infrastructure Orchestration (Terraform, SaltCloud), Cloud APIs (AWS, MS Azure, Google Cloud
    Platform), Docker, Kubernetes

    View full-size slide

  29. Continuous Delivery
    “Continuous Delivery is a software development discipline where you build
    software in such a way that the software can be released to production at any
    time.” Martin Fowler
    “Safely and quickly in a sustainable way.” Jezz Humble

    View full-size slide

  30. Are you ready for the Test Automation road trip?
    Big pain -> smaller pains
    Choose good testing tools - choose levels and types of tests wisely
    Optimize your tests and your infrastructure
    Metrics, metrics, metrics
    Provide product teams powerful, easy to use pipelines and stable environments
    Automation is not “one approach fits all”. Choose what's best for you.
    Thank you!
    André Carmo
    [email protected]
    github.com/atmcarmo
    @atmcarmo
    andrecarmo

    View full-size slide