Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LINE TECHPULSE 2020 - Improve Automated Acceptance Tests through Test Isolations

LINE TECHPULSE 2020 - Improve Automated Acceptance Tests through Test Isolations

Improve Automated Acceptance Tests through Test Isolations by Bryan Liu @ LINE TECHPULSE 2020 https://techpulse.line.me/

2102a6b8760bd6f57f672805723dd83a?s=128

line_developers_tw

December 18, 2020
Tweet

Transcript

  1. Bryan Liu / LINE TW Automation Engineer Improve Automated Acceptance

    Tests through Test Isolations
  2. Containerization and Cloud Native › Streamline delivery and deployment ›

    Simplify environments management › Fasten delivery cycle
  3. What’s Acceptance Test Production Acceptance Test Stage User Acceptance Test

    Commit Stage Capacity Testing Confidence ↑ Shift (Acceptance) Tests Left & Right
  4. What’s Hard on Automated Testing The Problems in Fast Delivery

    Cycle › Flaky tests due to timing issues, logical errors › Infrastructure, network, library versions › Unreliable test data Stability Ownership › Cross components, services and systems › Ownership relies on readability and maintainability Run slow › System and environment provision, cleanups takes time › Integrated and end-to-end tests run slow › Test cases increase quickly
  5. Why Test Isolation “「acceptance testing should be focused on providing

    a controllable environment … Integrating with real external systems removes our ability to do this.」” ~ Jaz Humble & David Farley: Continuous Delivery
  6. Why Test Isolation “「When the isolation of your acceptance test

    is good, another possibility to speed things up presents itself: running the tests in parallel .」” ~ Jaz Humble & David Farley: Continuous Delivery
  7. Three Levels of Test Isolation › NO dependency between tests

    and their results › NOT depends on the running order L2: Isolating test cases from each other L3: Isolating the system under test › Leveraging tools like: mock, stub, service virtualization tools › TestContainers: provision runtime dependencies with containers L1: Isolating test cases from themselves › Each test case is repeatable (run multiple times) › Run multiple times against the same env
  8. TestContext & Default Object const firstProduct = new Product('Test Product

    NO1') const secondProduct = new Product('Test Product NO2') let testContext = new TestContext() testContext.products.push(firstProduct, secondProduct) before(function() { testContext.products.forEach(prod => { cy.task('createDocument', {collectionName: ‘products’, …}) }) }) it('can checkout all items successfully', function() { // add to cart & do checkout cy.contains('div', testContext.products[0].title).click() cy.contains('div', testContext.products[1].title).click() cy.checkout(testContext.userName) )}
  9. Default Object class Product { constructor(name) { // An unique

    alias name this.title = `${name}_${new ShortUniqueId.randomUUID6}`, this.description = `Test description product ${this.title}! ...` this.imagePath = 'images/test_the_test.png', this.price = 22, // assign default values this.stock = 50 } } › Find 'functional entities’ › Shopping App -> create new product › Event App -> create new campaign › GitHub App -> create new account & repo › Give alias name for each entity
  10. TestContext class TestContext { constructor() { this.products = [], //

    to store test objects, status this.userName = 'defaultUser', // assign default values this.userEmail = 'linetw.qa5@corp.com', this.userPassword = 'test1234' } } › TextContext to store test intermediate status and data › TC create / teardown it's own entities › Easy parallel execution
  11. Show Test Intentions context('Test purchasing when short of inventory', ()

    => { let outOfStock = new Product('No Enough Inventory') outOfStock.stock = 0 // clearly shows what you want to TEST! let testContext = new TestContext() testContext.products.push(outOfStock) testContext.userName = 'ValidUser' // perhaps other variations testContext.userPassword = 'Wrong Password' it('add to cart should shows error message', function () { // add to cart ... cy.contains('div', testContext.products[0].title).click() cy.get('div#success').contains('short of inventory') }) })
  12. Summary › Achieve level I & II isolations › Show

    test intentions (important !) › Improve readability & maintainability › EASY parallel test execution !! TestContext & Default Object
  13. Productivity "Productivity is not everything, but in the long run,

    it is almost everything." ~ Paul Krugman
  14. Productivity is Everything What kills our productivity? › Clarity 條理分明

    (roles, audits, processes) › Accountability 專⼈專責 (someone to blame) › Measurement 衡量指標 (on personal / team?) TED Talks: How Too Many Rules at Work Keep You From Getting Things Done
  15. Productivity is Everything What should we do? › Get rid

    of rigid spec, details verification › Embrace fault tolerance, fuzzy, flexibility What DevOps tells us? › Deliver, recover and feedback fast › . . .
  16. API Test in Acceptance Test Accountability? APIs are way too

    important, so we assign a QA to verify that … › API is about interaction › Not just status code & response body › Contract can be verified in pre-commit phase (no need integrated env) › Test in production, E2E and integrated testing will verify the runtime integrity › Guarded by monitoring in production
  17. Monitoring & Testing in Production Shift-Right Testing

  18. API Integration - Contract Testing

  19. PACT Demo - Consumer provider.addInteraction({ state: 'single prod', uponReceiving: 'a

    request for JSON data', withRequest: { method: 'GET', path: '/prod/details', query: { t: 'LINE POP2' } }, willRespondWith: { status: 200, headers: {'Content-Type': 'application/json; charset=utf-8'}, body: EXPECTED_BODY } })
  20. PACT Demo - Consumer it('assert the JSON payload from the

    provider', done => { const shopClient = new ShopClient( ‘http://localhost:4000' // PACT runs stub server automatically ) const verificationPromise = shopClient.getOneProd('LINE POP2’) expect(verificationPromise) .to.eventually.have.property('title','LINE POP2’) expect(verificationPromise) .to.eventually.have.property('price') .notify(done) }) it('should validate the interactions and create a contract', () => { return provider.verify() }) })
  21. PACT Integration Flow

  22. TestContainers Increase Confidence of Acceptance Test Component / Service Level

    Acceptance Test › Easier to model behaviors (inputs & outputs) › With much higher confidence (vs. mock / in-memory DB)
  23. Demo - TestContainers const { GenericContainer } = require('testcontainers') describe('RedisCache',

    () => { before(async () => { container = await new GenericContainer(‘redis’) .withExposedPorts(6379).start() redisClient = redis.createClient(container.getMappedPort(6379)) }) after(async () => { await container.stop() }) it('should cache a value', async () => { await redisClient.set('key', ‘value’) // perform test scenario & assert result chai.expect(await redisClient.get('key')).eq('value') }) })
  24. Demo - TestContainers const { GenericContainer } = require('testcontainers') describe('RedisCache',

    () => { before(async () => { container = await new GenericContainer(‘redis’) .withExposedPorts(6379).start() redisClient = redis.createClient(container.getMappedPort(6379)) }) after(async () => { await container.stop() }) it('should cache a value', async () => { await redisClient.set('key', ‘value’) // perform test scenario & assert result chai.expect(await redisClient.get('key')).eq('value') }) })
  25. Demo - TestContainers const { GenericContainer } = require('testcontainers') describe('RedisCache',

    () => { before(async () => { container = await new GenericContainer(‘redis’) .withExposedPorts(6379).start() redisClient = redis.createClient(container.getMappedPort(6379)) }) after(async () => { await container.stop() }) it('should cache a value', async () => { await redisClient.set('key', ‘value’) // perform test scenario & assert result chai.expect(await redisClient.get('key')).eq('value') }) })
  26. TestContainers Runs Fast with More Confidence › Start a MongoDB

    container › Publish testing records in DB › Invoke the API call on Express › Assert responses › Tear down container › All in 52 ms with test coverage!
  27. Summary › Make test isolation part of test strategies ›

    Write more integrated testing › Test less with higher confidence › Leverage tools to achieve isolations › Shift-left & shift-right of tests Test with Isolations in Mind
  28. Thank you