Improve Automated Acceptance Tests through Test Isolations

雲端與容器的技術的確讓快速迭代更容易達成，但在持續集成裡的自動化驗收測試要如何一起來搭配呢？讓我們從測試隔離的角度來看，如何讓測試向左及向右移來讓自動化測試能更穩定且更有效率！

Transcript

2. Bryan Liu / LINE TW Automation Engineer Improve Automated Acceptance Tests through Test Isolations
3. Containerization and Cloud Native › Streamline delivery and deployment › Simplify environments management › Fasten delivery cycle
4. What’s Acceptance Test Production Acceptance Test Stage User Acceptance Test Commit Stage Capacity Testing Confidence ↑ Shift (Acceptance) Tests Left & Right
5. What’s Hard on Automated Testing The Problems in Fast Delivery Cycle › Flaky tests due to timing issues, logical errors › Infrastructure, network, library versions › Unreliable test data Stability Ownership › Cross components, services and systems › Ownership relies on readability and maintainability Run slow › System and environment provision, cleanups takes time › Integrated and end-to-end tests run slow › Test cases increase quickly
6. Why Test Isolation “「acceptance testing should be focused on providing a controllable environment … Integrating with real external systems removes our ability to do this.」” ~ Jaz Humble & David Farley: Continuous Delivery
7. Why Test Isolation “「When the isolation of your acceptance test is good, another possibility to speed things up presents itself: running the tests in parallel .」” ~ Jaz Humble & David Farley: Continuous Delivery
8. Three Levels of Test Isolation › NO dependency between tests and their results › NOT depends on the running order L2: Isolating test cases from each other L3: Isolating the system under test › Leveraging tools like: mock, stub, service virtualization tools › TestContainers: provision runtime dependencies with containers L1: Isolating test cases from themselves › Each test case is repeatable (run multiple times) › Run multiple times against the same env
9. TestContext & Default Object const firstProduct = new Product('Test Product NO1') const secondProduct = new Product('Test Product NO2') let testContext = new TestContext() testContext.products.push(firstProduct, secondProduct) before(function() { testContext.products.forEach(prod => { cy.task('createDocument', {collectionName: ‘products’, …}) }) }) it('can checkout all items successfully', function() { // add to cart & do checkout cy.contains('div', testContext.products[0].title).click() cy.contains('div', testContext.products[1].title).click() cy.checkout(testContext.userName) )}
10. Default Object class Product { constructor(name) { // An unique alias name this.title = `${name}_${new ShortUniqueId.randomUUID6}`, this.description = `Test description product ${this.title}! ...` this.imagePath = 'images/test_the_test.png', this.price = 22, // assign default values this.stock = 50 } } › Find 'functional entities’ › Shopping App -> create new product › Event App -> create new campaign › GitHub App -> create new account & repo › Give alias name for each entity
11. TestContext class TestContext { constructor() { this.products = [], // to store test objects, status this.userName = 'defaultUser', // assign default values this.userEmail = 'linetw.qa5@corp.com', this.userPassword = 'test1234' } } › TextContext to store test intermediate status and data › TC create / teardown it's own entities › Easy parallel execution
12. Show Test Intentions context('Test purchasing when short of inventory', () => { let outOfStock = new Product('No Enough Inventory') outOfStock.stock = 0 // clearly shows what you want to TEST! let testContext = new TestContext() testContext.products.push(outOfStock) testContext.userName = 'ValidUser' // perhaps other variations testContext.userPassword = 'Wrong Password' it('add to cart should shows error message', function () { // add to cart ... cy.contains('div', testContext.products[0].title).click() cy.get('div#success').contains('short of inventory') }) })
13. Summary › Achieve level I & II isolations › Show test intentions (important !) › Improve readability & maintainability › EASY parallel test execution !! TestContext & Default Object
14. Productivity "Productivity is not everything, but in the long run, it is almost everything." ~ Paul Krugman
15. Productivity is Everything What kills our productivity? › Clarity 條理分明 (roles, audits, processes) › Accountability 專⼈專責 (someone to blame) › Measurement 衡量指標 (on personal / team?) TED Talks: How Too Many Rules at Work Keep You From Getting Things Done
16. Productivity is Everything What should we do? › Get rid of rigid spec, details verification › Embrace fault tolerance, fuzzy, flexibility What DevOps tells us? › Deliver, recover and feedback fast › . . .
17. API Test in Acceptance Test Accountability? APIs are way too important, so we assign a QA to verify that … › API is about interaction › Not just status code & response body › Contract can be verified in pre-commit phase (no need integrated env) › Test in production, E2E and integrated testing will verify the runtime integrity › Guarded by monitoring in production
18. Monitoring & Testing in Production Shift-Right Testing
19. API Integration - Contract Testing
20. PACT Demo - Consumer provider.addInteraction({ state: 'single prod', uponReceiving: 'a request for JSON data', withRequest: { method: 'GET', path: '/prod/details', query: { t: 'LINE POP2' } }, willRespondWith: { status: 200, headers: {'Content-Type': 'application/json; charset=utf-8'}, body: EXPECTED_BODY } })
21. PACT Demo - Consumer it('assert the JSON payload from the provider', done => { const shopClient = new ShopClient( ‘http://localhost:4000' // PACT runs stub server automatically ) const verificationPromise = shopClient.getOneProd('LINE POP2’) expect(verificationPromise) .to.eventually.have.property('title','LINE POP2’) expect(verificationPromise) .to.eventually.have.property('price') .notify(done) }) it('should validate the interactions and create a contract', () => { return provider.verify() }) })
22. PACT Integration Flow
23. TestContainers Increase Confidence of Acceptance Test Component / Service Level Acceptance Test › Easier to model behaviors (inputs & outputs) › With much higher confidence (vs. mock / in-memory DB)
24. Demo - TestContainers const { GenericContainer } = require('testcontainers') describe('RedisCache', () => { before(async () => { container = await new GenericContainer(‘redis’) .withExposedPorts(6379).start() redisClient = redis.createClient(container.getMappedPort(6379)) }) after(async () => { await container.stop() }) it('should cache a value', async () => { await redisClient.set('key', ‘value’) // perform test scenario & assert result chai.expect(await redisClient.get('key')).eq('value') }) })
25. Demo - TestContainers const { GenericContainer } = require('testcontainers') describe('RedisCache', () => { before(async () => { container = await new GenericContainer(‘redis’) .withExposedPorts(6379).start() redisClient = redis.createClient(container.getMappedPort(6379)) }) after(async () => { await container.stop() }) it('should cache a value', async () => { await redisClient.set('key', ‘value’) // perform test scenario & assert result chai.expect(await redisClient.get('key')).eq('value') }) })
26. Demo - TestContainers const { GenericContainer } = require('testcontainers') describe('RedisCache', () => { before(async () => { container = await new GenericContainer(‘redis’) .withExposedPorts(6379).start() redisClient = redis.createClient(container.getMappedPort(6379)) }) after(async () => { await container.stop() }) it('should cache a value', async () => { await redisClient.set('key', ‘value’) // perform test scenario & assert result chai.expect(await redisClient.get('key')).eq('value') }) })
27. TestContainers Runs Fast with More Confidence › Start a MongoDB container › Publish testing records in DB › Invoke the API call on Express › Assert responses › Tear down container › All in 52 ms with test coverage!
28. Summary › Make test isolation part of test strategies › Write more integrated testing › Test less with higher confidence › Leverage tools to achieve isolations › Shift-left & shift-right of tests Test with Isolations in Mind
29. Thank you