Testing the Test Machine - How a Test Framework Tests Itself

Writing tests has been always the least favourite thing for developers to do when working on an exciting project. Problems arise when the code base grows and the number of regressions increase over time. This is particularly problematic if the project you are working on is responsible for testing other projects.

As the JavaScript ecosystem changes, so do the tools. Linter as well as unit test frameworks have become much more sufficient and provide features such as snapshotting and stubbing capabilities that make writing tests a lot easier. However there are more aspects of how to test software aside from linting and unit testing. It is important to apply the right strategy to gain the confidence that a test can provide. This sometimes requires some creative thinking.

In this talk, Christian Bromann will speak about his experience managing unit and other types of tests at a large scale. As a maintainer of the WebdriverIO framework he has helped develop a test system that ensures the stability of various framework features from a code to an e2e level.

Transcript

1. How a Test Framework Tests Itself
2. Howdy! Christian Bromann 2 christian-bromann @bromann
3. Session Agenda 3 WebdriverIO Overview Testing Challenges 1 2 3 Test Setup
4. 1. WebdriverIO Overview
5. WebdriverIO • • • • •
6. DEMO
7. How does Browser Automation work? 7 Chromedriver Geckodriver IEDriver EdgeDriver SafariDriver Appium Selendroid WebDriverAgent HTTP (WebDriver) Selenium Grid const elem = $("#myElem") elem.click()
8. How does Browser Automation work? 8 const elem = $("#myElem") elem.click() Dauerwerbesendung
9. Lerna To allow seamless maintenance of project based plugins vs. community plugins TypeScript To allow to use latest JavaScript language features and type safety Jest As the all in one unit test framework that includes stubbing and snapshots EsLint The standard for static code analysis 9 Project Tooling Stack
10. 10 Project Structure
11. 2. Testing Challenges
12. High risk of making other lifes miserable
13. Chain of commands 13 const elem = $("#myElem") elem.click() 2 HTTP Requests: Various Socket Messages (CDP):
14. A simple click has to take a long way User calls click command in their test elem.click() Protocol lib (WebDriver) triggers an HTTP request to a server WebdriverIO connects element data with click action browser driver transforms request to click into native messages Browser actually emulates a click action and returns a result 14
15. The Perception of Failure
16. Support of Multiple Automation Protocols 16 const elem = $("#myElem") elem.click()
17. Focus on what is important
18. Two Execution Modes 18 ASYNC SYNC
19. Reporting • ◦ ◦ ◦ • • @wdio/reporter
20. 3. Test Setup
21. Project Tests 21 LINT $ npm run test:eslint DEPENDENCIES $ npm run test:depcheck TYPINGS $ npm run test:typings UNIT $ npm run test:coverage SMOKE $ npm run test:smoke E2E $ npm run test:e2e
22. Static Code Analysis • EsLint Prettier • • • Pro Tip! "husky": { "hooks": { "pre-commit": "git diff-index --name-only --diff-filter=d HEAD | grep -E \"(.*)\\.js$\" | xargs node_modules/eslint/bin/eslint.js -c .eslintrc.js", "pre-push": "npm run test:eslint" } }
23. Dependency Management in Mono Repositories > console.log(require.main.paths); [ '/some/path/to/the/project/node_modules', '/some/path/to/the/node_modules', '/some/path/to/node_modules', '/some/path/node_modules', '/some/node_modules', '/node_modules' ]
24. 24 Project Structure
25. 25 Project Structure . └── my-project/ ├── node_modules/ │ ├── ... │ └── lodash.clonedeep/ │ ├── package.json │ └── index.js └── packages/ ├── .../ └── my-sub-package/ ├── node_modules/ │ └── ... ├── src/ │ └── index.js ├── build/ ├── package.json └── ... import cloneDeep from 'lodash.cloneDeep'
26. const bp = (await Promise.all(packages.map(async (pkg) => { const packagePath = path.join(ROOT_DIR, 'packages', pkg) const sh = `npx depcheck ${packagePath}` + '--json ' + '--ignore-dirs build,tests' const res = await new Promise( (resolve, reject) => shell.exec(sh, EXEC_OPTIONS, (code, stdout, stderr) => stderr ? reject(stderr) : resolve(stdout) ) ) const result = JSON.parse(res) result.package = pkg result.packagePath = packagePath return result }))).filter((result) => Object.keys(result.missing).length) Dependency Check • • • • The Script
27. Typing Checks
28. const packages = { 'devtools': 'packages/devtools', 'webdriver': 'packages/webdriver', 'webdriverio': 'packages/webdriverio', '@wdio/sync': 'packages/wdio-sync', ... } for (const packageName of Object.keys(packages)) { const destination = path.join( __dirname, outDir, 'node_modules', packageName) const packageDir = packages[packageName] const destDir = destination.split(path.sep) .slice(0, -1).join(path.sep) if (!fs.existsSync(destDir)) { mkdir('-p', destDir) } ln('-s', path.join(ROOT, packageDir), destination) } Type Checks • • • • only necessary if The Script
29. Unit Testing + =
30. describe('getCookies', () => { let browser beforeAll(async () => { browser = await remote({ baseUrl: 'http://foobar.com', capabilities: { browserName: 'foobar' } }) }) it('should return all cookies', async () => { const cookies = await browser.getCookies() expect(got.mock.calls[1][1].method).toBe('GET') expect(got.mock.calls[1][0].pathname) .toBe('/session/foobar-123/cookie') expect(cookies).toEqual([ { name: 'cookie1', value: 'dummy-value-1' }, { name: 'cookie2', value: 'dummy-value-2' }, { name: 'cookie3', value: 'dummy-value-3' }, ]) }) it('should support passing a string', async () => { const cookies = await browser.getCookies('cookie1') expect(got.mock.calls[0][1].method).toBe('GET') expect(got.mock.calls[0][0].pathname) .toBe('/session/foobar-123/cookie') expect(cookies).toEqual([ { name: 'cookie1', value: 'dummy-value-1' } ]) }) }) So, how are commands tested now?
31. Good Unit Tests • • • • • • • • •
32. Good Unit Tests • Have more unit tests than anything else • • • • • • • • Have more unit tests than anything else
33. Good Unit Tests • Have more unit tests than anything else • • • • • • • • Have more unit tests than anything else
34. Good Unit Tests • Have more unit tests than anything else • • • • • • • • Have more integration tests than anything else?
35. Good Unit Tests • • Keep tests isolated, atomic and deterministic • • • • • • • Use Mocks, Stubs as Spies! . └── packages/ ├── devtools/ │ ├── ... │ └── tests/ │ └── __mocks__/ │ └── devtools.js ├── wdio-mocha-framework/ │ ├── ... │ └── tests/ │ └── __mocks__/ │ └── mocha.js ├── wdio-logger/ │ ├── ... │ └── tests/ │ └── __mocks__/ │ ├── chalk.js │ ├── loglevel.js │ └── @wdio/ │ └── logger.js └── webdriverio/ ├── ... └── tests/ └── __mocks__/ └── webdriverio.js
36. Good Unit Tests • • Keep tests isolated, atomic and deterministic • • • • • • • Use Mocks, Stubs as Spies! import puppeteer from 'puppeteer-core' import { launch as launchChromeBrowser } from 'chrome-launcher' import launch from '../src/launcher' jest.mock('../src/finder/firefox', () => ({ darwin: jest.fn().mockReturnValue(['/path/to/firefox']), linux: jest.fn().mockReturnValue(['/path/to/firefox']), win32: jest.fn().mockReturnValue(['/path/to/firefox']) })) jest.mock('../src/finder/edge', () => ({ darwin: jest.fn().mockReturnValue(['/path/to/edge']), linux: jest.fn().mockReturnValue(['/path/to/edge']), win32: jest.fn().mockReturnValue(['/path/to/edge']) })) beforeEach(() => { puppeteer.connect.mockClear() puppeteer.launch.mockClear() launchChromeBrowser.mockClear() })
37. Good Unit Tests • • • keep a test simple and short • • • • • • Separate Logic export default class BaseReporter { constructor(config, cid, caps) { this.config = config this.cid = cid this.caps = caps this.reporters = config.reporters.map((reporter) => { // complex reporter initiation code // ... }) } // ... } export default class BaseReporter { // ... init(config, cid, caps) { this.config = config this.cid = cid this.caps = caps // ... this.reporters = config.reporters.map( this.initReporters.bind(this)) } initReporters (reporter) { // complex reporter initiation code // ... } }
38. Use Snapshots test('should do nothing if no error is', () => { const adapter = adapterFactory() let params = { type: 'foobar' } let message = adapter.formatMessage(params) expect(message).toMatchSnapshot() }) test('should format an error message', () => { const adapter = adapterFactory() const params = { type: 'foobar', err: new Error('uups') } const message = adapter.formatMessage(params) // delete stack to avoid differences // in stack paths delete message.error.stack expect(message).toMatchSnapshot() }) // Jest Snapshot v1, https://goo.gl/fbAQLP exports[`formatMessage should do nothing if no error or params are given 1`] = ` Object { "type": "foobar", } `; exports[`formatMessage should format an error message 1`] = ` Object { "error": Object { "actual": undefined, "expected": undefined, "message": "uups", "type": "Error", }, "type": "foobar", } `;
39. Good Unit Tests • • • • Have meaningful and understood test names • • • • • Structured Test Suites describe('ConfigParser', () => { describe('getFilePaths', () => { it('should throw if not a string', () => { expect(() => ( ConfigParser.getFilePaths(123) )).toThrow() }) }) describe('addConfigFile', () => { ... describe('merge', () => { ... describe('addService', () => { ... describe('getCapabilities', () => { ... describe('getSpecs', () => { ... describe('getConfig', () => { ... })
40. Good Unit Tests • • • • • • true !== false, give assertions context • • • Structured Test Suites • Mocha with page objects expect(received).toBe(expected) // Object.is equality Expected: true Received: false it('Mocha with page objects', async () => { // ... expect(ejs.renderFile).toBeCalledWith( '/foo/bar/example.e2e.js', answers, expect.any(Function) ) expect(fs.ensureDirSync).toBeCalledTimes(4) const mock = fs.writeFileSync.mock.calls const pomPath = '/page/objects/model/page.js' const testPath = '/example.e2e.js' expect(mock[0][0].endsWith(pomPath)).toBe(true) expect(mock[1][0].endsWith(testPath)).toBe(true) })
41. Good Unit Tests • • • • • • true !== false, give assertions context • • • Custom Assertion messages Custom message Expect /file.js to end with /page/objects/model/page.js expect(received).toBe(expected) // Object.is equality Expected: true Received: false test('Mocha with page objects', async () => { const mock = fs.writeFileSync.mock.calls const pomPath = '/page/objects/model/page.js' const testPath = '/example.e2e.js' expect( mock[0][0].endsWith(pomPath), `Expect ${mock[0][0]} to end with ${pomPath}` ).toBe(true) expect( mock[1][0].endsWith(testPath), `Expect ${mock[1][0]} to end with ${testPath}` ).toBe(true) })
42. Good Unit Tests • • • • • • • Don’t “foo”(l) yourself and use realistic input data • • Don’t use foobar extensively it('should detect saucelabs user ...', () => { const caps = detectBackend({ hostname: 'foobar', port: 1234, protocol: 'foobar', path: 'foobar' }) expect(caps.hostname).toBe('foobar') expect(caps.port).toBe(1234) expect(caps.protocol).toBe('foobar') expect(caps.path).toBe('foobar') }) it('should detect saucelabs user ...', () => { const caps = detectBackend({ hostname: 'foobar.com', port: 1234, protocol: 'ftp', path: '/foo/bar' }) expect(caps.hostname).toBe('foobar.com') expect(caps.port).toBe(1234) expect(caps.protocol).toBe('ftp') expect(caps.path).toBe('/foo/bar') })
43. Good Unit Tests • • • • • • • • Test coverage to detect untested areas • Define Coverage Treshold { ... "jest": { "coverageThreshold": { "global": { "branches": 80, "functions": 80, "lines": 80, "statements": -10 } } } }
44. Good Unit Tests • • • • • • • • Test coverage to detect untested areas • How many tests do I need? ┌----------------------------------------┐ | % Stmts | % Branch | % Funcs | % Lines | |---------|----------|---------|---------| | 98.96 | 95.84 | 98.07 | 99.04 | └----------------------------------------┘ Test Suites: 205 passed, 205 total Tests: 2049 passed, 2049 total Snapshots: 120 passed, 120 total Time: 126.998 s Ran all test suites.
45. Good Unit Tests • • • • • • • • • Don’t catch errors, expect them Expect Errors it('acceptAlert', async () => { await browser.acceptAlert() try { await browser.getAlertText() } catch (e) { expect(e.message).toBe('no such alert') } }) it('acceptAlert', async () => { expect.assertions(1) await browser.acceptAlert() try { await browser.getAlertText() } catch (e) { expect(e.message).toBe('no such alert') } })
46. Smoke Testing
47. Scope of smoke tests • • • • • • • • • •
49. Mock WebDriver Requests { "/session": { "POST": { "command": "newSession", "description": "The New Session command creates...", "ref": "https://w3c.github.io/webdriver/#dfn-new-sessions", "parameters": [{ "name": "capabilities", "type": "object", "description": "a JSON object, the set of...", "required": true }], "returns": { "type": "Object", "name": "session", "description": "Object containing sessionId and..." } } }, "/session/:sessionId": { "DELETE": { "command": "deleteSession", "description": "The Delete Session command closes...", "ref": "https://w3c.github.io/webdriver/#dfn-delete-session", "parameters": [] } } … } •
50. Mock WebDriver Requests this.mock = new WebDriverMock() this.command = this.mock.command // define required responses this.command.status() .reply(200, { value: {} }) this.command.newSession() .reply(200, () => ({ value: { ...NEW_SESSION_PAYLOAD, sessionId: uuidv4() } })) this.command.deleteSession() .reply(200, deleteSession) this.command.getTitle() .reply(200, { value: 'Mock Page Title' }) this.command.getUrl() .reply(200, { value: 'https://mymockpage.com' }) this.command.getElementRect(ELEMENT_ID) .reply(200, { value: { width: 1, height: 2, x: 3, y: 4 } }) this.command.getElementRect(ELEMENT_ALT) .reply(200, { value: { width: 10, height: 20, x: 30, y: 40 } }) this.command.getLogTypes() .reply(200, { value: [] }) • • • WebDriverMock •
51. Mock WebDriver Requests • • • WebDriverMock • • • // wdio/webdriver-mock-service global.browser.addCommand('waitForDisplayedScenario', this.waitForDisplayedScenario.bind(this)) waitForDisplayedScenario() { this.nockReset() const elemResponse = { 'element-6066-11e4-a52e-4f735466cecf': ELEMENT_ID } this.command.findElement() .once().reply(200, { value: elemResponse }) this.command.isElementDisplayed(ELEMENT_ID) .times(4).reply(200, { value: false }) this.command.isElementDisplayed(ELEMENT_ID) .once().reply(200, { value: true }) } // test-waitForElement.js describe('Mocha smoke test', () => { it('should be able to wait for an element', () => { browser.waitForDisplayedScenario() assert($('elem').waitForDisplayed(), true) }) })
52. E2E Testing
53. Support of two automation interfaces WebDriver • • • Chrome DevTools • • •
54. Support of two automation interfaces WebDriver { "/session": { "POST": { "command": "newSession", "description": "The New Session command creates...", "ref": "https://w3c.github.io/webdriver...", "parameters": [{ "name": "capabilities", "type": "object", "description": "a JSON object, the set of...", "required": true }], "returns": { "type": "Object", "name": "session", "description": "Object containing sessionId and..." } } } Chrome DevTools export default async function newSession ({ capabilities }) { const browser = await launch(capabilities) const sessionId = uuidv4() const [browserName, browserVersion] = ( await browser.version() ).split('/') sessionMap.set(sessionId, browser) return { sessionId, capabilities: { browserName, browserVersion, platformName: os.platform(), platformVersion: os.release() } } }
55. Run Headless Chrome • • • describe('cookies', () => { beforeAll(async () => { await browser.navigateTo('http://guinea-pig.webdriver.io') }) it('should have no cookies in the beginning', async () => { expect(await browser.getAllCookies()).toEqual([]) }) it('should add a cookie', async () => { await browser.addCookie({ name: 'foobar', value: 42 }) await browser.navigateTo('http://guinea-pig.webdriver.io') expect(await browser.getAllCookies()).toHaveLength(1) await browser.navigateTo('https://google.com') const cookies = await browser.getAllCookies() const ourCookie = cookies.find((cookie) => ( cookie.name ==='foobar' && cookie.value === '42' ) expect(ourCookie).toBe(undefined) }) it('deleteCookie', async () => { await browser.navigateTo('http://guinea-pig.webdriver.io') const cookie = await browser.getNamedCookie('foobar') expect(cookie.value).toBe('42') await browser.deleteCookie('foobar') await browser.navigateTo('http://guinea-pig.webdriver.io') expect(await browser.getAllCookies()).toEqual([]) }) })
56. AFTER MUCH TESTING
57. Takeaways • • • • • •
58. 58 Thank you!