How a Test Framework Tests Itself 

Howdy! Christian Bromann 2 christian-bromann @bromann 

Session Agenda 3 WebdriverIO Overview Testing Challenges 1 2 3 Test Setup 

1. WebdriverIO Overview 

WebdriverIO ● ● ● ● ● 

DEMO 

How does Browser Automation work? 7 Chromedriver Geckodriver IEDriver EdgeDriver SafariDriver Appium Selendroid WebDriverAgent HTTP (WebDriver) Selenium Grid const elem = $("#myElem") elem.click() 

How does Browser Automation work? 8 const elem = $("#myElem") elem.click() Dauerwerbesendung 

Lerna To allow seamless maintenance of project based plugins vs. community plugins TypeScript To allow to use latest JavaScript language features and type safety Jest As the all in one unit test framework that includes stubbing and snapshots EsLint The standard for static code analysis 9 Project Tooling Stack 

10 Project Structure 

2. Testing Challenges 

High risk of making other lifes miserable 

Chain of commands 13 const elem = $("#myElem") elem.click() 2 HTTP Requests: Various Socket Messages (CDP): 

A simple click has to take a long way User calls click command in their test elem.click() Protocol lib (WebDriver) triggers an HTTP request to a server WebdriverIO connects element data with click action browser driver transforms request to click into native messages Browser actually emulates a click action and returns a result 14 

The Perception of Failure 

Support of Multiple Automation Protocols 16 const elem = $("#myElem") elem.click() 

Focus on what is important 

Two Execution Modes 18 ASYNC SYNC 

Reporting ● ○ ○ ○ ● ● @wdio/reporter 

3. Test Setup 

Project Tests 21 LINT $ npm run test:eslint DEPENDENCIES $ npm run test:depcheck TYPINGS $ npm run test:typings UNIT $ npm run test:coverage SMOKE $ npm run test:smoke E2E $ npm run test:e2e 

Static Code Analysis ● EsLint Prettier ● ● ● Pro Tip! "husky": { "hooks": { "pre-commit": "git diff-index --name-only --diff-filter=d HEAD | grep -E \"(.*)\\.js$\" | xargs node_modules/eslint/bin/eslint.js -c .eslintrc.js", "pre-push": "npm run test:eslint" } } 

Dependency Management in Mono Repositories > console.log(require.main.paths); [ '/some/path/to/the/project/node_modules', '/some/path/to/the/node_modules', '/some/path/to/node_modules', '/some/path/node_modules', '/some/node_modules', '/node_modules' ] 

24 Project Structure 

25 Project Structure . └── my-project/ ├── node_modules/ │ ├── ... │ └── lodash.clonedeep/ │ ├── package.json │ └── index.js └── packages/ ├── .../ └── my-sub-package/ ├── node_modules/ │ └── ... ├── src/ │ └── index.js ├── build/ ├── package.json └── ... import cloneDeep from 'lodash.cloneDeep' 

const bp = (await Promise.all(packages.map(async (pkg) => { const packagePath = path.join(ROOT_DIR, 'packages', pkg) const sh = `npx depcheck ${packagePath}` + '--json ' + '--ignore-dirs build,tests' const res = await new Promise( (resolve, reject) => shell.exec(sh, EXEC_OPTIONS, (code, stdout, stderr) => stderr ? reject(stderr) : resolve(stdout) ) ) const result = JSON.parse(res) result.package = pkg result.packagePath = packagePath return result }))).filter((result) => Object.keys(result.missing).length) Dependency Check ● ● ● ● The Script 

Typing Checks 

const packages = { 'devtools': 'packages/devtools', 'webdriver': 'packages/webdriver', 'webdriverio': 'packages/webdriverio', '@wdio/sync': 'packages/wdio-sync', ... } for (const packageName of Object.keys(packages)) { const destination = path.join( __dirname, outDir, 'node_modules', packageName) const packageDir = packages[packageName] const destDir = destination.split(path.sep) .slice(0, -1).join(path.sep) if (!fs.existsSync(destDir)) { mkdir('-p', destDir) } ln('-s', path.join(ROOT, packageDir), destination) } Type Checks ● ● ● ● only necessary if The Script 

Unit Testing + = 

describe('getCookies', () => { let browser beforeAll(async () => { browser = await remote({ baseUrl: 'http://foobar.com', capabilities: { browserName: 'foobar' } }) }) it('should return all cookies', async () => { const cookies = await browser.getCookies() expect(got.mock.calls[1][1].method).toBe('GET') expect(got.mock.calls[1][0].pathname) .toBe('/session/foobar-123/cookie') expect(cookies).toEqual([ { name: 'cookie1', value: 'dummy-value-1' }, { name: 'cookie2', value: 'dummy-value-2' }, { name: 'cookie3', value: 'dummy-value-3' }, ]) }) it('should support passing a string', async () => { const cookies = await browser.getCookies('cookie1') expect(got.mock.calls[0][1].method).toBe('GET') expect(got.mock.calls[0][0].pathname) .toBe('/session/foobar-123/cookie') expect(cookies).toEqual([ { name: 'cookie1', value: 'dummy-value-1' } ]) }) }) So, how are commands tested now? 

Good Unit Tests ● ● ● ● ● ● ● ● ● 

Good Unit Tests ● Have more unit tests than anything else ● ● ● ● ● ● ● ● Have more unit tests than anything else 

Good Unit Tests ● Have more unit tests than anything else ● ● ● ● ● ● ● ● Have more unit tests than anything else 

Good Unit Tests ● Have more unit tests than anything else ● ● ● ● ● ● ● ● Have more integration tests than anything else? 

Good Unit Tests ● ● Keep tests isolated, atomic and deterministic ● ● ● ● ● ● ● Use Mocks, Stubs as Spies! . └── packages/ ├── devtools/ │ ├── ... │ └── tests/ │ └── __mocks__/ │ └── devtools.js ├── wdio-mocha-framework/ │ ├── ... │ └── tests/ │ └── __mocks__/ │ └── mocha.js ├── wdio-logger/ │ ├── ... │ └── tests/ │ └── __mocks__/ │ ├── chalk.js │ ├── loglevel.js │ └── @wdio/ │ └── logger.js └── webdriverio/ ├── ... └── tests/ └── __mocks__/ └── webdriverio.js 

Good Unit Tests ● ● Keep tests isolated, atomic and deterministic ● ● ● ● ● ● ● Use Mocks, Stubs as Spies! import puppeteer from 'puppeteer-core' import { launch as launchChromeBrowser } from 'chrome-launcher' import launch from '../src/launcher' jest.mock('../src/finder/firefox', () => ({ darwin: jest.fn().mockReturnValue(['/path/to/firefox']), linux: jest.fn().mockReturnValue(['/path/to/firefox']), win32: jest.fn().mockReturnValue(['/path/to/firefox']) })) jest.mock('../src/finder/edge', () => ({ darwin: jest.fn().mockReturnValue(['/path/to/edge']), linux: jest.fn().mockReturnValue(['/path/to/edge']), win32: jest.fn().mockReturnValue(['/path/to/edge']) })) beforeEach(() => { puppeteer.connect.mockClear() puppeteer.launch.mockClear() launchChromeBrowser.mockClear() }) 

Good Unit Tests ● ● ● keep a test simple and short ● ● ● ● ● ● Separate Logic export default class BaseReporter { constructor(config, cid, caps) { this.config = config this.cid = cid this.caps = caps this.reporters = config.reporters.map((reporter) => { // complex reporter initiation code // ... }) } // ... } export default class BaseReporter { // ... init(config, cid, caps) { this.config = config this.cid = cid this.caps = caps // ... this.reporters = config.reporters.map( this.initReporters.bind(this)) } initReporters (reporter) { // complex reporter initiation code // ... } } 

Use Snapshots test('should do nothing if no error is', () => { const adapter = adapterFactory() let params = { type: 'foobar' } let message = adapter.formatMessage(params) expect(message).toMatchSnapshot() }) test('should format an error message', () => { const adapter = adapterFactory() const params = { type: 'foobar', err: new Error('uups') } const message = adapter.formatMessage(params) // delete stack to avoid differences // in stack paths delete message.error.stack expect(message).toMatchSnapshot() }) // Jest Snapshot v1, https://goo.gl/fbAQLP exports[`formatMessage should do nothing if no error or params are given 1`] = ` Object { "type": "foobar", } `; exports[`formatMessage should format an error message 1`] = ` Object { "error": Object { "actual": undefined, "expected": undefined, "message": "uups", "type": "Error", }, "type": "foobar", } `; 

Good Unit Tests ● ● ● ● Have meaningful and understood test names ● ● ● ● ● Structured Test Suites describe('ConfigParser', () => { describe('getFilePaths', () => { it('should throw if not a string', () => { expect(() => ( ConfigParser.getFilePaths(123) )).toThrow() }) }) describe('addConfigFile', () => { ... describe('merge', () => { ... describe('addService', () => { ... describe('getCapabilities', () => { ... describe('getSpecs', () => { ... describe('getConfig', () => { ... }) 

Good Unit Tests ● ● ● ● ● ● true !== false, give assertions context ● ● ● Structured Test Suites ● Mocha with page objects expect(received).toBe(expected) // Object.is equality Expected: true Received: false it('Mocha with page objects', async () => { // ... expect(ejs.renderFile).toBeCalledWith( '/foo/bar/example.e2e.js', answers, expect.any(Function) ) expect(fs.ensureDirSync).toBeCalledTimes(4) const mock = fs.writeFileSync.mock.calls const pomPath = '/page/objects/model/page.js' const testPath = '/example.e2e.js' expect(mock[0][0].endsWith(pomPath)).toBe(true) expect(mock[1][0].endsWith(testPath)).toBe(true) }) 

Good Unit Tests ● ● ● ● ● ● true !== false, give assertions context ● ● ● Custom Assertion messages Custom message Expect /file.js to end with /page/objects/model/page.js expect(received).toBe(expected) // Object.is equality Expected: true Received: false test('Mocha with page objects', async () => { const mock = fs.writeFileSync.mock.calls const pomPath = '/page/objects/model/page.js' const testPath = '/example.e2e.js' expect( mock[0][0].endsWith(pomPath), `Expect ${mock[0][0]} to end with ${pomPath}` ).toBe(true) expect( mock[1][0].endsWith(testPath), `Expect ${mock[1][0]} to end with ${testPath}` ).toBe(true) }) 

Good Unit Tests ● ● ● ● ● ● ● Don’t “foo”(l) yourself and use realistic input data ● ● Don’t use foobar extensively it('should detect saucelabs user ...', () => { const caps = detectBackend({ hostname: 'foobar', port: 1234, protocol: 'foobar', path: 'foobar' }) expect(caps.hostname).toBe('foobar') expect(caps.port).toBe(1234) expect(caps.protocol).toBe('foobar') expect(caps.path).toBe('foobar') }) it('should detect saucelabs user ...', () => { const caps = detectBackend({ hostname: 'foobar.com', port: 1234, protocol: 'ftp', path: '/foo/bar' }) expect(caps.hostname).toBe('foobar.com') expect(caps.port).toBe(1234) expect(caps.protocol).toBe('ftp') expect(caps.path).toBe('/foo/bar') }) 

Good Unit Tests ● ● ● ● ● ● ● ● Test coverage to detect untested areas ● Define Coverage Treshold { ... "jest": { "coverageThreshold": { "global": { "branches": 80, "functions": 80, "lines": 80, "statements": -10 } } } } 

Good Unit Tests ● ● ● ● ● ● ● ● Test coverage to detect untested areas ● How many tests do I need? ┌----------------------------------------┐ | % Stmts | % Branch | % Funcs | % Lines | |---------|----------|---------|---------| | 98.96 | 95.84 | 98.07 | 99.04 | └----------------------------------------┘ Test Suites: 205 passed, 205 total Tests: 2049 passed, 2049 total Snapshots: 120 passed, 120 total Time: 126.998 s Ran all test suites. 

Good Unit Tests ● ● ● ● ● ● ● ● ● Don’t catch errors, expect them Expect Errors it('acceptAlert', async () => { await browser.acceptAlert() try { await browser.getAlertText() } catch (e) { expect(e.message).toBe('no such alert') } }) it('acceptAlert', async () => { expect.assertions(1) await browser.acceptAlert() try { await browser.getAlertText() } catch (e) { expect(e.message).toBe('no such alert') } }) 

Smoke Testing 

Scope of smoke tests ● ● ● ● ● ● ● ● ● ● 

No content

Mock WebDriver Requests { "/session": { "POST": { "command": "newSession", "description": "The New Session command creates...", "ref": "https://w3c.github.io/webdriver/#dfn-new-sessions", "parameters": [{ "name": "capabilities", "type": "object", "description": "a JSON object, the set of...", "required": true }], "returns": { "type": "Object", "name": "session", "description": "Object containing sessionId and..." } } }, "/session/:sessionId": { "DELETE": { "command": "deleteSession", "description": "The Delete Session command closes...", "ref": "https://w3c.github.io/webdriver/#dfn-delete-session", "parameters": [] } } … } ● 

Mock WebDriver Requests this.mock = new WebDriverMock() this.command = this.mock.command // define required responses this.command.status() .reply(200, { value: {} }) this.command.newSession() .reply(200, () => ({ value: { ...NEW_SESSION_PAYLOAD, sessionId: uuidv4() } })) this.command.deleteSession() .reply(200, deleteSession) this.command.getTitle() .reply(200, { value: 'Mock Page Title' }) this.command.getUrl() .reply(200, { value: 'https://mymockpage.com' }) this.command.getElementRect(ELEMENT_ID) .reply(200, { value: { width: 1, height: 2, x: 3, y: 4 } }) this.command.getElementRect(ELEMENT_ALT) .reply(200, { value: { width: 10, height: 20, x: 30, y: 40 } }) this.command.getLogTypes() .reply(200, { value: [] }) ● ● ● WebDriverMock ● 

Mock WebDriver Requests ● ● ● WebDriverMock ● ● ● // wdio/webdriver-mock-service global.browser.addCommand('waitForDisplayedScenario', this.waitForDisplayedScenario.bind(this)) waitForDisplayedScenario() { this.nockReset() const elemResponse = { 'element-6066-11e4-a52e-4f735466cecf': ELEMENT_ID } this.command.findElement() .once().reply(200, { value: elemResponse }) this.command.isElementDisplayed(ELEMENT_ID) .times(4).reply(200, { value: false }) this.command.isElementDisplayed(ELEMENT_ID) .once().reply(200, { value: true }) } // test-waitForElement.js describe('Mocha smoke test', () => { it('should be able to wait for an element', () => { browser.waitForDisplayedScenario() assert($('elem').waitForDisplayed(), true) }) }) 

E2E Testing 

Support of two automation interfaces WebDriver ● ● ● Chrome DevTools ● ● ● 

Support of two automation interfaces WebDriver { "/session": { "POST": { "command": "newSession", "description": "The New Session command creates...", "ref": "https://w3c.github.io/webdriver...", "parameters": [{ "name": "capabilities", "type": "object", "description": "a JSON object, the set of...", "required": true }], "returns": { "type": "Object", "name": "session", "description": "Object containing sessionId and..." } } } Chrome DevTools export default async function newSession ({ capabilities }) { const browser = await launch(capabilities) const sessionId = uuidv4() const [browserName, browserVersion] = ( await browser.version() ).split('/') sessionMap.set(sessionId, browser) return { sessionId, capabilities: { browserName, browserVersion, platformName: os.platform(), platformVersion: os.release() } } } 

Run Headless Chrome ● ● ● describe('cookies', () => { beforeAll(async () => { await browser.navigateTo('http://guinea-pig.webdriver.io') }) it('should have no cookies in the beginning', async () => { expect(await browser.getAllCookies()).toEqual([]) }) it('should add a cookie', async () => { await browser.addCookie({ name: 'foobar', value: 42 }) await browser.navigateTo('http://guinea-pig.webdriver.io') expect(await browser.getAllCookies()).toHaveLength(1) await browser.navigateTo('https://google.com') const cookies = await browser.getAllCookies() const ourCookie = cookies.find((cookie) => ( cookie.name ==='foobar' && cookie.value === '42' ) expect(ourCookie).toBe(undefined) }) it('deleteCookie', async () => { await browser.navigateTo('http://guinea-pig.webdriver.io') const cookie = await browser.getNamedCookie('foobar') expect(cookie.value).toBe('42') await browser.deleteCookie('foobar') await browser.navigateTo('http://guinea-pig.webdriver.io') expect(await browser.getAllCookies()).toEqual([]) }) }) 

AFTER MUCH TESTING 

Takeaways ● ● ● ● ● ● 

58 Thank you!