Browser Automation with WebDriver #BuzzJS Workshop

Slide 1

Slide 1 text

1 Browser Test Automation (in 2018) @ColinBendell CTO Office, Cloudinary

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Agenda Test Methodologies WebDriver.io Mocha tests How Browsers Work & Browser APIs Web Platform Test Mobile & Device Emulation Accessibility Operational: DNS, Paralyzing, CI Real User Testing 9:00 AM Welcome | 10:30 AM Break | 12:00 AM Lunch | 2:30 PM Break | 4:00 PM Wrap-up & Q&A

Slide 4

Slide 4 text

Cloudinary's Two Problems Felix Zilber New Digital-Asset-Management Front-End Automating CDN Changes

Slide 5

Slide 5 text

5 Why Do We Test?

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

10 Testing: Prepare for the Unexpected

Slide 11

Slide 11 text

“I just want to buy my Mom’s birthday present.” “I just want to know if it will rain today.” “I just want to share pictures of my vacation”

Slide 12

Slide 12 text

Testing === Meeting User Expectations

Slide 13

Slide 13 text

Testing Models

Slide 14

Slide 14 text

Kinds of Testing https://stackoverflow.com/questions/2741832/unit-tests-vs-functional-tests

Slide 15

Slide 15 text

Functional v. Non-Functional

Slide 16

Slide 16 text

Unit Testing • Smallest testable component • Usually simple inputs and outputs • Moch objects v. skeleton dev services

Slide 17

Slide 17 text

Unit Testing const expect = require("chai").expect; const converter = require("../app/converter"); describe("Color Code Converter", function() { describe("RGB to Hex conversion", function() { it("converts the basic colors", function() { var redHex = converter.rgbToHex(255, 0, 0); var greenHex = converter.rgbToHex(0, 255, 0); var blueHex = converter.rgbToHex(0, 0, 255); expect(redHex).to.equal("ff0000"); expect(greenHex).to.equal("00ff00"); expect(blueHex).to.equal("0000ff"); }); }); });

Slide 18

Slide 18 text

Unit Testing • Blurred line to integration tests • Encourages small component logic v. monoliths • Large volume of Tests

Slide 19

Slide 19 text

Integration Tests

Slide 20

Slide 20 text

Integration Tests describe("Color Code Converter API", function() { describe("RGB to Hex conversion", function() { let url = "http://localhost:3000/rgbToHex?red=255&green=255&blue=255"; it("returns status 200", function() {}); it("returns the color in hex", function() {}); }); describe("Hex to RGB conversion", function() { let url = "http://localhost:3000/hexToRgb?hex=00ff00"; it("returns status 200", function() {}); it("returns the color in RGB", function() {}); }); });

Slide 21

Slide 21 text

System Tests • Does it match design? • Meet expectations of design • Often focused on workflows • Volume, load, stress, security • Usability, Accessibility

Slide 22

Slide 22 text

System Tests

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Acceptance Tests • Meet business objectives • End-to-End • Can the end user accomplish what we wanted it to do? • Alpha/Beta Feedback cycle

Slide 25

Slide 25 text

Kinds of Testing

Slide 26

Slide 26 text

“We develop in the Zoo, But release in the Jungle” -Melanie Cey

Slide 27

Slide 27 text

Kinds of Testing https://stackoverflow.com/questions/2741832/unit-tests-vs-functional-tests

Slide 28

Slide 28 text

Why do we need browser testing • Functional User experience • Workflows • Design & Layout • Preserve Presentation • Above the fold, responsive web • Mobile experiences v. Watch v. ?? • Accessibility • Screen readers • audits • Performance measurement • Performance budgets • Request flows

Slide 29

Slide 29 text

Browser Eco System

Slide 30

Slide 30 text

2007 2008 2009 2010 Present IE7 iPhone Firefox 2 IE7 iPhone 3G Firefox 3 Android Chrome 1 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 iPad ??

Slide 31

Slide 31 text

2012: 3,977 Different Android Devices Source: OpenSignal 2012: 3,977 Different Android Devices

Slide 32

Slide 32 text

Source: OpenSignal 2014: 18.7k Different Android Devices

Slide 33

Slide 33 text

Source: OpenSignal 2015: 24.0k Different Android Devices

Slide 34

Slide 34 text

How many browsers in a typical Thanksgiving weekend?

Slide 35

Slide 35 text

How many browsers in a typical Thanksgiving weekend? 572

Slide 36

Slide 36 text

2007 2008 2009 2010 Present IE7 iPhone Firefox 2 IE7 iPhone 3G Firefox 3 Android Chrome 1 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 iPad 572 iOS Safari MacOS Safari Android Chrome Desktop Chrome Samsung IB Opera …

Slide 37

Slide 37 text

“…25% of new Android phones have only 512MB of RAM.” Jen Fitzpatrick VP of product management for Google Maps

Slide 38

Slide 38 text

Browser Based Testing

Slide 39

Slide 39 text

How do we test the Browser? • Checklists • Support tickets • Screen capture • Pixel comparison • Programmatic control • RUM Beacons

Slide 40

Slide 40 text

#1 reason checklists fail to catch browser regression: Humans

Slide 41

Slide 41 text

open -a Safari https://cloudinary.com killall Safari screencapture -x testoutput.png compare -identify -metric MAE baseline.png testoutput.png null >> same1.png[0] PNG 640x400 640x400+0+0 8-bit DirectClass 1.64KB 0.010u 0:00.019 >> same2.jpg[0] JPEG 640x400 640x400+0+0 8-bit DirectClass 3.65KB 0.000u 0:00.009 >> 0.196766 (3.00245e-06)

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

How do we test the Browser? • Checklists • Support tickets • Screen capture • Pixel comparison • Programmatic control • RUM Beacons

Slide 47

Slide 47 text

OS API application control • Access the window model directly • Simulate Keyboard strokes and mouse movements • Interact with any application like a human • Screen Readers use this approach to supersede the OS accessibility features

Slide 48

Slide 48 text

Programmatic Control

Slide 49

Slide 49 text

How do we test the Browser? • Checklists • Support tickets • Screen capture • Pixel comparison • Programmatic control • RUM Beacons

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

Selenium Evolution •Selenium •Selenium Core •Selenium RC Selenium 1 •Selenium WebDriver Selenium 2 •WebDriver Selenium 3 Gecko Driver (+Marionette) Chrome Driver EdgeDriver Apple Safari Driver Firefox Driver Safari Driver … https://www.seleniumhq.org/download/

Slide 56

Slide 56 text

Selenium 3 (W3C WebDriver) Selenium Bindings Java Ruby C# Python JavaScript ? Selenium Grid ChromeDriver GeckoDriver SafariDriver EdgeDriver OperaDriver Browsers Chrome Firefox Safari Edge Opera JSON Wire Protocol WebDriver Wire Protocol HTTP HTTP

Slide 57

Slide 57 text

WebDriver.io WebDriver (or Selenium Grid) ChromeDriver GeckoDriver SafariDriver EdgeDriver OperaDriver Browsers Chrome Firefox Safari Edge Opera WebDriver Wire Protocol HTTP HTTP NodeJS

Slide 58

Slide 58 text

curl –X POST http://localhost:9515/session -d '{"desiredCapabilities":{"browserName":"chrome"}}' {"sessionId":"your-session-id-here","status":0,"value":{...}} curl http://localhost:9515/session/$SESSION_ID/url \ -d '{"url":"http://www.cnn.com/"} ' {"sessionId":"...","status":0,"value":null} curl http://localhost:9515/session/$SESSION_ID/element -d '{"using":"tagName","value":"h1"}' {"sessionId":"...","status":0, "value":{"ELEMENT":"element-object-id-here"}} curl http://localhost:9515/session/$SESSION_ID/element/$OBJECT_ID/text {"sessionId":"...","status":0,"value":"Example Domain"} curl –X DELETE http://localhost:9515/session/your-session-id-here {"sessionId":"...","status":0,"value":null}

Slide 59

Slide 59 text

WebDriver.io

Slide 60

Slide 60 text

Toolchain overview WebDriver.io Jenkins Web Platform Test Allure Chrome Driver Mocha Chrome Axe

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

Exercise 1: Getting Setup https://github.com/colinbendell/webdriver- workshop Request https://duckduckgo.com Enter search term (eg: "Mr. Dressup") Navigate to the 4th result and report the

Slide 63

Slide 63 text

git checkout https://github.com/colinbendell/webdriver-workshop

Slide 64

Slide 64 text

npm install chromedriver

Slide 65

Slide 65 text

npm install webdriverio

Slide 66

Slide 66 text

const webdriverio = require('webdriverio');

Slide 67

Slide 67 text

let options = { host: "localhost", port: 9515, path: "/", desiredCapabilities: { browserName: 'chrome' }; const webdriverio = require('webdriverio');

Slide 68

Slide 68 text

let options = { host: "localhost", port: 9515, path: "/", desiredCapabilities: { browserName: 'chrome' }; let browser = webdriverio.remote(options); const webdriverio = require('webdriverio');

Slide 69

Slide 69 text

let options = { host: "localhost", port: 9515, path: "/", desiredCapabilities: { browserName: 'chrome' }; let browser = webdriverio.remote(options); const webdriverio = require('webdriverio'); browser.init();

Slide 70

Slide 70 text

Slide 71

Slide 71 text

Slide 72

Slide 72 text

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

No content

Slide 75

Slide 75 text

Exercise 1: Getting Setup https://github.com/colinbendell/webdriver- workshop Request https://duckduckgo.com Enter search term (eg: "Mr. Dressup") Navigate to the 4th result and report the

Slide 76

Slide 76 text

Testing with Mocha

Slide 77

Slide 77 text

No content

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

No content

Slide 80

Slide 80 text

https://mochajs.org/

Slide 81

Slide 81 text

http://www.chaijs.com

Slide 82

Slide 82 text

npm install chai wdio-mocha-framework

Slide 83

Slide 83 text

npm install wdio-chromedriver-service

Slide 84

Slide 84 text

npm install allure allure-commandline wdio-allure-reporter

Slide 85

Slide 85 text

wdio config

Slide 86

Slide 86 text

========================= WDIO Configuration Helper ========================= ? Where do you want to execute your tests? On my local machine ? Which framework do you want to use? mocha ? Shall I install the framework adapter for you? Yes ? Where are your test specs located? ./src/**/*_spec.js ? Which reporter do you want to use? dot, allure ? Shall I install the reporter library for you? Yes ? Do you want to add a service to your test setup? Chromedriver ? Shall I install the services for you? Yes ? Level of logging verbosity error ? In which directory should screenshots gets saved if a command fails? ./errorShots/ ? What is the base url? http://localhost

Slide 87

Slide 87 text

Exercise 2: Workflow Test - Adding TODO spec Install mocha, chai, allure create wdio.conf.js Navigate to todomvc.com Using Angular implementation: • add a ToDo • marking it complete • clear the list

Slide 88

Slide 88 text

No content

Slide 89

Slide 89 text

No content

Slide 90

Slide 90 text

No content

Slide 91

Slide 91 text

No content

Slide 92

Slide 92 text

Exercise 3: POM spec Repeat Exercise 2 Use Page Object Model Simplify Allure output with wdio- allure-ts

Slide 93

Slide 93 text

Take Action • Create workflows easily with mocha • Use a Page-Object-Model to abstract selectors for reduced maintenance • Create good & actionable outputs

Slide 94

Slide 94 text

How Browsers Work

Slide 95

Slide 95 text

How Browsers Are Supposed to work

Slide 96

Slide 96 text

No content

Slide 97

Slide 97 text

No content

Slide 98

Slide 98 text

Page Load Time: Seconds

Slide 99

Slide 99 text

No content

Slide 100

Slide 100 text

2 Second Magic Number Page Load Time: Seconds

Slide 101

Slide 101 text

No content

Slide 102

Slide 102 text

! " ! 50ms DNS Lookup 100ms ShoesByColin.com 8.8.8.8

Slide 103

Slide 103 text

! " ! 50ms DNS Lookup 100ms ShoesByColin.com 8.8.8.8

Slide 104

Slide 104 text

! " ! 50ms DNS Lookup 100ms TCP Connect +100ms ShoesByColin.com 8.8.8.8

Slide 105

Slide 105 text

! " ! 50ms DNS Lookup 100ms TCP Connect +100ms TLS Negotiation +200ms ShoesByColin.com 8.8.8.8

Slide 106

Slide 106 text

! " ! ! 50ms DNS Lookup 100ms TCP Connect +100ms TLS Negotiation +200ms Application +100ms + ?? ms ShoesByColin.com 8.8.8.8

Slide 107

Slide 107 text

! " ! ! 50ms DNS Lookup 100ms TCP Connect +100ms TLS Negotiation +200ms Application +100ms + ?? ms ~ 500ms Network + 500ms? App

Slide 108

Slide 108 text

No content

Slide 109

Slide 109 text

HTML DOM JavaScript CSS Render Tree Layout Paint CSSOM Images Video JS Exec

Slide 110

Slide 110 text

Browser APIs

Slide 111

Slide 111 text

Real User Monitoring: JavaScript Timers Steve Souders Philip Tellis https://github.com/stevesouders/episodes https://github.com/lognormal/boomerang

Slide 112

Slide 112 text

No content

Slide 113

Slide 113 text

https://www.w3.org/wiki/Web_Performance/Publications

Slide 114

Slide 114 text

No content

Slide 115

Slide 115 text

Navigation Timing API

Slide 116

Slide 116 text

window.performance

Slide 117

Slide 117 text

{ "timeOrigin": 1528900579651.6672, "timing": { "navigationStart": 1528900579651, "unloadEventStart": 0, "unloadEventEnd": 0, "redirectStart": 0, "redirectEnd": 0, "fetchStart": 1528900583010, "domainLookupStart": 1528900579730, "domainLookupEnd": 1528900579730, "connectStart": 1528900579730, "connectEnd": 1528900579761, "secureConnectionStart": 1528900579733, "requestStart": 1528900579761, "responseStart": 1528900582922, "responseEnd": 1528900583010, "domLoading": 1528900583014, "domInteractive": 1528900583849, "domContentLoadedEventStart": 1528900583849, "domContentLoadedEventEnd": 1528900583876, "domComplete": 1528900585616, "loadEventStart": 1528900585617, "loadEventEnd": 1528900585856 }, "navigation": {"type": 0, "redirectCount": 0 } }

Slide 118

Slide 118 text

function getPageLoadTime() { let timing = window.performance.timing; return timing.loadEventStart - timing.navigationStart }

Slide 119

Slide 119 text

Slide 120

Slide 120 text

No content

Slide 121

Slide 121 text

No content

Slide 122

Slide 122 text

window.performance

Slide 123

Slide 123 text

{ "name": "https://ajax.googleapis.com/ajax/libs/jquery/1.11.2/jquery.min.js", "entryType": "resource", "startTime": 928.70000000039, "duration": 425.8000000008906, "initiatorType": "script", "nextHopProtocol": "h2", "workerStart": 0, "redirectStart": 0, "redirectEnd": 0, "fetchStart": 928.70000000039, "domainLookupStart": 937.4000000025262, "domainLookupEnd": 945.9000000024389, "connectStart": 945.9000000024389, "connectEnd": 967.5999999999476, "secureConnectionStart": 946.2000000021362, "requestStart": 949.5999999999185, "responseStart": 970.9000000002561, "responseEnd": 1354.5000000012806, "transferSize": 33798, "encodedBodySize": 33495, "decodedBodySize": 95931, "serverTiming": [] }

Slide 124

Slide 124 text

let url = “https://bendell.ca/logo.png”; let me = performance.getEntriesByName(url)[0]; let timings = { loadTime: me.duration, dns: me.domainLookupEnd - me.domainLookupStart, tcp: me.connectEnd - me.connectStart, waiting: me.responseStart - me.requestStart, fetch: me.responseEnd - me.responseStart }

Slide 125

Slide 125 text

200 OK HTTP/1.1 Timing-Allow-Origin: * Content-Type: application/javascript Content-Length: 1234

Slide 126

Slide 126 text

• Slowest resources • Time to first image • Response time by domain • Time a group of assets • Response time by initiator type (element type) • Browser cache-hit ratio for resources • Resource Timing API • Other Uses More: http://www.slideshare.net/bluesmoon/beyond-page-level-metrics

Slide 127

Slide 127 text

No content

Slide 128

Slide 128 text

How long does it take to display the main product image on my site?

Slide 129

Slide 129 text

function loadTemplate() { performance.mark(“startTask”); doTemplateStuff() //do stuff (assumes promise()) .then(() => { performance.mark(“stopTask”); //Measure the duration between the two marks performance.measure(“taskDuration”, “startTask”,“stopTask”); }); }

Slide 130

Slide 130 text

https://stevesouders.com/blog/2015/05/12/hero-image-custom-metrics/

Slide 131

Slide 131 text

No content

Slide 132

Slide 132 text

GET /resource HTTP/1.1 Host: example.com HTTP/1.1 200 OK Server-Timing: cdn, dur="130" Server-Timing: cloudinary, dur="10", desc="new trans"

Slide 133

Slide 133 text

{ let url = 'https://example.com/resource.jpg’; let severEntries[] = window.performance.getEntriesByName(url) .filter((name) => name === "server"); }

Slide 134

Slide 134 text

No content

Slide 135

Slide 135 text

No content

Slide 136

Slide 136 text

No content

Slide 137

Slide 137 text

Slide 138

Slide 138 text

No content

Slide 139

Slide 139 text

Exercise 4: performance budget Navigate to amazon.com Calculate the total bytes downloaded Ensure that the page load time < 2s and bytes < 2MiB Repeat with Disney.com

Slide 140

Slide 140 text

Take Action • Add performance.mark() to measure YOUR key metrics • Eg: Hero Image, Product Image, Sport Scores • Keep performance.mark() in production code! • Define Hard and Soft targets that synthetic tests should hit • BHAG: 2s & 1MiB

Slide 141

Slide 141 text

Mobile Display & Network Throttling Tim Kadlec

Slide 142

Slide 142 text

No content

Slide 143

Slide 143 text

No content

Slide 144

Slide 144 text

No content

Slide 145

Slide 145 text

No content

Slide 146

Slide 146 text

chromeOptions: { args:["--incognito", "--headless", "--disable-gpu"] },

Slide 147

Slide 147 text

chromeOptions: { args:["--incognito", "--disable-gpu"], mobileEmulation: { deviceMetrics: { width: 360, height: 720, pixelRatio: 3}, networkThrottle: 'Fast 3G' } }

Slide 148

Slide 148 text

No content

Slide 149

Slide 149 text

Take Action • Run Tests in different viewport configurations to test RWD • Experiment with different Network Conditions • Similarly with different CPU throttling

Slide 150

Slide 150 text

Web Platform Test

Slide 151

Slide 151 text

https://web-platform-tests.org

Slide 152

Slide 152 text

https://platform.html5.org

Slide 153

Slide 153 text

https://darobin.github.io/test-harness-tutorial/docs/using-testharness.html

Slide 154

Slide 154 text

Why Inside v. Outside Browser Testing? • Hawthorne Effect • Platform Widgets & Controls • APIs

Slide 155

Slide 155 text

Exercise 5: WPT Implement Responsive Images and Test the naturalWidth

Slide 156

Slide 156 text

A11y with AXE Léonie Watson Marcy Sutton Katie Sylor-Miller

Slide 157

Slide 157 text

“Websites, tools, and technologies are designed and developed so that people with disabilities can use them.” ACCESSIBILITY (a11y) - W3C Web Accessibility Initiative

Slide 158

Slide 158 text

THE AMERICANS WITH DISABILITIES ACT (ADA) “No individual shall be discriminated against on the basis of disability in the full and equal enjoyment of the goods, services, facilities, privileges, advantages, or accommodations of any place of public accommodation” - ADA Sec. 12182 (a)

Slide 159

Slide 159 text

AMERICANS WITH A DISABILITY UNITED STATES CENSUS REPORT, JULY 2012 57 M 19%

Slide 160

Slide 160 text

No content

Slide 161

Slide 161 text

No content

Slide 162

Slide 162 text

The Basics • Web Content Accessibility Guidelines (WCAG 2.0) • Developed by the W3C Web Accessibility Three levels of compliance - A, AA, AAA

Slide 163

Slide 163 text

No content

Slide 164

Slide 164 text

No content

Slide 165

Slide 165 text

Exercise 6: A11y Add axe-core npm Load 3 news websites Compare the violations Create a test case to fail on a11y violations

Slide 166

Slide 166 text

Take Action • Good Usability === Good Accessibility • Use Semantic HTML (also great for SEO) • Add `alt` and `description` tags to media and anchors • Test your site (axe-core.com) • Try the Funkity-Disability-Simulator Chrome Extension https://chrome.google.com/webstore/detail/funkify-disability- simula/ojcijjdchelkddboickefhnbdpeajdjg?hl=en • Be an ally

Slide 167

Slide 167 text

DNS

Slide 168

Slide 168 text

Take Action

Slide 169

Slide 169 text

Parallization

Slide 170

Slide 170 text

Take Action

Slide 171

Slide 171 text

CI/CD/CB with Jenkins

Slide 172

Slide 172 text

Take Action

Slide 173

Slide 173 text

Real User Monitoring with Lux

Slide 174

Slide 174 text

Synthetic Real • Consistent • Precise CPU, Network use • Availability • Engagement • Business Metrics (Rev) • Market Segment • Layout • Design

Slide 175

Slide 175 text

No content

Slide 176

Slide 176 text

No content

Slide 177

Slide 177 text

No content

Slide 178

Slide 178 text

Take Action • Collect Real-User browser exerpiences • Audit user experience (load time, downloaded content) • Periodically run tests in the wild • Add window.addEventListener(‘error’) to capture console errors • Use Beacon API to ensure metrics are returned

Slide 179

Slide 179 text

Summary • Browser testing is easy! • Synthetic testing with: • WebDriver.io for easy functional testing • WebPageTest.org to augment performance testing • Use WebPlatformTest for direct platform tests • Don’t just test workflows: Design, Performance, and Accessibility • Use Real-User feedback to compare tests in the wild

Slide 180

Slide 180 text

@ColinBendell

Slide 181

Slide 181 text

Further Reading • http://webdriver.io/api.html • https://github.com/cloudinary/wdio-allure-ts • https://github.com/cloudinary/wdio-allure-ts-example • https://www.npmjs.com/package/webdriver-manager • http://blog.kevinlamping.com/testing-your-login-an-in-depth-webdriverio-tutorial/ • https://www.browserstack.com/automate/webdriverio • https://medium.com/@boriscoder/setting-up-selenium-tests-with-webdriver-io-cc7fc3c86629 • https://w3c.github.io/webdriver/ • https://www.slideshare.net/LinkMeSrl/webdriverio?qid=8cdf045d-9868-4ca8-a8ad-8aefb5c22289&v=&b=&from_search=1 • https://peter.sh/experiments/chromium-command-line-switches/#load-extension • https://www.ghacks.net/2013/10/06/list-useful-google-chrome-command-line-switches/ • https://deliciousbrains.com/how-were-automating-acceptance-testing/ • https://medium.com/tech-tajawal/page-object-model-pom-design-pattern-f9588630800b