State of web integration testing Where we are now, and where we should go 1 Matti Schneider @matti_sg UNDERLINED WORDS IN TRANSCRIPT MEAN YOU SHOULD PASS TO THE NEXT SLIDE. Good afternoon everyone, and welcome to this session! I'll be talking about the current state of web integration testing: what it is exactly, which practices it covers, how you could do it today, the limitations you'll probably face, and how we should be doing it in the future. 

2 Matti Schneider @matti_sg My name is Matti Schneider. The content in this presentation is based on 5 months of full-time research on the topic I did last year, and the development of a web integration testing framework, Watai, that basically aimed at going beyond the limitations shared by all the tools that were available at the time. It also includes experience gained during the last months where I've been working at MESH, a startup targeting surgery services. In this context, our web application's quality is not simply a matter of customer satisfaction, it is absolutely critical for the daily work of health professionals. Finally, please note that I'll be mostly taking the point of view of a development team wanting to do automated testing, rather than of a specialized QA team, since that's where my own experience is coming from. That's where I'm talking from. 

2 Matti Schneider github.com/MattiSG/Watai @matti_sg My name is Matti Schneider. The content in this presentation is based on 5 months of full-time research on the topic I did last year, and the development of a web integration testing framework, Watai, that basically aimed at going beyond the limitations shared by all the tools that were available at the time. It also includes experience gained during the last months where I've been working at MESH, a startup targeting surgery services. In this context, our web application's quality is not simply a matter of customer satisfaction, it is absolutely critical for the daily work of health professionals. Finally, please note that I'll be mostly taking the point of view of a development team wanting to do automated testing, rather than of a specialized QA team, since that's where my own experience is coming from. That's where I'm talking from. 

2 Matti Schneider github.com/MattiSG/Watai @matti_sg My name is Matti Schneider. The content in this presentation is based on 5 months of full-time research on the topic I did last year, and the development of a web integration testing framework, Watai, that basically aimed at going beyond the limitations shared by all the tools that were available at the time. It also includes experience gained during the last months where I've been working at MESH, a startup targeting surgery services. In this context, our web application's quality is not simply a matter of customer satisfaction, it is absolutely critical for the daily work of health professionals. Finally, please note that I'll be mostly taking the point of view of a development team wanting to do automated testing, rather than of a specialized QA team, since that's where my own experience is coming from. That's where I'm talking from. 

3 Vocabulary We need to share the same language. If you've read about the subject, you might have noticed the terminology depends on the author. 

4 The first level, the one that is closest to the code, is unit testing. Unit testing is concerned with the smallest piece of code that contains business logic. Usually, it means methods or functions. In the context of web applications, the tested “units” may reside either on the server or the client-side. Both parts should be unit-tested if you're to be serious about quality. 

4 unit The first level, the one that is closest to the code, is unit testing. Unit testing is concerned with the smallest piece of code that contains business logic. Usually, it means methods or functions. In the context of web applications, the tested “units” may reside either on the server or the client-side. Both parts should be unit-tested if you're to be serious about quality. 

5 unit Testing parts of the programs that implement independant features. The exact “part” depends on the architecture, the environment, the size of the system… May be classes, modules, or whole parts in distributed architectures. In the context of web applications, this could be for example a JSON endpoint or a frontend component. 

5 unit functional Testing parts of the programs that implement independant features. The exact “part” depends on the architecture, the environment, the size of the system… May be classes, modules, or whole parts in distributed architectures. In the context of web applications, this could be for example a JSON endpoint or a frontend component. 

acceptance functional system validation customer end-to-end 6 unit functional Testing level in which individual software modules are tested as a group. It also means testing through the same access means as the final users will. In the context of web applications, it means testing the result of combining client- and server-side code, and accessing it through a browser. 

acceptance functional system validation customer end-to-end 6 unit functional integration Testing level in which individual software modules are tested as a group. It also means testing through the same access means as the final users will. In the context of web applications, it means testing the result of combining client- and server-side code, and accessing it through a browser. 

7 customer end-to-end acceptance validation system functional unit functional integration Many synonyms for the same reality. The most problematic: “functional”. Some authors list only “functional testing” and “unit testing”. It's up to you, but I personally prefer differentiating. In this presentation, we'll only refer to integration testing. “Customer”, “end-to-end”: a bit dated. “System”: includes data. “Validation”, “Acceptance”: performed by the customer. May detect performance, usability or data issues. Let's see what this “reality” really looks like. 

8 [video demonstration of automated integration tests of the DuckDuckGo engine running] 

9 acceptance functional system validation customer end-to-end unit functional integration So, that was an example of what is a web integration test. Let's make sure we also agree on the goals of such a practice. 

10 Web integration testing's goals are the following. First, ensure that a functionality, a feature that adds customer value, is present. Second, proving that this functionality behaves properly. That is, given a set of inputs, and a navigation path, the expected information is outputted to the user. Finally, in the context of web applications, there's one important evil we're fighting against: platform discrepancies. So, we're not only concerned about functionalities being present, nor behaving properly, but about them behaving properly in all browsers. Now, how do we do this? 

functionality 10 Web integration testing's goals are the following. First, ensure that a functionality, a feature that adds customer value, is present. Second, proving that this functionality behaves properly. That is, given a set of inputs, and a navigation path, the expected information is outputted to the user. Finally, in the context of web applications, there's one important evil we're fighting against: platform discrepancies. So, we're not only concerned about functionalities being present, nor behaving properly, but about them behaving properly in all browsers. Now, how do we do this? 

functionality 10 correctness Web integration testing's goals are the following. First, ensure that a functionality, a feature that adds customer value, is present. Second, proving that this functionality behaves properly. That is, given a set of inputs, and a navigation path, the expected information is outputted to the user. Finally, in the context of web applications, there's one important evil we're fighting against: platform discrepancies. So, we're not only concerned about functionalities being present, nor behaving properly, but about them behaving properly in all browsers. Now, how do we do this? 

functionality 10 correctness consistency Web integration testing's goals are the following. First, ensure that a functionality, a feature that adds customer value, is present. Second, proving that this functionality behaves properly. That is, given a set of inputs, and a navigation path, the expected information is outputted to the user. Finally, in the context of web applications, there's one important evil we're fighting against: platform discrepancies. So, we're not only concerned about functionalities being present, nor behaving properly, but about them behaving properly in all browsers. Now, how do we do this? 

Tools 11 Important problem against adoption: language and articles not clear. There is no “tool”, only a complex tool stack. Seeing the history of the current web automation stack is probably the best way to understand it properly. 

12 Super Webapp So, let's say you have this great web application, almost ready for production, you want to test. 

13 Super Webapp You want to test this application against all browsers your users could use to get access to it. 

14 Super Webapp Not only do you want to test against all these browsers, you actually want to test against all versions of them your users could be using. 

15 That's a lot of browsers. Luckily, there's one thing developers do well: adding abstraction. 

15 That's a lot of browsers. Luckily, there's one thing developers do well: adding abstraction. 

16 Browser controller Two tools went this way: Sahi, and Selenium. We'll focus on Selenium, as Sahi is more directed at professional QA teams, and Selenium at developers. It would definitely be worth a look if you can afford having a dedicated testing team though. 

17 Selenium is a library to automate browsers, started back in 2004. Its original API is in Java. Java was sexy at the time, but other languages started getting used for web projects. And teams wanted tools in the language they were using for the main application development instead of changing context all the time. So, what did developers do? They added a layer of abstraction! 

17 Selenium is a library to automate browsers, started back in 2004. Its original API is in Java. Java was sexy at the time, but other languages started getting used for web projects. And teams wanted tools in the language they were using for the main application development instead of changing context all the time. So, what did developers do? They added a layer of abstraction! 

Java Ruby .NET Node.js PHP 18 Now, you could write your tests in your preferred language through bindings for the Selenium API, or through JSON over HTTP. However, the Java API was just one part of Selenium. The other end, the one automating browsers, was a JavaScript library injected in each browser, that would simulate user interaction. Over time, JavaScript security concerns were raised, differences with native capabilities grew wider, and simulating user behavior became harder. Luckily, advanced automation was possible for each browser, individually. For example, Firefox exposes an automation API. IE is automatable through Windows accessibility services, and so on. But then you'd lose this ability to write the same code for all browsers… So, what did we do? Add another layer! 

Java Ruby .NET Node.js PHP 18 Now, you could write your tests in your preferred language through bindings for the Selenium API, or through JSON over HTTP. However, the Java API was just one part of Selenium. The other end, the one automating browsers, was a JavaScript library injected in each browser, that would simulate user interaction. Over time, JavaScript security concerns were raised, differences with native capabilities grew wider, and simulating user behavior became harder. Luckily, advanced automation was possible for each browser, individually. For example, Firefox exposes an automation API. IE is automatable through Windows accessibility services, and so on. But then you'd lose this ability to write the same code for all browsers… So, what did we do? Add another layer! 

Java Ruby .NET Node.js PHP Safari driver IE driver Chrome driver Firefox driver Opera driver 19 A project already implemented such an approach: WebDriver. It was merged with the JavaScript Selenium library (Selenium Remote Control) in 2009, giving birth to Selenium 2.0. Actually, if you read about Selenium, Selenium 2.0 or WebDriver today, it's most usually referring to the same thing, that very stack. Which is a great stack for automation. But even though it has a checkbox in its logo, Selenium is only a browser automation tool, not a testing tool. So, what's a developer to do? Add abstraction! 

Java Ruby .NET Node.js PHP Safari driver IE driver Chrome driver Firefox driver Opera driver 19 A project already implemented such an approach: WebDriver. It was merged with the JavaScript Selenium library (Selenium Remote Control) in 2009, giving birth to Selenium 2.0. Actually, if you read about Selenium, Selenium 2.0 or WebDriver today, it's most usually referring to the same thing, that very stack. Which is a great stack for automation. But even though it has a checkbox in its logo, Selenium is only a browser automation tool, not a testing tool. So, what's a developer to do? Add abstraction! 

Java Ruby .NET Node.js PHP Safari driver IE driver Chrome driver Firefox driver Opera driver Watir Watij Watin Intern Mink 20 So, libraries helping with testing were developed over all bindings, bundling assertions, runners… Now we have a great tool for integration testing! Then the second great browsers' arms race started, and updates to the browsers became much more frequent. What happens then if a browser's updated, for example, and it breaks compatibility with one of the tools at the other end of that stack? That's when you start realizing it's slow and heavy. So developers did the second thing they do best: restart from scratch. 

Java Ruby .NET Node.js PHP Safari driver IE driver Chrome driver Firefox driver Opera driver Watir Watij Watin Intern Mink 20 So, libraries helping with testing were developed over all bindings, bundling assertions, runners… Now we have a great tool for integration testing! Then the second great browsers' arms race started, and updates to the browsers became much more frequent. What happens then if a browser's updated, for example, and it breaks compatibility with one of the tools at the other end of that stack? That's when you start realizing it's slow and heavy. So developers did the second thing they do best: restart from scratch. 

Java Ruby .NET Node.js PHP Safari driver IE driver Chrome driver Firefox driver Opera driver Watir Watij Watin Intern Mink 20 So, libraries helping with testing were developed over all bindings, bundling assertions, runners… Now we have a great tool for integration testing! Then the second great browsers' arms race started, and updates to the browsers became much more frequent. What happens then if a browser's updated, for example, and it breaks compatibility with one of the tools at the other end of that stack? That's when you start realizing it's slow and heavy. So developers did the second thing they do best: restart from scratch. 

21 PhantomJS was created about two years ago by Ariya Hidayat. 

22 It is a headless (hence the “phantom” part of its name) WebKit engine. That is, you start it up through the command-line, and it doesn't show any graphical user interface at all. Of course, interacting with a browser you can't see is kind of hard. So, PhantomJS is also a JavaScript (hence the “JS” part) API to control that browser. That's a great, fast and sexy tool for browser automation! But once again, not dedicated specifically to testing. 

23 So Nicolas Perriault created CasperJS, a set of JavaScript scripting and testing utilities. Now we have a great stack for web integration testing! Or maybe not. It is very important to understand that even though this stack is lovely for developers for its simplicity and speed, it can not be considered an actual integration testing stack. Indeed, unless your users are accessing your application through a browser that doesn't have an interface, which I doubt, you are not testing what your users will get. The engine can differ (WebKit is now used only in Safari) but, beyond the rendering engine, the JavaScript engine is different too. We've experienced first hand that tests passing under Phantom can fail in actual browsers. Or the other way around. 

24 Still, that stack is attractive. So much that last month, Laurent Jouanneau created SlimerJS, a project aiming to reproduce what Phantom did for WebKit, but for the Gecko engine. It is compatible with Phantom's API, making Casper able to control both. 

Java Ruby .NET Node.js PHP Safari driver IE driver Chrome driver Firefox driver Opera driver Watir Watij Watin wd Mink 25 If this new stack starts reminding you of the earlier jellyfish, it might not be an entire coincidence. Actually, we may as well accept that all browsers are different, and that automating them requires a specific strategy for each. And that these strategies should be abstracted, as well as the languages the tests are written in. What will happen when JavaScript is not hype anymore, and that we have a huge legacy of Casper-written tests? Will we write CasperClojure? And now, the two stacks are even starting to communicate, for example with GhostDriver by Ivan De Marino, which allows Selenium to control Phantom. 

Java Ruby .NET Node.js PHP Safari driver IE driver Chrome driver Firefox driver Opera driver Watir Watij Watin wd Mink Ghost driver 25 If this new stack starts reminding you of the earlier jellyfish, it might not be an entire coincidence. Actually, we may as well accept that all browsers are different, and that automating them requires a specific strategy for each. And that these strategies should be abstracted, as well as the languages the tests are written in. What will happen when JavaScript is not hype anymore, and that we have a huge legacy of Casper-written tests? Will we write CasperClojure? And now, the two stacks are even starting to communicate, for example with GhostDriver by Ivan De Marino, which allows Selenium to control Phantom. 

Complexity 26 You've understood that this tool's problem is not its power, it's its complexity. Luckily, there are solutions to hide this complexity. We'll see two examples of tools that do. 

27 Capybara Capybara is packaged as a gem, and very well-known in the Ruby / Rails community. It glues all the components listed earlier together, so that you don't have to install them manually. It also bundles some test parsers. 

28 This is a good time to have a slightly off-topic note. I'd like to address one common misconception about Cucumber, a Ruby tool. Cucumber often comes with Capybara, and one can easily read advice about doing web integration testing with it. Unfortunately, this is a stretch, and this stretch hurts adoption by tying the practice to a specific format that is not always liked. Cucumber is only a behavior-driven development (BDD) framework, not an integration testing tool in itself. 

28 BDD framework This is a good time to have a slightly off-topic note. I'd like to address one common misconception about Cucumber, a Ruby tool. Cucumber often comes with Capybara, and one can easily read advice about doing web integration testing with it. Unfortunately, this is a stretch, and this stretch hurts adoption by tying the practice to a specific format that is not always liked. Cucumber is only a behavior-driven development (BDD) framework, not an integration testing tool in itself. 

29 Given-When-Then This means that Cucumber is able, through transformation rules you have to define, to transform feature descriptions written in a pseudo-English language (Gherkin), emblematically composed of Given-When-Then parts, in testing code. This testing code can be anything, including unit tests. Then, indeed, if you add the Watir (Web Integration Testing In Ruby) API in the testing code, you are able to control Selenium and, in the end, to automate browsers for testing. So, while it is not technically wrong to say so, remember that doing “web integration testing with Cucumber” is quite a stretch, and that the tool itself is not specific to integration testing, even less so a requirement. 

29 • – • – • – Given-When-Then Testing code This means that Cucumber is able, through transformation rules you have to define, to transform feature descriptions written in a pseudo-English language (Gherkin), emblematically composed of Given-When-Then parts, in testing code. This testing code can be anything, including unit tests. Then, indeed, if you add the Watir (Web Integration Testing In Ruby) API in the testing code, you are able to control Selenium and, in the end, to automate browsers for testing. So, while it is not technically wrong to say so, remember that doing “web integration testing with Cucumber” is quite a stretch, and that the tool itself is not specific to integration testing, even less so a requirement. 

29 • – • – • – Given-When-Then Testing code This means that Cucumber is able, through transformation rules you have to define, to transform feature descriptions written in a pseudo-English language (Gherkin), emblematically composed of Given-When-Then parts, in testing code. This testing code can be anything, including unit tests. Then, indeed, if you add the Watir (Web Integration Testing In Ruby) API in the testing code, you are able to control Selenium and, in the end, to automate browsers for testing. So, while it is not technically wrong to say so, remember that doing “web integration testing with Cucumber” is quite a stretch, and that the tool itself is not specific to integration testing, even less so a requirement. 

29 • – • – • – Given-When-Then Testing code Browser automation This means that Cucumber is able, through transformation rules you have to define, to transform feature descriptions written in a pseudo-English language (Gherkin), emblematically composed of Given-When-Then parts, in testing code. This testing code can be anything, including unit tests. Then, indeed, if you add the Watir (Web Integration Testing In Ruby) API in the testing code, you are able to control Selenium and, in the end, to automate browsers for testing. So, while it is not technically wrong to say so, remember that doing “web integration testing with Cucumber” is quite a stretch, and that the tool itself is not specific to integration testing, even less so a requirement. 

30 An alternative to Capybara is The Intern, a framework made available a few months ago. It is also a “big piece of glue”, packaging a Selenium-controlling library, assertion libraries, test runners… but, this time, making you write tests in JavaScript instead of Ruby. Additionnally, it does a very cool thing: it dispatches both your integration tests and frontend unit tests across all the browsers you're targeting. Let's now see how to actually run such tests across all these browsers. 

Environment 31 As we've seen in our tool stack tour, testing on all browsers is mandatory, and is done by automating those browsers. But, it also means that all of them have to be somehow available for us to automate. This is where it's getting worse: beyond the stack, the execution environment also plays a role. 

Local testing 32 The easiest way to get access to browsers is to use the ones available on your development machine. As we've seen, to automate them, the corresponding drivers have to be installed too. Finally, a Selenium server has to be available to communicate your instructions to these drivers. This server comes as a Java executable archive. Then, you simply have to start that service through a line of command to open a Selenium server on some local port and direct your testing framework at that local port to use all browsers available on the local system. However, you're limited by your operating system. For example, you cannot test Safari on non-OSX, nor IE on non-Windows. Also, browsers do not always behave exactly the same on different platforms. As we've experienced first-hand, for example, a Firefox on OS X does not have the same bugs than the same version of Firefox on Windows… To tackle that problem, luckily, we can bring our old friend abstraction! 

Local testing 32 The easiest way to get access to browsers is to use the ones available on your development machine. As we've seen, to automate them, the corresponding drivers have to be installed too. Finally, a Selenium server has to be available to communicate your instructions to these drivers. This server comes as a Java executable archive. Then, you simply have to start that service through a line of command to open a Selenium server on some local port and direct your testing framework at that local port to use all browsers available on the local system. However, you're limited by your operating system. For example, you cannot test Safari on non-OSX, nor IE on non-Windows. Also, browsers do not always behave exactly the same on different platforms. As we've experienced first-hand, for example, a Firefox on OS X does not have the same bugs than the same version of Firefox on Windows… To tackle that problem, luckily, we can bring our old friend abstraction! 

Local testing 32 drivers + The easiest way to get access to browsers is to use the ones available on your development machine. As we've seen, to automate them, the corresponding drivers have to be installed too. Finally, a Selenium server has to be available to communicate your instructions to these drivers. This server comes as a Java executable archive. Then, you simply have to start that service through a line of command to open a Selenium server on some local port and direct your testing framework at that local port to use all browsers available on the local system. However, you're limited by your operating system. For example, you cannot test Safari on non-OSX, nor IE on non-Windows. Also, browsers do not always behave exactly the same on different platforms. As we've experienced first-hand, for example, a Firefox on OS X does not have the same bugs than the same version of Firefox on Windows… To tackle that problem, luckily, we can bring our old friend abstraction! 

Local testing 32 drivers + + The easiest way to get access to browsers is to use the ones available on your development machine. As we've seen, to automate them, the corresponding drivers have to be installed too. Finally, a Selenium server has to be available to communicate your instructions to these drivers. This server comes as a Java executable archive. Then, you simply have to start that service through a line of command to open a Selenium server on some local port and direct your testing framework at that local port to use all browsers available on the local system. However, you're limited by your operating system. For example, you cannot test Safari on non-OSX, nor IE on non-Windows. Also, browsers do not always behave exactly the same on different platforms. As we've experienced first-hand, for example, a Firefox on OS X does not have the same bugs than the same version of Firefox on Windows… To tackle that problem, luckily, we can bring our old friend abstraction! 

Local testing 32 java -jar selenium-server-standalone-2.35.0.jar -p 4444 drivers + + The easiest way to get access to browsers is to use the ones available on your development machine. As we've seen, to automate them, the corresponding drivers have to be installed too. Finally, a Selenium server has to be available to communicate your instructions to these drivers. This server comes as a Java executable archive. Then, you simply have to start that service through a line of command to open a Selenium server on some local port and direct your testing framework at that local port to use all browsers available on the local system. However, you're limited by your operating system. For example, you cannot test Safari on non-OSX, nor IE on non-Windows. Also, browsers do not always behave exactly the same on different platforms. As we've experienced first-hand, for example, a Firefox on OS X does not have the same bugs than the same version of Firefox on Windows… To tackle that problem, luckily, we can bring our old friend abstraction! 

Virtualized 33 Local testing If you use virtual machines, you may use as many browser/platform combinations you want. However, all these OSs loaded alongside will quickly bring your development machine to its knees. So, you'll need servers… and then, you'll have to maintain the associated infrastructure and, before you know it, you're paying for your own QA lab, which will probably end up costing you more than the benefits of testing automatically. This observation of course depends on your size. If you can afford having an infrastructure-dedicated team and already have a QA team, then it may definitely be worth it. Otherwise, if your team is mostly developers, you'll probably be better off externalizing these costs. 

Virtualized 33 Local testing If you use virtual machines, you may use as many browser/platform combinations you want. However, all these OSs loaded alongside will quickly bring your development machine to its knees. So, you'll need servers… and then, you'll have to maintain the associated infrastructure and, before you know it, you're paying for your own QA lab, which will probably end up costing you more than the benefits of testing automatically. This observation of course depends on your size. If you can afford having an infrastructure-dedicated team and already have a QA team, then it may definitely be worth it. Otherwise, if your team is mostly developers, you'll probably be better off externalizing these costs. 

34 Continuous integration The duration of integration tests really makes them well-suited for continuous integration (CI). Hosted CI is a really great thing, and many services will provide you with a Selenium server, drivers and the latest browsers out- of-the-box. We're for example using CircleCI, a startup created at the end of 2012, and are really happy with their service. However, no matter the tool, you will still be limited to the OS your tests are running in (usually a Linux box), and you can't control the stack. So, you get the latest browsers… always. Even when a browser updates, and the driver is updated, and your tests are not compatible with that latest version, and it is two days before a deadline. Obviously, what you gain in ease of use is lost in control. 

35 Reliability As you have understood with the impact of the environment, the second pain in web integration testing is its reliability. And unstable tests are worse than no tests at all: it leads to false confidence, and to ignore actual failures. Luckily, there's once again a solution to mitigate this lack of reliability. 

36 SauceLabs is a startup that aims to solve this exact problem. It maintains VMs with all OS/browser combinations, created on demand and available over the internet through a Selenium endpoint. Most importantly, they offer a no-brainer tunnel, so you can easily test an application locally without deploying it over the internet. Using their service is not exclusive at all of hosted continuous integration services. Actually, we are using both in combination, and that works really well. 

APIs 37 We've seen the concepts, the environment, and some tools that seem to be doing the work. Let's now have a quick look at how you would actually write tests with them! 

38 return this.remote .get(require.toUrl(url)) .elementById('new-todo') .clickElement() .keys('Task 1') .keys("\n") .keys('Task 2') .keys("\n") .keys('Task 3') .getAttribute('value') .then(function(val) { assert.ok(val.indexOf('Task 3') > -1); }) scenario "Signing in with correct credentials" do page.visit "/sessions/new" page.fill_in "email", :with => "[email protected]" page.fill_in "password", :with => "qwerty" page.click_button "Login" page.should have_content("Welcome, [email protected]!") end Intern Capybara Here are small code excerpts from standard example tests written with the two tools I presented earlier. The goal is not that you look at them in details. If you want to try them, you will do so on your own. What I want you to note now is the global flow, what the code says. It's all about visiting an URL, walking the DOM to select an element, sending clicks and keystroke events, getting HTML attributes… 

Old web 39 Basically, the current tools for web testing target the old web. The web of websites, not the one of web applications. If you're developing a webapp with a serious frontend today, you don't think about pages. You think about components and APIs. I don't mean to diminish these tools. They do a great work. But they are not in phase with the current practices of web development. Do you really want to be walking the DOM in your tests while you have an object layer in your main code? You'd never do that in unit tests. That makes your tests fragile, would it be only by forcing white-box testing on you. As long as the testing tools are stuck behind like this, testing will be painful. So, today, I'd like to start a discussion on what should be the future of web integration testing. Here is a proposal based on my own research and experiments. 

40 Component Fixture • — • — • — • — Scenario We're designing web applications through components, pieces of content that present the user with graphical elements and behaviors triggered by interaction with these elements. What about modeling all of that for automation? Then, we would write usage scenarios, that would define both how those components are expected to interact and their status at any point through interactions. Finally, we should maximize the maintainability of our tests by extracting everything that is different from this logic, such as specific data or user messages, so that changing a phrasing does not break our testing ability. Let's see how tests using such an architecture could actually be written. 

Component 41 The introductory video we saw at the beginning of this presentation checked the presence of a disambiguation infobox on an ambiguous query on the DuckDuckGo search engine. We will follow this example. As we've seen, we'd first model the components. Two are relevant to this test: the search bar itself, and the expected infobox. They both have elements, which are easily definable through a name describing their usage, mapped to CSS selectors to locate them. The search bar, though, also presents users with a possible action, made of combining simple interactions with its different elements. This has to be written with code, here in JavaScript. Now that the needed components have been defined, let's specify how they are expected to interact. 

Component 41 SearchBar field: 'input[name=q]', submitButton: '#search_button_homepage', searchFor: function searchFor(term) { return this.setField(term)() .then(this.submit()); } InfoBox header: '#zero_click_heading' The introductory video we saw at the beginning of this presentation checked the presence of a disambiguation infobox on an ambiguous query on the DuckDuckGo search engine. We will follow this example. As we've seen, we'd first model the components. Two are relevant to this test: the search bar itself, and the expected infobox. They both have elements, which are easily definable through a name describing their usage, mapped to CSS selectors to locate them. The search bar, though, also presents users with a possible action, made of combining simple interactions with its different elements. This has to be written with code, here in JavaScript. Now that the needed components have been defined, let's specify how they are expected to interact. 

42 Scenario We first write a short human-readable description of the feature we're testing. Then, this feature is tested through a scenario. This scenario relies on the defined components to both act on them and define their expected status. 

42 Scenario Toto = Toto ? InfoBoxFeature description: 'Looking up an ambiguous term should present an info box.', scenario: [ SearchBarWidget.searchFor(lookupTerm), { 'ZeroClickWidget.header': 'Meanings of ' + lookupTerm } ] We first write a short human-readable description of the feature we're testing. Then, this feature is tested through a scenario. This scenario relies on the defined components to both act on them and define their expected status. 

43 Fixture • — • — • — • — Finally, the data that is used to trigger the behavior (here, the term considered as “ambiguous” enough to display a disambiguation infobox) is injected into the test. 

Toto 43 Fixture SearchFixture lookupTerm = "Toto" • — • — • — • — Finally, the data that is used to trigger the behavior (here, the term considered as “ambiguous” enough to display a disambiguation infobox) is injected into the test. 

44 Component Scenario Fixture • — • — • — • — github.com/MattiSG/Watai The code you've just seen is taken from the first example bundled with the testing framework I wrote, Watai. Now, I won't lie to you: Watai is currently considered in beta. We've been using it in production successfully for the last 11 months, but it still has some rough edges on syntax and ease-of-use. If the example appealed to you, the best is that you give it a quick try. The end vision would be that every Web Component delivered includes its automation description. You can view this as a “human API specification”, just like components are expected to have unit tests to check their developer API. This would allow black-box testing of web applications, even when they are made of the combination of several third-party components with specific code, just like we do for other levels of testing. 

Recap 45 Let's review what we've seen today. 

46 We've seen what is web integration testing. It is about simulating user interaction across all target browsers. It can be done manually, but it is very costly. We want to automate it. 

47 The current automation stack is ugly. But there's no real way to get away from the fact that each browser is different, that we need to abstract away their differences, and that we also need to abstract away from implementation languages. There are tools that bundle all the stack so that you don't have to maintain it yourself. Find the one that suits your particular language / framework / methodology. 

48 Maintaining your own lab is costly. Running tests is slow. The best solution is probably to run “smoke tests” locally while developing on whichever browser (possibly headless) is available on your dev machines, and delegate testing across target browsers to a hosted CI tool. 

49 • — • — • — • — Finally, we've seen that all current popular frameworks have a common weakness: they are not built for the modern web. The alternative solutions are still under development. 

To-do 50 So, what if you wanted to start using web integration testing on your project tomorrow? First of all, get familiar with the notions. If I've done my job well today, you can tick that one now. Then, try tools that appeal to you and your team. There are some well-known, go-to solutions, but they all depend on your preferred language and framework. Based on these first tries, evaluate RoI on your project. The global idea is to consider if the regression costs are higher than those of inserting a new testing tool and platform. Finally, since all this is quite in a lively time, participate in the community. It's only by getting everyone on the board that we'll be able to create a web of quality. 

To-do 50 Learn concepts ‣ So, what if you wanted to start using web integration testing on your project tomorrow? First of all, get familiar with the notions. If I've done my job well today, you can tick that one now. Then, try tools that appeal to you and your team. There are some well-known, go-to solutions, but they all depend on your preferred language and framework. Based on these first tries, evaluate RoI on your project. The global idea is to consider if the regression costs are higher than those of inserting a new testing tool and platform. Finally, since all this is quite in a lively time, participate in the community. It's only by getting everyone on the board that we'll be able to create a web of quality. 

To-do 50 Learn concepts ✓ So, what if you wanted to start using web integration testing on your project tomorrow? First of all, get familiar with the notions. If I've done my job well today, you can tick that one now. Then, try tools that appeal to you and your team. There are some well-known, go-to solutions, but they all depend on your preferred language and framework. Based on these first tries, evaluate RoI on your project. The global idea is to consider if the regression costs are higher than those of inserting a new testing tool and platform. Finally, since all this is quite in a lively time, participate in the community. It's only by getting everyone on the board that we'll be able to create a web of quality. 

To-do 50 Learn concepts ‣ Try tools ✓ So, what if you wanted to start using web integration testing on your project tomorrow? First of all, get familiar with the notions. If I've done my job well today, you can tick that one now. Then, try tools that appeal to you and your team. There are some well-known, go-to solutions, but they all depend on your preferred language and framework. Based on these first tries, evaluate RoI on your project. The global idea is to consider if the regression costs are higher than those of inserting a new testing tool and platform. Finally, since all this is quite in a lively time, participate in the community. It's only by getting everyone on the board that we'll be able to create a web of quality. 

To-do 50 Learn concepts ‣ Try tools ‣ Evaluate RoI ✓ So, what if you wanted to start using web integration testing on your project tomorrow? First of all, get familiar with the notions. If I've done my job well today, you can tick that one now. Then, try tools that appeal to you and your team. There are some well-known, go-to solutions, but they all depend on your preferred language and framework. Based on these first tries, evaluate RoI on your project. The global idea is to consider if the regression costs are higher than those of inserting a new testing tool and platform. Finally, since all this is quite in a lively time, participate in the community. It's only by getting everyone on the board that we'll be able to create a web of quality. 

To-do 50 Learn concepts ‣ Try tools ‣ Evaluate RoI ‣ Give feedback ✓ So, what if you wanted to start using web integration testing on your project tomorrow? First of all, get familiar with the notions. If I've done my job well today, you can tick that one now. Then, try tools that appeal to you and your team. There are some well-known, go-to solutions, but they all depend on your preferred language and framework. Based on these first tries, evaluate RoI on your project. The global idea is to consider if the regression costs are higher than those of inserting a new testing tool and platform. Finally, since all this is quite in a lively time, participate in the community. It's only by getting everyone on the board that we'll be able to create a web of quality. 

Thanks! • Image credits • From The Noun Project • Code: d • Gear: Johan H. W. Basberg • Cursor: Yellow • All software logos © their respective owners • Thanks • Nicolas Dupont • Thomas De Bona • Fabien Massol • Chloé Chevalier • Anouchka Labonne • Gilles Fabio • Watai sponsors 51 Matti Schneider — @matti_sg github.com/MattiSG/Watai Thank you for your attention. Additional resources: • Functional GUI Testing Automation Patterns: a bit of theory of testing • 5 Reasons I Chose Selenium over PhantomJS • Running Headless Selenium with Chrome with a VM and Vagrant: get your lab ready quickly • Design Patterns for Customer Testing — Misha Rybalov, Quality Centered Developer: excellent abstract patterns • Compatibility/Web Testing wiki at Mozilla: a list of tools obtained through a poll by Karl Dubost on his audience • The introduction of WebDriver (2009)