State of Web Integration Testing

State of web
integration testing
Where we are now, and where we should go
1
Matti Schneider
@matti_sg
UNDERLINED WORDS IN TRANSCRIPT MEAN YOU SHOULD PASS TO THE NEXT SLIDE.
Good afternoon everyone, and welcome to this session!
I'll be talking about the current state of web integration testing: what it is exactly, which
practices it covers, how you could do it today, the limitations you'll probably face, and how
we should be doing it in the future.

View Slide

2
Matti Schneider
@matti_sg
My name is Matti Schneider.
The content in this presentation is based on 5 months of full-time research on the topic I did last year, and the
development of a web integration testing framework, Watai, that basically aimed at going beyond the limitations
shared by all the tools that were available at the time.
It also includes experience gained during the last months where I've been working at MESH, a startup targeting
surgery services. In this context, our web application's quality is not simply a matter of customer satisfaction, it is
absolutely critical for the daily work of health professionals.
Finally, please note that I'll be mostly taking the point of view of a development team wanting to do automated
testing, rather than of a specialized QA team, since that's where my own experience is coming from.
That's where I'm talking from.

View Slide

2
Matti Schneider
github.com/MattiSG/Watai
@matti_sg
My name is Matti Schneider.
The content in this presentation is based on 5 months of full-time research on the topic I did last year, and the
development of a web integration testing framework, Watai, that basically aimed at going beyond the limitations
shared by all the tools that were available at the time.
It also includes experience gained during the last months where I've been working at MESH, a startup targeting
surgery services. In this context, our web application's quality is not simply a matter of customer satisfaction, it is
absolutely critical for the daily work of health professionals.
Finally, please note that I'll be mostly taking the point of view of a development team wanting to do automated
testing, rather than of a specialized QA team, since that's where my own experience is coming from.
That's where I'm talking from.

View Slide

3
Vocabulary
We need to share the same language. If you've read about the subject, you might have
noticed the terminology depends on the author.

View Slide

4
The ﬁrst level, the one that is closest to the code, is unit testing.
Unit testing is concerned with the smallest piece of code that contains business logic.
Usually, it means methods or functions.
In the context of web applications, the tested “units” may reside either on the server or the
client-side. Both parts should be unit-tested if you're to be serious about quality.

View Slide

4
unit
The ﬁrst level, the one that is closest to the code, is unit testing.
Unit testing is concerned with the smallest piece of code that contains business logic.
Usually, it means methods or functions.
In the context of web applications, the tested “units” may reside either on the server or the
client-side. Both parts should be unit-tested if you're to be serious about quality.

View Slide

5
unit
Testing parts of the programs that implement independant features.
The exact “part” depends on the architecture, the environment, the size of the system… May be
classes, modules, or whole parts in distributed architectures.
In the context of web applications, this could be for example a JSON endpoint or a frontend
component.

View Slide

5
unit
functional
Testing parts of the programs that implement independant features.
The exact “part” depends on the architecture, the environment, the size of the system… May be
classes, modules, or whole parts in distributed architectures.
In the context of web applications, this could be for example a JSON endpoint or a frontend
component.

View Slide

acceptance
functional
system
validation
customer
end-to-end
6
unit
functional
Testing level in which individual software modules are tested as a group. It also means
testing through the same access means as the ﬁnal users will.
In the context of web applications, it means testing the result of combining client- and
server-side code, and accessing it through a browser.

View Slide

acceptance
functional
system
validation
customer
end-to-end
6
unit
functional
integration
Testing level in which individual software modules are tested as a group. It also means
testing through the same access means as the ﬁnal users will.
In the context of web applications, it means testing the result of combining client- and
server-side code, and accessing it through a browser.

View Slide

7
customer
end-to-end
acceptance
validation
system
functional
unit
functional
integration
Many synonyms for the same reality.
The most problematic: “functional”. Some authors list only “functional testing” and “unit testing”. It's up to you, but I
personally prefer differentiating. In this presentation, we'll only refer to integration testing.
“Customer”, “end-to-end”: a bit dated.
“System”: includes data.
“Validation”, “Acceptance”: performed by the customer. May detect performance, usability or data issues.
Let's see what this “reality” really looks like.

View Slide

8
[video demonstration of automated integration tests of the DuckDuckGo engine running]

View Slide

9
acceptance
functional
system
validation
customer
end-to-end
unit
functional
integration
So, that was an example of what is a web integration test.
Let's make sure we also agree on the goals of such a practice.

View Slide

10
Web integration testing's goals are the following.
First, ensure that a functionality, a feature that adds customer value, is present.
Second, proving that this functionality behaves properly. That is, given a set of inputs, and a
navigation path, the expected information is outputted to the user.
Finally, in the context of web applications, there's one important evil we're ﬁghting against:
platform discrepancies. So, we're not only concerned about functionalities being present, nor
behaving properly, but about them behaving properly in all browsers.
Now, how do we do this?

View Slide

functionality
10

View Slide

functionality
10
correctness

View Slide

functionality
10
correctness
consistency

View Slide

Tools
11
Important problem against adoption: language and articles not clear.
There is no “tool”, only a complex tool stack.
Seeing the history of the current web automation stack is probably the best way to
understand it properly.

View Slide

12
Super
Webapp
So, let's say you have this great web application, almost ready for production, you want to
test.

View Slide

13
Super
Webapp
You want to test this application against all browsers your users could use to get access to it.

View Slide

14
Super
Webapp
Not only do you want to test against all these browsers, you actually want to test against all
versions of them your users could be using.

View Slide

15
That's a lot of browsers.
Luckily, there's one thing developers do well: adding abstraction.

View Slide

16
Browser
controller
Two tools went this way: Sahi, and Selenium. We'll focus on Selenium, as Sahi is more
directed at professional QA teams, and Selenium at developers. It would deﬁnitely be worth a
look if you can afford having a dedicated testing team though.

View Slide

17
Selenium is a library to automate browsers, started back in 2004. Its original API is in Java.
Java was sexy at the time, but other languages started getting used for web projects. And
teams wanted tools in the language they were using for the main application development
instead of changing context all the time.
So, what did developers do? They added a layer of abstraction!

View Slide

Java
Ruby
.NET
Node.js
PHP
18
Now, you could write your tests in your preferred language through bindings for the Selenium API, or through JSON over HTTP.
However, the Java API was just one part of Selenium. The other end, the one automating browsers, was a JavaScript library
injected in each browser, that would simulate user interaction.
Over time, JavaScript security concerns were raised, differences with native capabilities grew wider, and simulating user behavior
became harder.
Luckily, advanced automation was possible for each browser, individually. For example, Firefox exposes an automation API. IE is
automatable through Windows accessibility services, and so on. But then you'd lose this ability to write the same code for all
browsers…
So, what did we do? Add another layer!

View Slide

Java
Ruby
.NET
Node.js
PHP
Safari
driver
IE
driver
Chrome
driver
Firefox
driver
Opera
driver
19
A project already implemented such an approach: WebDriver. It was merged with the JavaScript Selenium library
(Selenium Remote Control) in 2009, giving birth to Selenium 2.0.
Actually, if you read about Selenium, Selenium 2.0 or WebDriver today, it's most usually referring to the same thing, that
very stack.
Which is a great stack for automation. But even though it has a checkbox in its logo, Selenium is only a browser
automation tool, not a testing tool. So, what's a developer to do? Add abstraction!

View Slide

Java
Ruby
.NET
Node.js
PHP
Safari
driver
IE
driver
Chrome
driver
Firefox
driver
Opera
driver
Watir
Watij
Watin
Intern
Mink
20
So, libraries helping with testing were developed over all bindings, bundling assertions, runners… Now
we have a great tool for integration testing!
Then the second great browsers' arms race started, and updates to the browsers became much more
frequent. What happens then if a browser's updated, for example, and it breaks compatibility with one
of the tools at the other end of that stack?
That's when you start realizing it's slow and heavy.
So developers did the second thing they do best: restart from scratch.

View Slide

21
PhantomJS was created about two years ago by Ariya Hidayat.

View Slide

22
It is a headless (hence the “phantom” part of its name) WebKit engine. That is, you start it up
through the command-line, and it doesn't show any graphical user interface at all.
Of course, interacting with a browser you can't see is kind of hard. So, PhantomJS is also a
JavaScript (hence the “JS” part) API to control that browser.
That's a great, fast and sexy tool for browser automation! But once again, not dedicated
speciﬁcally to testing.

View Slide

23
So Nicolas Perriault created CasperJS, a set of JavaScript scripting and testing utilities.
Now we have a great stack for web integration testing! Or maybe not.
It is very important to understand that even though this stack is lovely for developers for its simplicity and speed, it can not be
considered an actual integration testing stack. Indeed, unless your users are accessing your application through a browser that
doesn't have an interface, which I doubt, you are not testing what your users will get.
The engine can differ (WebKit is now used only in Safari) but, beyond the rendering engine, the JavaScript engine is different too.
We've experienced ﬁrst hand that tests passing under Phantom can fail in actual browsers. Or the other way around.

View Slide

24
Still, that stack is attractive. So much that last month, Laurent Jouanneau created SlimerJS, a
project aiming to reproduce what Phantom did for WebKit, but for the Gecko engine.
It is compatible with Phantom's API, making Casper able to control both.

View Slide

Java
Ruby
.NET
Node.js
PHP
Safari
driver
IE
driver
Chrome
driver
Firefox
driver
Opera
driver
Watir
Watij
Watin
wd
Mink
25
If this new stack starts reminding you of the earlier jellyﬁsh, it might not be an entire coincidence.
Actually, we may as well accept that all browsers are different, and that automating them requires a speciﬁc strategy for
each. And that these strategies should be abstracted, as well as the languages the tests are written in. What will happen
when JavaScript is not hype anymore, and that we have a huge legacy of Casper-written tests? Will we write
CasperClojure?
And now, the two stacks are even starting to communicate, for example with GhostDriver by Ivan De Marino, which
allows Selenium to control Phantom.

View Slide

Java
Ruby
.NET
Node.js
PHP
Safari
driver
IE
driver
Chrome
driver
Firefox
driver
Opera
driver
Watir
Watij
Watin
wd
Mink
Ghost
driver
25
If this new stack starts reminding you of the earlier jellyﬁsh, it might not be an entire coincidence.
Actually, we may as well accept that all browsers are different, and that automating them requires a speciﬁc strategy for
each. And that these strategies should be abstracted, as well as the languages the tests are written in. What will happen
when JavaScript is not hype anymore, and that we have a huge legacy of Casper-written tests? Will we write
CasperClojure?
And now, the two stacks are even starting to communicate, for example with GhostDriver by Ivan De Marino, which
allows Selenium to control Phantom.

View Slide

Complexity
26
You've understood that this tool's problem is not its power, it's its complexity.
Luckily, there are solutions to hide this complexity. We'll see two examples of tools that do.

View Slide

27
Capybara
Capybara is packaged as a gem, and very well-known in the Ruby / Rails community.
It glues all the components listed earlier together, so that you don't have to install them
manually.
It also bundles some test parsers.

View Slide

28
This is a good time to have a slightly off-topic note. I'd like to address one common misconception about
Cucumber, a Ruby tool.
Cucumber often comes with Capybara, and one can easily read advice about doing web integration testing with it.
Unfortunately, this is a stretch, and this stretch hurts adoption by tying the practice to a speciﬁc format that is not
always liked.
Cucumber is only a behavior-driven development (BDD) framework, not an integration testing tool in itself.

View Slide

28
BDD framework
This is a good time to have a slightly off-topic note. I'd like to address one common misconception about
Cucumber, a Ruby tool.
Cucumber often comes with Capybara, and one can easily read advice about doing web integration testing with it.
Unfortunately, this is a stretch, and this stretch hurts adoption by tying the practice to a speciﬁc format that is not
always liked.
Cucumber is only a behavior-driven development (BDD) framework, not an integration testing tool in itself.

View Slide

29
Given-When-Then
This means that Cucumber is able, through transformation rules you have to deﬁne, to transform feature descriptions written in
a pseudo-English language (Gherkin), emblematically composed of Given-When-Then parts, in testing code. This testing code
can be anything, including unit tests.
Then, indeed, if you add the Watir (Web Integration Testing In Ruby) API in the testing code, you are able to control Selenium
and, in the end, to automate browsers for testing.
So, while it is not technically wrong to say so, remember that doing “web integration testing with Cucumber” is quite a stretch,
and that the tool itself is not speciﬁc to integration testing, even less so a requirement.

View Slide

29
• –
• –
• –
Given-When-Then
Testing code

View Slide

29
• –
• –
• –
Given-When-Then
Testing code
Browser
automation

View Slide

30
An alternative to Capybara is The Intern, a framework made available a few months ago.
It is also a “big piece of glue”, packaging a Selenium-controlling library, assertion libraries, test runners…
but, this time, making you write tests in JavaScript instead of Ruby.
Additionnally, it does a very cool thing: it dispatches both your integration tests and frontend unit tests
across all the browsers you're targeting.
Let's now see how to actually run such tests across all these browsers.

View Slide

Environment
31
As we've seen in our tool stack tour, testing on all browsers is mandatory, and is done by
automating those browsers.
But, it also means that all of them have to be somehow available for us to automate. This is
where it's getting worse: beyond the stack, the execution environment also plays a role.

View Slide

Local testing
32
The easiest way to get access to browsers is to use the ones available on your development machine.
As we've seen, to automate them, the corresponding drivers have to be installed too.
Finally, a Selenium server has to be available to communicate your instructions to these drivers. This server comes as a Java executable archive.
Then, you simply have to start that service through a line of command to open a Selenium server on some local port and direct your testing framework at that
local port to use all browsers available on the local system.
However, you're limited by your operating system. For example, you cannot test Safari on non-OSX, nor IE on non-Windows. Also, browsers do not always
behave exactly the same on different platforms. As we've experienced ﬁrst-hand, for example, a Firefox on OS X does not have the same bugs than the same
version of Firefox on Windows…
To tackle that problem, luckily, we can bring our old friend abstraction!

View Slide

Local testing
32
drivers
+

View Slide

Local testing
32
drivers
+ +

View Slide

Local testing
32
java -jar selenium-server-standalone-2.35.0.jar -p 4444
drivers
+ +

View Slide

Virtualized
33
Local testing
If you use virtual machines, you may use as many browser/platform combinations you want.
However, all these OSs loaded alongside will quickly bring your development machine to its knees. So, you'll need
servers… and then, you'll have to maintain the associated infrastructure and, before you know it, you're paying for your
own QA lab, which will probably end up costing you more than the beneﬁts of testing automatically.
This observation of course depends on your size. If you can afford having an infrastructure-dedicated team and already
have a QA team, then it may deﬁnitely be worth it.
Otherwise, if your team is mostly developers, you'll probably be better off externalizing these costs.

View Slide

34
Continuous integration
The duration of integration tests really makes them well-suited for continuous integration (CI).
Hosted CI is a really great thing, and many services will provide you with a Selenium server, drivers and the latest browsers out-
of-the-box. We're for example using CircleCI, a startup created at the end of 2012, and are really happy with their service.
However, no matter the tool, you will still be limited to the OS your tests are running in (usually a Linux box), and you can't
control the stack. So, you get the latest browsers… always. Even when a browser updates, and the driver is updated, and your
tests are not compatible with that latest version, and it is two days before a deadline.
Obviously, what you gain in ease of use is lost in control.

View Slide

35
Reliability
As you have understood with the impact of the environment, the second pain in web
integration testing is its reliability.
And unstable tests are worse than no tests at all: it leads to false conﬁdence, and to ignore
actual failures.
Luckily, there's once again a solution to mitigate this lack of reliability.

View Slide

36
SauceLabs is a startup that aims to solve this exact problem. It maintains VMs with all OS/browser
combinations, created on demand and available over the internet through a Selenium endpoint.
Most importantly, they offer a no-brainer tunnel, so you can easily test an application locally without
deploying it over the internet.
Using their service is not exclusive at all of hosted continuous integration services. Actually, we are using
both in combination, and that works really well.

View Slide

APIs
37
We've seen the concepts, the environment, and some tools that seem to be doing the work.
Let's now have a quick look at how you would actually write tests with them!

View Slide

38
return this.remote
.get(require.toUrl(url))
.elementById('new-todo')
.clickElement()
.keys('Task 1')
.keys("\n")
.keys('Task 2')
.keys("\n")
.keys('Task 3')
.getAttribute('value')
.then(function(val) {
assert.ok(val.indexOf('Task 3') > -1);
})
scenario "Signing in with correct credentials" do
page.visit "/sessions/new"
page.fill_in "email", :with => "[email protected]"
page.fill_in "password", :with => "qwerty"
page.click_button "Login"
page.should have_content("Welcome, [email protected]!")
end
Intern
Capybara
Here are small code excerpts from standard example tests written with the two tools I presented
earlier.
The goal is not that you look at them in details. If you want to try them, you will do so on your
own.
What I want you to note now is the global ﬂow, what the code says.
It's all about visiting an URL, walking the DOM to select an element, sending clicks and keystroke
events, getting HTML attributes…

View Slide

Old web
39
Basically, the current tools for web testing target the old web. The web of websites, not the one of web applications. If you're developing a webapp
with a serious frontend today, you don't think about pages. You think about components and APIs.
I don't mean to diminish these tools. They do a great work. But they are not in phase with the current practices of web development. Do you really
want to be walking the DOM in your tests while you have an object layer in your main code? You'd never do that in unit tests. That makes your tests
fragile, would it be only by forcing white-box testing on you.
As long as the testing tools are stuck behind like this, testing will be painful. So, today, I'd like to start a discussion on what should be the future of
web integration testing. Here is a proposal based on my own research and experiments.

View Slide

40
Component Fixture
• —
• —
• —
• —
Scenario
We're designing web applications through components, pieces of content that present the user with graphical elements
and behaviors triggered by interaction with these elements. What about modeling all of that for automation?
Then, we would write usage scenarios, that would deﬁne both how those components are expected to interact and their
status at any point through interactions.
Finally, we should maximize the maintainability of our tests by extracting everything that is different from this logic,
such as speciﬁc data or user messages, so that changing a phrasing does not break our testing ability.
Let's see how tests using such an architecture could actually be written.

View Slide

Component
41
The introductory video we saw at the beginning of this presentation checked the presence of a disambiguation infobox on an
ambiguous query on the DuckDuckGo search engine. We will follow this example.
As we've seen, we'd first model the components. Two are relevant to this test: the search bar itself, and the expected infobox.
They both have elements, which are easily definable through a name describing their usage, mapped to CSS selectors to locate
them. The search bar, though, also presents users with a possible action, made of combining simple interactions with its
different elements. This has to be written with code, here in JavaScript.
Now that the needed components have been defined, let's specify how they are expected to interact.

View Slide

Component
41
SearchBar
field: 'input[name=q]',
submitButton: '#search_button_homepage',
searchFor: function searchFor(term) {
return this.setField(term)()
.then(this.submit());
}
InfoBox
header: '#zero_click_heading'
The introductory video we saw at the beginning of this presentation checked the presence of a disambiguation infobox on an
ambiguous query on the DuckDuckGo search engine. We will follow this example.
As we've seen, we'd first model the components. Two are relevant to this test: the search bar itself, and the expected infobox.
They both have elements, which are easily definable through a name describing their usage, mapped to CSS selectors to locate
them. The search bar, though, also presents users with a possible action, made of combining simple interactions with its
different elements. This has to be written with code, here in JavaScript.
Now that the needed components have been defined, let's specify how they are expected to interact.

View Slide

42
Scenario
We first write a short human-readable description of the feature we're testing.
Then, this feature is tested through a scenario. This scenario relies on the defined
components to both act on them and define their expected status.

View Slide

42
Scenario
Toto
= Toto ?
InfoBoxFeature
description:
'Looking up an ambiguous term should
present an info box.',
scenario: [
SearchBarWidget.searchFor(lookupTerm),
{
'ZeroClickWidget.header':
'Meanings of ' + lookupTerm
}
]
We first write a short human-readable description of the feature we're testing.
Then, this feature is tested through a scenario. This scenario relies on the defined
components to both act on them and define their expected status.

View Slide

43
Fixture
• —
• —
• —
• —
Finally, the data that is used to trigger the behavior (here, the term considered as
“ambiguous” enough to display a disambiguation infobox) is injected into the test.

View Slide

Toto
43
Fixture
SearchFixture
lookupTerm = "Toto"
• —
• —
• —
• —
Finally, the data that is used to trigger the behavior (here, the term considered as
“ambiguous” enough to display a disambiguation infobox) is injected into the test.

View Slide

44
Component Scenario Fixture
• —
• —
• —
• —
The code you've just seen is taken from the first example bundled with the testing framework I wrote, Watai.
Now, I won't lie to you: Watai is currently considered in beta. We've been using it in production successfully for the last 11
months, but it still has some rough edges on syntax and ease-of-use. If the example appealed to you, the best is that you give it
a quick try.
The end vision would be that every Web Component delivered includes its automation description. You can view this as a
“human API specification”, just like components are expected to have unit tests to check their developer API. This would allow
black-box testing of web applications, even when they are made of the combination of several third-party components with
specific code, just like we do for other levels of testing.

View Slide

Recap
45
Let's review what we've seen today.

View Slide

46
We've seen what is web integration testing. It is about simulating user interaction across all
target browsers. It can be done manually, but it is very costly. We want to automate it.

View Slide

47
The current automation stack is ugly. But there's no real way to get away from the fact that
each browser is different, that we need to abstract away their differences, and that we also
need to abstract away from implementation languages.
There are tools that bundle all the stack so that you don't have to maintain it yourself. Find
the one that suits your particular language / framework / methodology.

View Slide

48
Maintaining your own lab is costly. Running tests is slow. The best solution is probably to run
“smoke tests” locally while developing on whichever browser (possibly headless) is available
on your dev machines, and delegate testing across target browsers to a hosted CI tool.

View Slide

49
• —
• —
• —
• —
Finally, we've seen that all current popular frameworks have a common weakness: they are
not built for the modern web. The alternative solutions are still under development.

View Slide

To-do
50
So, what if you wanted to start using web integration testing on your project tomorrow?
First of all, get familiar with the notions. If I've done my job well today, you can tick that one now.
Then, try tools that appeal to you and your team. There are some well-known, go-to solutions, but they all depend on your preferred language and
framework.
Based on these ﬁrst tries, evaluate RoI on your project. The global idea is to consider if the regression costs are higher than those of inserting a
new testing tool and platform.
Finally, since all this is quite in a lively time, participate in the community. It's only by getting everyone on the board that we'll be able to create a
web of quality.

View Slide

To-do
50
Learn concepts
‣
framework.
web of quality.

View Slide

To-do
50
Learn concepts
✓
framework.
web of quality.

View Slide

To-do
50
Learn concepts
‣ Try tools
✓
framework.
web of quality.

View Slide

To-do
50
Learn concepts
‣ Try tools
‣ Evaluate RoI
✓
framework.
web of quality.

View Slide

To-do
50
Learn concepts
‣ Try tools
‣ Evaluate RoI
‣ Give feedback
✓
framework.
web of quality.

View Slide

Thanks!
• Image credits
• From The Noun Project
• Code: d
• Gear: Johan H. W. Basberg
• Cursor: Yellow
• All software logos © their
respective owners
• Thanks
• Nicolas Dupont
• Thomas De Bona
• Fabien Massol
• Chloé Chevalier
• Anouchka Labonne
• Gilles Fabio
• Watai sponsors
51
Matti Schneider — @matti_sg
Thank you for your attention.
Additional resources:
• Functional GUI Testing Automation Patterns: a bit of theory of testing
• 5 Reasons I Chose Selenium over PhantomJS
• Running Headless Selenium with Chrome with a VM and Vagrant: get your lab ready quickly
• Design Patterns for Customer Testing — Misha Rybalov, Quality Centered Developer: excellent abstract patterns
• Compatibility/Web Testing wiki at Mozilla: a list of tools obtained through a poll by Karl Dubost on his audience
• The introduction of WebDriver (2009)

View Slide

State of Web Integration Testing

State of Web Integration Testing

Matti Schneider

More Decks by Matti Schneider

Other Decks in Programming

Featured

Transcript