Upgrade to Pro — share decks privately, control downloads, hide ads and more …

State of Web Integration Testing

State of Web Integration Testing

Video of talk available at http://www.youtube.com/watch?v=ioh8eFd_nEo

Automated web integration testing is very powerful, yet still a messy business. This session aims at making this messy ground clearer through proper definitions, an overview of the state of the art, presentation of current best practices, and a reflection on what this domain is still missing to reveal its full strength.

You will learn from the experience gained through 5 months of full-time research and development on the topic, and one year of usage in a continuous deployment environment, with fully automated cross-browser test suites being run in the cloud on every push.

By the end of the presentation, you will know exactly what is (and what isn't) integration testing on the web, what you can expect from it, how your production pipeline could integrate it, and which difficulties we as a community should focus on solving.

Matti Schneider

September 13, 2013
Tweet

More Decks by Matti Schneider

Other Decks in Programming

Transcript

  1. State of web
    integration testing
    Where we are now, and where we should go
    1
    Matti Schneider
    @matti_sg
    UNDERLINED WORDS IN TRANSCRIPT MEAN YOU SHOULD PASS TO THE NEXT SLIDE.
    Good afternoon everyone, and welcome to this session!
    I'll be talking about the current state of web integration testing: what it is exactly, which
    practices it covers, how you could do it today, the limitations you'll probably face, and how
    we should be doing it in the future.

    View Slide

  2. 2
    Matti Schneider
    @matti_sg
    My name is Matti Schneider.
    The content in this presentation is based on 5 months of full-time research on the topic I did last year, and the
    development of a web integration testing framework, Watai, that basically aimed at going beyond the limitations
    shared by all the tools that were available at the time.
    It also includes experience gained during the last months where I've been working at MESH, a startup targeting
    surgery services. In this context, our web application's quality is not simply a matter of customer satisfaction, it is
    absolutely critical for the daily work of health professionals.
    Finally, please note that I'll be mostly taking the point of view of a development team wanting to do automated
    testing, rather than of a specialized QA team, since that's where my own experience is coming from.
    That's where I'm talking from.

    View Slide

  3. 2
    Matti Schneider
    github.com/MattiSG/Watai
    @matti_sg
    My name is Matti Schneider.
    The content in this presentation is based on 5 months of full-time research on the topic I did last year, and the
    development of a web integration testing framework, Watai, that basically aimed at going beyond the limitations
    shared by all the tools that were available at the time.
    It also includes experience gained during the last months where I've been working at MESH, a startup targeting
    surgery services. In this context, our web application's quality is not simply a matter of customer satisfaction, it is
    absolutely critical for the daily work of health professionals.
    Finally, please note that I'll be mostly taking the point of view of a development team wanting to do automated
    testing, rather than of a specialized QA team, since that's where my own experience is coming from.
    That's where I'm talking from.

    View Slide

  4. 2
    Matti Schneider
    github.com/MattiSG/Watai
    @matti_sg
    My name is Matti Schneider.
    The content in this presentation is based on 5 months of full-time research on the topic I did last year, and the
    development of a web integration testing framework, Watai, that basically aimed at going beyond the limitations
    shared by all the tools that were available at the time.
    It also includes experience gained during the last months where I've been working at MESH, a startup targeting
    surgery services. In this context, our web application's quality is not simply a matter of customer satisfaction, it is
    absolutely critical for the daily work of health professionals.
    Finally, please note that I'll be mostly taking the point of view of a development team wanting to do automated
    testing, rather than of a specialized QA team, since that's where my own experience is coming from.
    That's where I'm talking from.

    View Slide

  5. 3
    Vocabulary
    We need to share the same language. If you've read about the subject, you might have
    noticed the terminology depends on the author.

    View Slide

  6. 4
    The first level, the one that is closest to the code, is unit testing.
    Unit testing is concerned with the smallest piece of code that contains business logic.
    Usually, it means methods or functions.
    In the context of web applications, the tested “units” may reside either on the server or the
    client-side. Both parts should be unit-tested if you're to be serious about quality.

    View Slide

  7. 4
    unit
    The first level, the one that is closest to the code, is unit testing.
    Unit testing is concerned with the smallest piece of code that contains business logic.
    Usually, it means methods or functions.
    In the context of web applications, the tested “units” may reside either on the server or the
    client-side. Both parts should be unit-tested if you're to be serious about quality.

    View Slide

  8. 5
    unit
    Testing parts of the programs that implement independant features.
    The exact “part” depends on the architecture, the environment, the size of the system… May be
    classes, modules, or whole parts in distributed architectures.
    In the context of web applications, this could be for example a JSON endpoint or a frontend
    component.

    View Slide

  9. 5
    unit
    functional
    Testing parts of the programs that implement independant features.
    The exact “part” depends on the architecture, the environment, the size of the system… May be
    classes, modules, or whole parts in distributed architectures.
    In the context of web applications, this could be for example a JSON endpoint or a frontend
    component.

    View Slide

  10. acceptance
    functional
    system
    validation
    customer
    end-to-end
    6
    unit
    functional
    Testing level in which individual software modules are tested as a group. It also means
    testing through the same access means as the final users will.
    In the context of web applications, it means testing the result of combining client- and
    server-side code, and accessing it through a browser.

    View Slide

  11. acceptance
    functional
    system
    validation
    customer
    end-to-end
    6
    unit
    functional
    integration
    Testing level in which individual software modules are tested as a group. It also means
    testing through the same access means as the final users will.
    In the context of web applications, it means testing the result of combining client- and
    server-side code, and accessing it through a browser.

    View Slide

  12. 7
    customer
    end-to-end
    acceptance
    validation
    system
    functional
    unit
    functional
    integration
    Many synonyms for the same reality.
    The most problematic: “functional”. Some authors list only “functional testing” and “unit testing”. It's up to you, but I
    personally prefer differentiating. In this presentation, we'll only refer to integration testing.
    “Customer”, “end-to-end”: a bit dated.
    “System”: includes data.
    “Validation”, “Acceptance”: performed by the customer. May detect performance, usability or data issues.
    Let's see what this “reality” really looks like.

    View Slide

  13. 8
    [video demonstration of automated integration tests of the DuckDuckGo engine running]

    View Slide

  14. 9
    acceptance
    functional
    system
    validation
    customer
    end-to-end
    unit
    functional
    integration
    So, that was an example of what is a web integration test.
    Let's make sure we also agree on the goals of such a practice.

    View Slide

  15. 10
    Web integration testing's goals are the following.
    First, ensure that a functionality, a feature that adds customer value, is present.
    Second, proving that this functionality behaves properly. That is, given a set of inputs, and a
    navigation path, the expected information is outputted to the user.
    Finally, in the context of web applications, there's one important evil we're fighting against:
    platform discrepancies. So, we're not only concerned about functionalities being present, nor
    behaving properly, but about them behaving properly in all browsers.
    Now, how do we do this?

    View Slide

  16. functionality
    10
    Web integration testing's goals are the following.
    First, ensure that a functionality, a feature that adds customer value, is present.
    Second, proving that this functionality behaves properly. That is, given a set of inputs, and a
    navigation path, the expected information is outputted to the user.
    Finally, in the context of web applications, there's one important evil we're fighting against:
    platform discrepancies. So, we're not only concerned about functionalities being present, nor
    behaving properly, but about them behaving properly in all browsers.
    Now, how do we do this?

    View Slide

  17. functionality
    10
    correctness
    Web integration testing's goals are the following.
    First, ensure that a functionality, a feature that adds customer value, is present.
    Second, proving that this functionality behaves properly. That is, given a set of inputs, and a
    navigation path, the expected information is outputted to the user.
    Finally, in the context of web applications, there's one important evil we're fighting against:
    platform discrepancies. So, we're not only concerned about functionalities being present, nor
    behaving properly, but about them behaving properly in all browsers.
    Now, how do we do this?

    View Slide

  18. functionality
    10
    correctness
    consistency
    Web integration testing's goals are the following.
    First, ensure that a functionality, a feature that adds customer value, is present.
    Second, proving that this functionality behaves properly. That is, given a set of inputs, and a
    navigation path, the expected information is outputted to the user.
    Finally, in the context of web applications, there's one important evil we're fighting against:
    platform discrepancies. So, we're not only concerned about functionalities being present, nor
    behaving properly, but about them behaving properly in all browsers.
    Now, how do we do this?

    View Slide

  19. Tools
    11
    Important problem against adoption: language and articles not clear.
    There is no “tool”, only a complex tool stack.
    Seeing the history of the current web automation stack is probably the best way to
    understand it properly.

    View Slide

  20. 12
    Super
    Webapp
    So, let's say you have this great web application, almost ready for production, you want to
    test.

    View Slide

  21. 13
    Super
    Webapp
    You want to test this application against all browsers your users could use to get access to it.

    View Slide

  22. 14
    Super
    Webapp
    Not only do you want to test against all these browsers, you actually want to test against all
    versions of them your users could be using.

    View Slide

  23. 15
    That's a lot of browsers.
    Luckily, there's one thing developers do well: adding abstraction.

    View Slide

  24. 15
    That's a lot of browsers.
    Luckily, there's one thing developers do well: adding abstraction.

    View Slide

  25. 16
    Browser
    controller
    Two tools went this way: Sahi, and Selenium. We'll focus on Selenium, as Sahi is more
    directed at professional QA teams, and Selenium at developers. It would definitely be worth a
    look if you can afford having a dedicated testing team though.

    View Slide

  26. 17
    Selenium is a library to automate browsers, started back in 2004. Its original API is in Java.
    Java was sexy at the time, but other languages started getting used for web projects. And
    teams wanted tools in the language they were using for the main application development
    instead of changing context all the time.
    So, what did developers do? They added a layer of abstraction!

    View Slide

  27. 17
    Selenium is a library to automate browsers, started back in 2004. Its original API is in Java.
    Java was sexy at the time, but other languages started getting used for web projects. And
    teams wanted tools in the language they were using for the main application development
    instead of changing context all the time.
    So, what did developers do? They added a layer of abstraction!

    View Slide

  28. Java
    Ruby
    .NET
    Node.js
    PHP
    18
    Now, you could write your tests in your preferred language through bindings for the Selenium API, or through JSON over HTTP.
    However, the Java API was just one part of Selenium. The other end, the one automating browsers, was a JavaScript library
    injected in each browser, that would simulate user interaction.
    Over time, JavaScript security concerns were raised, differences with native capabilities grew wider, and simulating user behavior
    became harder.
    Luckily, advanced automation was possible for each browser, individually. For example, Firefox exposes an automation API. IE is
    automatable through Windows accessibility services, and so on. But then you'd lose this ability to write the same code for all
    browsers…
    So, what did we do? Add another layer!

    View Slide

  29. Java
    Ruby
    .NET
    Node.js
    PHP
    18
    Now, you could write your tests in your preferred language through bindings for the Selenium API, or through JSON over HTTP.
    However, the Java API was just one part of Selenium. The other end, the one automating browsers, was a JavaScript library
    injected in each browser, that would simulate user interaction.
    Over time, JavaScript security concerns were raised, differences with native capabilities grew wider, and simulating user behavior
    became harder.
    Luckily, advanced automation was possible for each browser, individually. For example, Firefox exposes an automation API. IE is
    automatable through Windows accessibility services, and so on. But then you'd lose this ability to write the same code for all
    browsers…
    So, what did we do? Add another layer!

    View Slide

  30. Java
    Ruby
    .NET
    Node.js
    PHP
    Safari
    driver
    IE
    driver
    Chrome
    driver
    Firefox
    driver
    Opera
    driver
    19
    A project already implemented such an approach: WebDriver. It was merged with the JavaScript Selenium library
    (Selenium Remote Control) in 2009, giving birth to Selenium 2.0.
    Actually, if you read about Selenium, Selenium 2.0 or WebDriver today, it's most usually referring to the same thing, that
    very stack.
    Which is a great stack for automation. But even though it has a checkbox in its logo, Selenium is only a browser
    automation tool, not a testing tool. So, what's a developer to do? Add abstraction!

    View Slide

  31. Java
    Ruby
    .NET
    Node.js
    PHP
    Safari
    driver
    IE
    driver
    Chrome
    driver
    Firefox
    driver
    Opera
    driver
    19
    A project already implemented such an approach: WebDriver. It was merged with the JavaScript Selenium library
    (Selenium Remote Control) in 2009, giving birth to Selenium 2.0.
    Actually, if you read about Selenium, Selenium 2.0 or WebDriver today, it's most usually referring to the same thing, that
    very stack.
    Which is a great stack for automation. But even though it has a checkbox in its logo, Selenium is only a browser
    automation tool, not a testing tool. So, what's a developer to do? Add abstraction!

    View Slide

  32. Java
    Ruby
    .NET
    Node.js
    PHP
    Safari
    driver
    IE
    driver
    Chrome
    driver
    Firefox
    driver
    Opera
    driver
    Watir
    Watij
    Watin
    Intern
    Mink
    20
    So, libraries helping with testing were developed over all bindings, bundling assertions, runners… Now
    we have a great tool for integration testing!
    Then the second great browsers' arms race started, and updates to the browsers became much more
    frequent. What happens then if a browser's updated, for example, and it breaks compatibility with one
    of the tools at the other end of that stack?
    That's when you start realizing it's slow and heavy.
    So developers did the second thing they do best: restart from scratch.

    View Slide

  33. Java
    Ruby
    .NET
    Node.js
    PHP
    Safari
    driver
    IE
    driver
    Chrome
    driver
    Firefox
    driver
    Opera
    driver
    Watir
    Watij
    Watin
    Intern
    Mink
    20
    So, libraries helping with testing were developed over all bindings, bundling assertions, runners… Now
    we have a great tool for integration testing!
    Then the second great browsers' arms race started, and updates to the browsers became much more
    frequent. What happens then if a browser's updated, for example, and it breaks compatibility with one
    of the tools at the other end of that stack?
    That's when you start realizing it's slow and heavy.
    So developers did the second thing they do best: restart from scratch.

    View Slide

  34. Java
    Ruby
    .NET
    Node.js
    PHP
    Safari
    driver
    IE
    driver
    Chrome
    driver
    Firefox
    driver
    Opera
    driver
    Watir
    Watij
    Watin
    Intern
    Mink
    20
    So, libraries helping with testing were developed over all bindings, bundling assertions, runners… Now
    we have a great tool for integration testing!
    Then the second great browsers' arms race started, and updates to the browsers became much more
    frequent. What happens then if a browser's updated, for example, and it breaks compatibility with one
    of the tools at the other end of that stack?
    That's when you start realizing it's slow and heavy.
    So developers did the second thing they do best: restart from scratch.

    View Slide

  35. 21
    PhantomJS was created about two years ago by Ariya Hidayat.

    View Slide

  36. 22
    It is a headless (hence the “phantom” part of its name) WebKit engine. That is, you start it up
    through the command-line, and it doesn't show any graphical user interface at all.
    Of course, interacting with a browser you can't see is kind of hard. So, PhantomJS is also a
    JavaScript (hence the “JS” part) API to control that browser.
    That's a great, fast and sexy tool for browser automation! But once again, not dedicated
    specifically to testing.

    View Slide

  37. 23
    So Nicolas Perriault created CasperJS, a set of JavaScript scripting and testing utilities.
    Now we have a great stack for web integration testing! Or maybe not.
    It is very important to understand that even though this stack is lovely for developers for its simplicity and speed, it can not be
    considered an actual integration testing stack. Indeed, unless your users are accessing your application through a browser that
    doesn't have an interface, which I doubt, you are not testing what your users will get.
    The engine can differ (WebKit is now used only in Safari) but, beyond the rendering engine, the JavaScript engine is different too.
    We've experienced first hand that tests passing under Phantom can fail in actual browsers. Or the other way around.

    View Slide

  38. 24
    Still, that stack is attractive. So much that last month, Laurent Jouanneau created SlimerJS, a
    project aiming to reproduce what Phantom did for WebKit, but for the Gecko engine.
    It is compatible with Phantom's API, making Casper able to control both.

    View Slide

  39. Java
    Ruby
    .NET
    Node.js
    PHP
    Safari
    driver
    IE
    driver
    Chrome
    driver
    Firefox
    driver
    Opera
    driver
    Watir
    Watij
    Watin
    wd
    Mink
    25
    If this new stack starts reminding you of the earlier jellyfish, it might not be an entire coincidence.
    Actually, we may as well accept that all browsers are different, and that automating them requires a specific strategy for
    each. And that these strategies should be abstracted, as well as the languages the tests are written in. What will happen
    when JavaScript is not hype anymore, and that we have a huge legacy of Casper-written tests? Will we write
    CasperClojure?
    And now, the two stacks are even starting to communicate, for example with GhostDriver by Ivan De Marino, which
    allows Selenium to control Phantom.

    View Slide

  40. Java
    Ruby
    .NET
    Node.js
    PHP
    Safari
    driver
    IE
    driver
    Chrome
    driver
    Firefox
    driver
    Opera
    driver
    Watir
    Watij
    Watin
    wd
    Mink
    Ghost
    driver
    25
    If this new stack starts reminding you of the earlier jellyfish, it might not be an entire coincidence.
    Actually, we may as well accept that all browsers are different, and that automating them requires a specific strategy for
    each. And that these strategies should be abstracted, as well as the languages the tests are written in. What will happen
    when JavaScript is not hype anymore, and that we have a huge legacy of Casper-written tests? Will we write
    CasperClojure?
    And now, the two stacks are even starting to communicate, for example with GhostDriver by Ivan De Marino, which
    allows Selenium to control Phantom.

    View Slide

  41. Complexity
    26
    You've understood that this tool's problem is not its power, it's its complexity.
    Luckily, there are solutions to hide this complexity. We'll see two examples of tools that do.

    View Slide

  42. 27
    Capybara
    Capybara is packaged as a gem, and very well-known in the Ruby / Rails community.
    It glues all the components listed earlier together, so that you don't have to install them
    manually.
    It also bundles some test parsers.

    View Slide

  43. 28
    This is a good time to have a slightly off-topic note. I'd like to address one common misconception about
    Cucumber, a Ruby tool.
    Cucumber often comes with Capybara, and one can easily read advice about doing web integration testing with it.
    Unfortunately, this is a stretch, and this stretch hurts adoption by tying the practice to a specific format that is not
    always liked.
    Cucumber is only a behavior-driven development (BDD) framework, not an integration testing tool in itself.

    View Slide

  44. 28
    BDD framework
    This is a good time to have a slightly off-topic note. I'd like to address one common misconception about
    Cucumber, a Ruby tool.
    Cucumber often comes with Capybara, and one can easily read advice about doing web integration testing with it.
    Unfortunately, this is a stretch, and this stretch hurts adoption by tying the practice to a specific format that is not
    always liked.
    Cucumber is only a behavior-driven development (BDD) framework, not an integration testing tool in itself.

    View Slide

  45. 29
    Given-When-Then
    This means that Cucumber is able, through transformation rules you have to define, to transform feature descriptions written in
    a pseudo-English language (Gherkin), emblematically composed of Given-When-Then parts, in testing code. This testing code
    can be anything, including unit tests.
    Then, indeed, if you add the Watir (Web Integration Testing In Ruby) API in the testing code, you are able to control Selenium
    and, in the end, to automate browsers for testing.
    So, while it is not technically wrong to say so, remember that doing “web integration testing with Cucumber” is quite a stretch,
    and that the tool itself is not specific to integration testing, even less so a requirement.

    View Slide

  46. 29
    • –
    • –
    • –
    Given-When-Then
    Testing code
    This means that Cucumber is able, through transformation rules you have to define, to transform feature descriptions written in
    a pseudo-English language (Gherkin), emblematically composed of Given-When-Then parts, in testing code. This testing code
    can be anything, including unit tests.
    Then, indeed, if you add the Watir (Web Integration Testing In Ruby) API in the testing code, you are able to control Selenium
    and, in the end, to automate browsers for testing.
    So, while it is not technically wrong to say so, remember that doing “web integration testing with Cucumber” is quite a stretch,
    and that the tool itself is not specific to integration testing, even less so a requirement.

    View Slide

  47. 29
    • –
    • –
    • –
    Given-When-Then
    Testing code
    This means that Cucumber is able, through transformation rules you have to define, to transform feature descriptions written in
    a pseudo-English language (Gherkin), emblematically composed of Given-When-Then parts, in testing code. This testing code
    can be anything, including unit tests.
    Then, indeed, if you add the Watir (Web Integration Testing In Ruby) API in the testing code, you are able to control Selenium
    and, in the end, to automate browsers for testing.
    So, while it is not technically wrong to say so, remember that doing “web integration testing with Cucumber” is quite a stretch,
    and that the tool itself is not specific to integration testing, even less so a requirement.

    View Slide

  48. 29
    • –
    • –
    • –
    Given-When-Then
    Testing code
    Browser
    automation
    This means that Cucumber is able, through transformation rules you have to define, to transform feature descriptions written in
    a pseudo-English language (Gherkin), emblematically composed of Given-When-Then parts, in testing code. This testing code
    can be anything, including unit tests.
    Then, indeed, if you add the Watir (Web Integration Testing In Ruby) API in the testing code, you are able to control Selenium
    and, in the end, to automate browsers for testing.
    So, while it is not technically wrong to say so, remember that doing “web integration testing with Cucumber” is quite a stretch,
    and that the tool itself is not specific to integration testing, even less so a requirement.

    View Slide

  49. 30
    An alternative to Capybara is The Intern, a framework made available a few months ago.
    It is also a “big piece of glue”, packaging a Selenium-controlling library, assertion libraries, test runners…
    but, this time, making you write tests in JavaScript instead of Ruby.
    Additionnally, it does a very cool thing: it dispatches both your integration tests and frontend unit tests
    across all the browsers you're targeting.
    Let's now see how to actually run such tests across all these browsers.

    View Slide

  50. Environment
    31
    As we've seen in our tool stack tour, testing on all browsers is mandatory, and is done by
    automating those browsers.
    But, it also means that all of them have to be somehow available for us to automate. This is
    where it's getting worse: beyond the stack, the execution environment also plays a role.

    View Slide

  51. Local testing
    32
    The easiest way to get access to browsers is to use the ones available on your development machine.
    As we've seen, to automate them, the corresponding drivers have to be installed too.
    Finally, a Selenium server has to be available to communicate your instructions to these drivers. This server comes as a Java executable archive.
    Then, you simply have to start that service through a line of command to open a Selenium server on some local port and direct your testing framework at that
    local port to use all browsers available on the local system.
    However, you're limited by your operating system. For example, you cannot test Safari on non-OSX, nor IE on non-Windows. Also, browsers do not always
    behave exactly the same on different platforms. As we've experienced first-hand, for example, a Firefox on OS X does not have the same bugs than the same
    version of Firefox on Windows…
    To tackle that problem, luckily, we can bring our old friend abstraction!

    View Slide

  52. Local testing
    32
    The easiest way to get access to browsers is to use the ones available on your development machine.
    As we've seen, to automate them, the corresponding drivers have to be installed too.
    Finally, a Selenium server has to be available to communicate your instructions to these drivers. This server comes as a Java executable archive.
    Then, you simply have to start that service through a line of command to open a Selenium server on some local port and direct your testing framework at that
    local port to use all browsers available on the local system.
    However, you're limited by your operating system. For example, you cannot test Safari on non-OSX, nor IE on non-Windows. Also, browsers do not always
    behave exactly the same on different platforms. As we've experienced first-hand, for example, a Firefox on OS X does not have the same bugs than the same
    version of Firefox on Windows…
    To tackle that problem, luckily, we can bring our old friend abstraction!

    View Slide

  53. Local testing
    32
    drivers
    +
    The easiest way to get access to browsers is to use the ones available on your development machine.
    As we've seen, to automate them, the corresponding drivers have to be installed too.
    Finally, a Selenium server has to be available to communicate your instructions to these drivers. This server comes as a Java executable archive.
    Then, you simply have to start that service through a line of command to open a Selenium server on some local port and direct your testing framework at that
    local port to use all browsers available on the local system.
    However, you're limited by your operating system. For example, you cannot test Safari on non-OSX, nor IE on non-Windows. Also, browsers do not always
    behave exactly the same on different platforms. As we've experienced first-hand, for example, a Firefox on OS X does not have the same bugs than the same
    version of Firefox on Windows…
    To tackle that problem, luckily, we can bring our old friend abstraction!

    View Slide

  54. Local testing
    32
    drivers
    + +
    The easiest way to get access to browsers is to use the ones available on your development machine.
    As we've seen, to automate them, the corresponding drivers have to be installed too.
    Finally, a Selenium server has to be available to communicate your instructions to these drivers. This server comes as a Java executable archive.
    Then, you simply have to start that service through a line of command to open a Selenium server on some local port and direct your testing framework at that
    local port to use all browsers available on the local system.
    However, you're limited by your operating system. For example, you cannot test Safari on non-OSX, nor IE on non-Windows. Also, browsers do not always
    behave exactly the same on different platforms. As we've experienced first-hand, for example, a Firefox on OS X does not have the same bugs than the same
    version of Firefox on Windows…
    To tackle that problem, luckily, we can bring our old friend abstraction!

    View Slide

  55. Local testing
    32
    java -jar selenium-server-standalone-2.35.0.jar -p 4444
    drivers
    + +
    The easiest way to get access to browsers is to use the ones available on your development machine.
    As we've seen, to automate them, the corresponding drivers have to be installed too.
    Finally, a Selenium server has to be available to communicate your instructions to these drivers. This server comes as a Java executable archive.
    Then, you simply have to start that service through a line of command to open a Selenium server on some local port and direct your testing framework at that
    local port to use all browsers available on the local system.
    However, you're limited by your operating system. For example, you cannot test Safari on non-OSX, nor IE on non-Windows. Also, browsers do not always
    behave exactly the same on different platforms. As we've experienced first-hand, for example, a Firefox on OS X does not have the same bugs than the same
    version of Firefox on Windows…
    To tackle that problem, luckily, we can bring our old friend abstraction!

    View Slide

  56. Virtualized
    33
    Local testing
    If you use virtual machines, you may use as many browser/platform combinations you want.
    However, all these OSs loaded alongside will quickly bring your development machine to its knees. So, you'll need
    servers… and then, you'll have to maintain the associated infrastructure and, before you know it, you're paying for your
    own QA lab, which will probably end up costing you more than the benefits of testing automatically.
    This observation of course depends on your size. If you can afford having an infrastructure-dedicated team and already
    have a QA team, then it may definitely be worth it.
    Otherwise, if your team is mostly developers, you'll probably be better off externalizing these costs.

    View Slide

  57. Virtualized
    33
    Local testing
    If you use virtual machines, you may use as many browser/platform combinations you want.
    However, all these OSs loaded alongside will quickly bring your development machine to its knees. So, you'll need
    servers… and then, you'll have to maintain the associated infrastructure and, before you know it, you're paying for your
    own QA lab, which will probably end up costing you more than the benefits of testing automatically.
    This observation of course depends on your size. If you can afford having an infrastructure-dedicated team and already
    have a QA team, then it may definitely be worth it.
    Otherwise, if your team is mostly developers, you'll probably be better off externalizing these costs.

    View Slide

  58. 34
    Continuous integration
    The duration of integration tests really makes them well-suited for continuous integration (CI).
    Hosted CI is a really great thing, and many services will provide you with a Selenium server, drivers and the latest browsers out-
    of-the-box. We're for example using CircleCI, a startup created at the end of 2012, and are really happy with their service.
    However, no matter the tool, you will still be limited to the OS your tests are running in (usually a Linux box), and you can't
    control the stack. So, you get the latest browsers… always. Even when a browser updates, and the driver is updated, and your
    tests are not compatible with that latest version, and it is two days before a deadline.
    Obviously, what you gain in ease of use is lost in control.

    View Slide

  59. 35
    Reliability
    As you have understood with the impact of the environment, the second pain in web
    integration testing is its reliability.
    And unstable tests are worse than no tests at all: it leads to false confidence, and to ignore
    actual failures.
    Luckily, there's once again a solution to mitigate this lack of reliability.

    View Slide

  60. 36
    SauceLabs is a startup that aims to solve this exact problem. It maintains VMs with all OS/browser
    combinations, created on demand and available over the internet through a Selenium endpoint.
    Most importantly, they offer a no-brainer tunnel, so you can easily test an application locally without
    deploying it over the internet.
    Using their service is not exclusive at all of hosted continuous integration services. Actually, we are using
    both in combination, and that works really well.

    View Slide

  61. APIs
    37
    We've seen the concepts, the environment, and some tools that seem to be doing the work.
    Let's now have a quick look at how you would actually write tests with them!

    View Slide

  62. 38
    return this.remote
    .get(require.toUrl(url))
    .elementById('new-todo')
    .clickElement()
    .keys('Task 1')
    .keys("\n")
    .keys('Task 2')
    .keys("\n")
    .keys('Task 3')
    .getAttribute('value')
    .then(function(val) {
    assert.ok(val.indexOf('Task 3') > -1);
    })
    scenario "Signing in with correct credentials" do
    page.visit "/sessions/new"
    page.fill_in "email", :with => "[email protected]"
    page.fill_in "password", :with => "qwerty"
    page.click_button "Login"
    page.should have_content("Welcome, [email protected]!")
    end
    Intern
    Capybara
    Here are small code excerpts from standard example tests written with the two tools I presented
    earlier.
    The goal is not that you look at them in details. If you want to try them, you will do so on your
    own.
    What I want you to note now is the global flow, what the code says.
    It's all about visiting an URL, walking the DOM to select an element, sending clicks and keystroke
    events, getting HTML attributes…

    View Slide

  63. Old web
    39
    Basically, the current tools for web testing target the old web. The web of websites, not the one of web applications. If you're developing a webapp
    with a serious frontend today, you don't think about pages. You think about components and APIs.
    I don't mean to diminish these tools. They do a great work. But they are not in phase with the current practices of web development. Do you really
    want to be walking the DOM in your tests while you have an object layer in your main code? You'd never do that in unit tests. That makes your tests
    fragile, would it be only by forcing white-box testing on you.
    As long as the testing tools are stuck behind like this, testing will be painful. So, today, I'd like to start a discussion on what should be the future of
    web integration testing. Here is a proposal based on my own research and experiments.

    View Slide

  64. 40
    Component Fixture
    • —
    • —
    • —
    • —
    Scenario
    We're designing web applications through components, pieces of content that present the user with graphical elements
    and behaviors triggered by interaction with these elements. What about modeling all of that for automation?
    Then, we would write usage scenarios, that would define both how those components are expected to interact and their
    status at any point through interactions.
    Finally, we should maximize the maintainability of our tests by extracting everything that is different from this logic,
    such as specific data or user messages, so that changing a phrasing does not break our testing ability.
    Let's see how tests using such an architecture could actually be written.

    View Slide

  65. Component
    41
    The introductory video we saw at the beginning of this presentation checked the presence of a disambiguation infobox on an
    ambiguous query on the DuckDuckGo search engine. We will follow this example.
    As we've seen, we'd first model the components. Two are relevant to this test: the search bar itself, and the expected infobox.
    They both have elements, which are easily definable through a name describing their usage, mapped to CSS selectors to locate
    them. The search bar, though, also presents users with a possible action, made of combining simple interactions with its
    different elements. This has to be written with code, here in JavaScript.
    Now that the needed components have been defined, let's specify how they are expected to interact.

    View Slide

  66. Component
    41
    SearchBar
    field: 'input[name=q]',
    submitButton: '#search_button_homepage',
    searchFor: function searchFor(term) {
    return this.setField(term)()
    .then(this.submit());
    }
    InfoBox
    header: '#zero_click_heading'
    The introductory video we saw at the beginning of this presentation checked the presence of a disambiguation infobox on an
    ambiguous query on the DuckDuckGo search engine. We will follow this example.
    As we've seen, we'd first model the components. Two are relevant to this test: the search bar itself, and the expected infobox.
    They both have elements, which are easily definable through a name describing their usage, mapped to CSS selectors to locate
    them. The search bar, though, also presents users with a possible action, made of combining simple interactions with its
    different elements. This has to be written with code, here in JavaScript.
    Now that the needed components have been defined, let's specify how they are expected to interact.

    View Slide

  67. 42
    Scenario
    We first write a short human-readable description of the feature we're testing.
    Then, this feature is tested through a scenario. This scenario relies on the defined
    components to both act on them and define their expected status.

    View Slide

  68. 42
    Scenario
    Toto
    = Toto ?
    InfoBoxFeature
    description:
    'Looking up an ambiguous term should
    present an info box.',
    scenario: [
    SearchBarWidget.searchFor(lookupTerm),
    {
    'ZeroClickWidget.header':
    'Meanings of ' + lookupTerm
    }
    ]
    We first write a short human-readable description of the feature we're testing.
    Then, this feature is tested through a scenario. This scenario relies on the defined
    components to both act on them and define their expected status.

    View Slide

  69. 43
    Fixture
    • —
    • —
    • —
    • —
    Finally, the data that is used to trigger the behavior (here, the term considered as
    “ambiguous” enough to display a disambiguation infobox) is injected into the test.

    View Slide

  70. Toto
    43
    Fixture
    SearchFixture
    lookupTerm = "Toto"
    • —
    • —
    • —
    • —
    Finally, the data that is used to trigger the behavior (here, the term considered as
    “ambiguous” enough to display a disambiguation infobox) is injected into the test.

    View Slide

  71. 44
    Component Scenario Fixture
    • —
    • —
    • —
    • —
    github.com/MattiSG/Watai
    The code you've just seen is taken from the first example bundled with the testing framework I wrote, Watai.
    Now, I won't lie to you: Watai is currently considered in beta. We've been using it in production successfully for the last 11
    months, but it still has some rough edges on syntax and ease-of-use. If the example appealed to you, the best is that you give it
    a quick try.
    The end vision would be that every Web Component delivered includes its automation description. You can view this as a
    “human API specification”, just like components are expected to have unit tests to check their developer API. This would allow
    black-box testing of web applications, even when they are made of the combination of several third-party components with
    specific code, just like we do for other levels of testing.

    View Slide

  72. Recap
    45
    Let's review what we've seen today.

    View Slide

  73. 46
    We've seen what is web integration testing. It is about simulating user interaction across all
    target browsers. It can be done manually, but it is very costly. We want to automate it.

    View Slide

  74. 47
    The current automation stack is ugly. But there's no real way to get away from the fact that
    each browser is different, that we need to abstract away their differences, and that we also
    need to abstract away from implementation languages.
    There are tools that bundle all the stack so that you don't have to maintain it yourself. Find
    the one that suits your particular language / framework / methodology.

    View Slide

  75. 48
    Maintaining your own lab is costly. Running tests is slow. The best solution is probably to run
    “smoke tests” locally while developing on whichever browser (possibly headless) is available
    on your dev machines, and delegate testing across target browsers to a hosted CI tool.

    View Slide

  76. 49
    • —
    • —
    • —
    • —
    Finally, we've seen that all current popular frameworks have a common weakness: they are
    not built for the modern web. The alternative solutions are still under development.

    View Slide

  77. To-do
    50
    So, what if you wanted to start using web integration testing on your project tomorrow?
    First of all, get familiar with the notions. If I've done my job well today, you can tick that one now.
    Then, try tools that appeal to you and your team. There are some well-known, go-to solutions, but they all depend on your preferred language and
    framework.
    Based on these first tries, evaluate RoI on your project. The global idea is to consider if the regression costs are higher than those of inserting a
    new testing tool and platform.
    Finally, since all this is quite in a lively time, participate in the community. It's only by getting everyone on the board that we'll be able to create a
    web of quality.

    View Slide

  78. To-do
    50
    Learn concepts

    So, what if you wanted to start using web integration testing on your project tomorrow?
    First of all, get familiar with the notions. If I've done my job well today, you can tick that one now.
    Then, try tools that appeal to you and your team. There are some well-known, go-to solutions, but they all depend on your preferred language and
    framework.
    Based on these first tries, evaluate RoI on your project. The global idea is to consider if the regression costs are higher than those of inserting a
    new testing tool and platform.
    Finally, since all this is quite in a lively time, participate in the community. It's only by getting everyone on the board that we'll be able to create a
    web of quality.

    View Slide

  79. To-do
    50
    Learn concepts

    So, what if you wanted to start using web integration testing on your project tomorrow?
    First of all, get familiar with the notions. If I've done my job well today, you can tick that one now.
    Then, try tools that appeal to you and your team. There are some well-known, go-to solutions, but they all depend on your preferred language and
    framework.
    Based on these first tries, evaluate RoI on your project. The global idea is to consider if the regression costs are higher than those of inserting a
    new testing tool and platform.
    Finally, since all this is quite in a lively time, participate in the community. It's only by getting everyone on the board that we'll be able to create a
    web of quality.

    View Slide

  80. To-do
    50
    Learn concepts
    ‣ Try tools

    So, what if you wanted to start using web integration testing on your project tomorrow?
    First of all, get familiar with the notions. If I've done my job well today, you can tick that one now.
    Then, try tools that appeal to you and your team. There are some well-known, go-to solutions, but they all depend on your preferred language and
    framework.
    Based on these first tries, evaluate RoI on your project. The global idea is to consider if the regression costs are higher than those of inserting a
    new testing tool and platform.
    Finally, since all this is quite in a lively time, participate in the community. It's only by getting everyone on the board that we'll be able to create a
    web of quality.

    View Slide

  81. To-do
    50
    Learn concepts
    ‣ Try tools
    ‣ Evaluate RoI

    So, what if you wanted to start using web integration testing on your project tomorrow?
    First of all, get familiar with the notions. If I've done my job well today, you can tick that one now.
    Then, try tools that appeal to you and your team. There are some well-known, go-to solutions, but they all depend on your preferred language and
    framework.
    Based on these first tries, evaluate RoI on your project. The global idea is to consider if the regression costs are higher than those of inserting a
    new testing tool and platform.
    Finally, since all this is quite in a lively time, participate in the community. It's only by getting everyone on the board that we'll be able to create a
    web of quality.

    View Slide

  82. To-do
    50
    Learn concepts
    ‣ Try tools
    ‣ Evaluate RoI
    ‣ Give feedback

    So, what if you wanted to start using web integration testing on your project tomorrow?
    First of all, get familiar with the notions. If I've done my job well today, you can tick that one now.
    Then, try tools that appeal to you and your team. There are some well-known, go-to solutions, but they all depend on your preferred language and
    framework.
    Based on these first tries, evaluate RoI on your project. The global idea is to consider if the regression costs are higher than those of inserting a
    new testing tool and platform.
    Finally, since all this is quite in a lively time, participate in the community. It's only by getting everyone on the board that we'll be able to create a
    web of quality.

    View Slide

  83. Thanks!
    • Image credits
    • From The Noun Project
    • Code: d
    • Gear: Johan H. W. Basberg
    • Cursor: Yellow
    • All software logos © their
    respective owners
    • Thanks
    • Nicolas Dupont
    • Thomas De Bona
    • Fabien Massol
    • Chloé Chevalier
    • Anouchka Labonne
    • Gilles Fabio
    • Watai sponsors
    51
    Matti Schneider — @matti_sg
    github.com/MattiSG/Watai
    Thank you for your attention.
    Additional resources:
    • Functional GUI Testing Automation Patterns: a bit of theory of testing
    • 5 Reasons I Chose Selenium over PhantomJS
    • Running Headless Selenium with Chrome with a VM and Vagrant: get your lab ready quickly
    • Design Patterns for Customer Testing — Misha Rybalov, Quality Centered Developer: excellent abstract patterns
    • Compatibility/Web Testing wiki at Mozilla: a list of tools obtained through a poll by Karl Dubost on his audience
    • The introduction of WebDriver (2009)

    View Slide