Javascript Testing: The Holy Grail

Javascript Testing: The Holy Grail

Approaching the Holy Grail of Javascript application testing. Presented at RejectJS in Berlin


Adam Hawkins

October 04, 2012


  1. None
  2. Presented at RejectJS Berlin 2012

  3. Ich bin • Adam Hawkins • tw://adman65 • adn://twinturbo •

    gh://twinturbo •
  4. Should I listen to this guy?

  5. You’re god damn right

  6. Who, What, Why?

  7. What is the Holy Grail? • Browser independent • All

    tests written in JavaScript • Unit and integration support • Test against a single browser in development • Test against all target browsers in CI • Runs from the CLI--not by opening some random HTML page.
  8. Why Care? • Testing + Multi browser CI is the

    only way to know that your application is functioning correctly. • Manually test if you want....
  9. Where are We Now? • Individual aspects, but nothing working

    in concert • Classic choose any 3: cheap, fast, scalable • Choose one implementation and lose one of the three
  10. How did We Get Here? • Javascript not really considered

    a first class language until recently • Javascript traditionally confined to the browser • Focus on Single Page Applications/Client side JS MVC/Insert your term here
  11. Learning to test my javascript app What have you been

    doing all these years?
  12. This is harder than Haskell

  13. Let’s See What’s Out There

  14. *this is for testing Javascript applications

  15. PhantomJS

  16. $ phantomjs phantomjs> console.log("Yo Dawg"); Yo Dawg undefined phantomjs>

  17. var page = require('webpage').create(); var url = '';, function

    (status) { page.includeJs("jquery.min.js", function() { // jQuery is loaded, now manipulate the DOM }); page.evalute(function() { // evaluate code in the remote context $("title").html("ZOMG"); }); phantom.exit(); });
  18. • Headless browser • Execute JavaScript programmatically in a real

    content • Real DOM
  19. CasperJS

  20. casper.start "", -> @test.assertTitle "Google", "google homepage title incorrect" @test.assertExists

    'form[action="/search"]', "main form is found" @fill 'form[action="/search"]', q: "foo", true casper.then -> @test.assertTitle "foo - Recherche Google", "google title is ok" @test.assertUrlMatch /q=foo/, "search term has been submitted" @test.assertEval (-> __utils__.findAll("h3.r").length >= 10 ), "google search for \"foo\" retrieves 10 or more results" -> @test.renderResults true
  21. • Built on top of PhantomJS • Used to write

    integration tests • DOM interaction API. Example: fillIn(“foo”, “bar”) • Should not be used to write unit tests • Good for a small # of test cases • Tests bound PhantomJS
  22. JSTestDriver Made by:

  23. GreeterTest = TestCase("GreeterTest"); GreeterTest.prototype.testGreet = function() { var greeter =

    new myapp.Greeter(); assertEquals("Hello World!", greeter.greet("World")); };
  24. JSTestDriver Architecture

  25. • Runs from the command line • Supports multiple browsers

    • Unclear how to do integration tests • Browser independent, but test runner dependent • Must maintain browser installations • Tests bound to test runner
  26. ZombieJS

  27. // Load the page from localhost browser = new Browser()

    browser.visit("http://localhost:3000/", function () { // Fill email, password and submit form browser. fill("email", "zombie@underworld.dead"). fill("password", "eat-the-living"). pressButton("Sign Me Up!", function() { // Form submitted, new page loaded. assert.ok(browser.success); assert.equal(browser.text("title"), "BRAINZ"); }); });
  28. • Clear support for integration tests • Decent DOM interaction

    API • Built on Node • Tests bound to ZombieJS • Similar to CasperJS + PhantomJS
  29. Capybara

  30. # rspec describe "the signup process", :type => :request do

    it "signs me in" do # capybara within("#session") do fill_in 'Login', :with => '' fill_in 'Password', :with => 'password' end click_link 'Sign in' end end
  31. • Multiple adapters for headless or real browsers • Easy

    to maintain large test suites • Write tests in Ruby • Test code and app code exist in separate threads. This can be confusing.
  32. What do these have in Common?

  33. • No unified framework for unit tests and integration tests

    • Dependency hell (c++, python, node, java, or ruby) • Tests bound to test runner • Not all methods CLI friendly
  34. Approaching the Holy Grail

  35. How can we combine these things?

  36. 1. Remove test runner dependence 2. Unify integration and unit

    tests 3. Run in a headless browser for local tests 4. Run in real browsers in CI 5. Try to do it with Javascript
  37. Step 1: Writing Tests

  38. None
  39. // this is a unit test test("hello test", function() {

    ok(1 == "1", "Passed!"); }); // this is an integration test test("user can fill in the form", function() { fillIn("#todo", "Fix JS Testing"); press("#todo-form submit"); assertContains("#todos", "Fix JS Testing"); });
  40. * Use jQuery to implement a higher level DOM interaction

    UI similar to capybara for tests
  41. Step 2: Run The Tests

  42. <!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title>QUnit Example</title> <link rel="stylesheet"

    href="/resources/qunit.css"> </head> <body> <div id="qunit"></div> <script src="apps.js"></script> <script src="tests.js"></script> </body> </html>
  43. # Use phantomJS to open the HTML page # and

    capture the results $ phantomjs run_qunit.js tests.html ..................... 10 Tests, 10 Passes, 0 Failures
  44. Step 3: Multiple Browsers

  45. None
  46. <!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title>Integration Tests</title> <link rel="stylesheet"

    href="/resources/qunit.css"> </head> <body> <div id="qunit"></div> <script src="apps.js"></script> <script src="tests.js"></script> <!-- Added this to hook into test swarm --> <script src=""></script> </body> </html>
  47. # Upload that HTML file to your CI server #

    so it can be accessed with test swarm $ swarm execute tests.js Running in all browsers.... IE9: 10 tests, 10 Passed, 0 Failures IE6, 10 tests, 0 Passed, 12349871234987 Failures Chrome: 10 tests, 10 Passed, 0 Failures FireFox: 10 tests, 10 Passed, 0 Failures
  48. # ENV variables determine build # Ideally set in your

    CI process $ swarm execute tests.js BROWSER=IE9 $ swarm execute tests.js BROWSER=FF,SAFARI,CHROME
  49. Is this Better?

  50. You’re god damn right

  51. Why? • Completely browser independent • All tests javascript •

    Test runner independent • CI Friendly • Tests and app code running in the same thread. Read: direct access to app + ui.
  52. The community can solve this problem

  53. I’ve Already Started •

  54. Bitte warten Sie

  55. TL;DR: There is no Holy Grail

  56. None
  57. Why? • Requirements change test system • Testing backend +

    frontend requires a different approach • The holy grail for that is use testing tools for what language your backend is in • Things like HTML5 Audio/Video can induce complexity
  58. Moral of the Story • Testing Javascript applications is difficult

    but possible • Decide if you want to test your frontend and backend together. • Oh ya....test that shit!
  59. Die Slides