Javascript Testing: The Holy Grail

Approaching the Holy Grail of Javascript application testing. Presented at RejectJS in Berlin

Transcript

2. Presented at RejectJS Berlin 2012
3. Ich bin • Adam Hawkins • tw://adman65 • adn://twinturbo • gh://twinturbo • http://broadcastingadam.com
4. Should I listen to this guy?
5. You’re god damn right
6. Who, What, Why?
7. What is the Holy Grail? • Browser independent • All tests written in JavaScript • Unit and integration support • Test against a single browser in development • Test against all target browsers in CI • Runs from the CLI--not by opening some random HTML page.
8. Why Care? • Testing + Multi browser CI is the only way to know that your application is functioning correctly. • Manually test if you want....
9. Where are We Now? • Individual aspects, but nothing working in concert • Classic choose any 3: cheap, fast, scalable • Choose one implementation and lose one of the three
10. How did We Get Here? • Javascript not really considered a first class language until recently • Javascript traditionally confined to the browser • Focus on Single Page Applications/Client side JS MVC/Insert your term here
11. Learning to test my javascript app What have you been doing all these years?
12. This is harder than Haskell
13. Let’s See What’s Out There
14. *this is for testing Javascript applications
15. PhantomJS
16. $ phantomjs phantomjs> console.log("Yo Dawg"); Yo Dawg undefined phantomjs>
17. var page = require('webpage').create(); var url = 'http://www.phantomjs.org/'; page.open(url, function (status) { page.includeJs("jquery.min.js", function() { // jQuery is loaded, now manipulate the DOM }); page.evalute(function() { // evaluate code in the remote context $("title").html("ZOMG"); }); phantom.exit(); });
18. • Headless browser • Execute JavaScript programmatically in a real content • Real DOM
19. CasperJS
20. casper.start "http://www.google.fr/", -> @test.assertTitle "Google", "google homepage title incorrect" @test.assertExists 'form[action="/search"]', "main form is found" @fill 'form[action="/search"]', q: "foo", true casper.then -> @test.assertTitle "foo - Recherche Google", "google title is ok" @test.assertUrlMatch /q=foo/, "search term has been submitted" @test.assertEval (-> __utils__.findAll("h3.r").length >= 10 ), "google search for \"foo\" retrieves 10 or more results" casper.run -> @test.renderResults true
21. • Built on top of PhantomJS • Used to write integration tests • DOM interaction API. Example: fillIn(“foo”, “bar”) • Should not be used to write unit tests • Good for a small # of test cases • Tests bound PhantomJS
22. JSTestDriver Made by:
23. GreeterTest = TestCase("GreeterTest"); GreeterTest.prototype.testGreet = function() { var greeter = new myapp.Greeter(); assertEquals("Hello World!", greeter.greet("World")); };
24. JSTestDriver Architecture
25. • Runs from the command line • Supports multiple browsers • Unclear how to do integration tests • Browser independent, but test runner dependent • Must maintain browser installations • Tests bound to test runner
26. ZombieJS
27. // Load the page from localhost browser = new Browser() browser.visit("http://localhost:3000/", function () { // Fill email, password and submit form browser. fill("email", "zombie@underworld.dead"). fill("password", "eat-the-living"). pressButton("Sign Me Up!", function() { // Form submitted, new page loaded. assert.ok(browser.success); assert.equal(browser.text("title"), "BRAINZ"); }); });
28. • Clear support for integration tests • Decent DOM interaction API • Built on Node • Tests bound to ZombieJS • Similar to CasperJS + PhantomJS
29. Capybara
30. # rspec describe "the signup process", :type => :request do it "signs me in" do # capybara within("#session") do fill_in 'Login', :with => 'user@example.com' fill_in 'Password', :with => 'password' end click_link 'Sign in' end end
31. • Multiple adapters for headless or real browsers • Easy to maintain large test suites • Write tests in Ruby • Test code and app code exist in separate threads. This can be confusing.
32. What do these have in Common?
33. • No unified framework for unit tests and integration tests • Dependency hell (c++, python, node, java, or ruby) • Tests bound to test runner • Not all methods CLI friendly
34. Approaching the Holy Grail
35. How can we combine these things?
36. 1. Remove test runner dependence 2. Unify integration and unit tests 3. Run in a headless browser for local tests 4. Run in real browsers in CI 5. Try to do it with Javascript
37. Step 1: Writing Tests
39. // this is a unit test test("hello test", function() { ok(1 == "1", "Passed!"); }); // this is an integration test test("user can fill in the form", function() { fillIn("#todo", "Fix JS Testing"); press("#todo-form submit"); assertContains("#todos", "Fix JS Testing"); });
40. * Use jQuery to implement a higher level DOM interaction UI similar to capybara for tests
41. Step 2: Run The Tests
42. <!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title>QUnit Example</title> <link rel="stylesheet" href="/resources/qunit.css"> </head> <body> <div id="qunit"></div> <script src="apps.js"></script> <script src="tests.js"></script> </body> </html>
43. # Use phantomJS to open the HTML page # and capture the results $ phantomjs run_qunit.js tests.html ..................... 10 Tests, 10 Passes, 0 Failures
44. Step 3: Multiple Browsers
46. <!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title>Integration Tests</title> <link rel="stylesheet" href="/resources/qunit.css"> </head> <body> <div id="qunit"></div> <script src="apps.js"></script> <script src="tests.js"></script> <!-- Added this to hook into test swarm --> <script src="http://example.org/testswarm/js/inject.js"></script> </body> </html>
47. # Upload that HTML file to your CI server # so it can be accessed with test swarm $ swarm execute tests.js Running in all browsers.... IE9: 10 tests, 10 Passed, 0 Failures IE6, 10 tests, 0 Passed, 12349871234987 Failures Chrome: 10 tests, 10 Passed, 0 Failures FireFox: 10 tests, 10 Passed, 0 Failures
48. # ENV variables determine build # Ideally set in your CI process $ swarm execute tests.js BROWSER=IE9 $ swarm execute tests.js BROWSER=FF,SAFARI,CHROME
49. Is this Better?
50. You’re god damn right
51. Why? • Completely browser independent • All tests javascript • Test runner independent • CI Friendly • Tests and app code running in the same thread. Read: direct access to app + ui.
52. The community can solve this problem
53. I’ve Already Started • https://github.com/radiumsoftware/iridium
54. Bitte warten Sie
55. TL;DR: There is no Holy Grail
57. Why? • Requirements change test system • Testing backend + frontend requires a different approach • The holy grail for that is use testing tools for what language your backend is in • Things like HTML5 Audio/Video can induce complexity
58. Moral of the Story • Testing Javascript applications is difficult but possible • Decide if you want to test your frontend and backend together. • Oh ya....test that shit!
59. Die Slides https://speakerdeck.com/u/twinturbo