Advanced UI Testing

Dive deep into your application with Advanced UI Testing! After a brief background on exploratory and feature testing, lift the fog on how selenium testing actually works at the network layer, and discuss sample code from real automation tests. Topics include leveraging jquery to speed iteration in development, demystifying xPath, strategies for validating features behind multi-arm bandits, running headless, identifying race conditions, and work with iFrames.

Transcript

1. Advanced UI Testing Code Samples: github.com/matt-land/uitest Slides: speakerdeck.com/mattland/advanced-ui-testing
2. We help people get storage. Move in and move on.
3. Matt Land CERTIFIED A R C H I T E CT 2 linkedin.com/in/matthewland
5. I Write Tests
6. Testing Pyramid* manual tests Exploratory Feature * close enough Cost Time
7. Exploratory Testing • Finding finding out our software actually works • Spending our time where we have the most to gain • Critical Path (sign in, sign up, payment settings) • Where users spend the bulk of time • Rewards creativity of the developer • Excellent for existing projects with no tests • Excellent for ‘new to you’ products
8. Feature Testing • Behavior Driven Development • Test Driven Development • Test as contract: This feature was important enough to have a test written for it. This contract forbids you from breaking said feature.
9. Some of Each • Exploring a site before creating a test plan • Exploring how things actually work • Become more Feature Test driven as a test suite matures
10. Developing a New Test • What am I testing? • How can I capture information from UI or DB? • What is the current application flow in pseudo code? • Wait for everything. • $this->fail() at each step to see what was captured.
11. * Time: 11:49 PM */ use \PHPUnit_Extensions_Selenium2TestCase_Keys as Keys; class SearchTest extends MyAbstract { public function testSearch() { //go to the google home page //wait for it to load //enter search terms in the query box //click search //verify i get results } } What I am testing Pseudo code *This would be the only test in this file. Each test gets its own.
12. Standard Recipe Setup the environment in persistence Load the UI Manipulate the UI Bounce the UI, because last time was just jQuery Verify the side effects in persistence
13. Process Overview PDO Driver :3306 phpunit php-1 runs a reflection Selenium :4444 Firefox :7055 Apache :80 :443 php-1 runs the app paratest junit.xml terminal output return code Ant/Maven Hudson/ Bamboo bash
14. Setup the Environment • Inject directly into the storage engine OR use the UI • Use the factory pattern to setup complex conditions on each test run OR make permanent data • Shell accounts with semi randomized objects for your app
15. Test & Interact • Assert things are on the page with ::byId, ::byTag, ::byXpath, ::byCssSelector, ::by ClassName, ::byName • Interact with ->click(), ->clear(), ->submit(), - >value, ::keys($keys) • Wait for something to happen (another assertion) • Continue with the next command
16. Bouncing • Verify both PHP and JavaScript set the correct page state after an event • Perform the click to test JS first • Wait and assert • Reload the current url with self::url() • Make the same assertions
17. Side Effects • Objects have changed since creation at the beginning of test • Query the storage engine at the end of the test for states not revealed in the UI • Should be moved to a (much faster) service test • But ‘Bad’ Tests > No Tests. Balance technical debt.
18. */ use \PHPUnit_Extensions_Selenium2TestCase_Keys as Keys; class SearchTest extends MyAbstract { protected function waitForText($string) { $maxTime = time() + 15; //wait 15 sec max while (stripos(self::source(), '<body') === false) { usleep(333333); if (time() > $maxTime) { throw new \Exception('Timeout waiting for page to load'); } }; while (stripos(self::byTag('body')->text(), $string) === false) { usleep(333333); if (time() > $maxTime) { throw new \Exception('Timeout waiting for string ' . $string); } } return true; } public function testSearch() { self::url('/'); self::waitForText('Privacy'); self::byName('q')->value('ninjas or monkeys'); self::keys(Keys::ENTER); self::waitForText('results'); } } waiting waiting https://github.com/matt-land/uitest
19. class SignupTest extends MyAbstract { public function testSignup() { //signup values $user = []; $names = ['elephant', 'chains', 'climbing', 'purplish', 'keen', 'exhaled', 'packe $user['email'] = uniqid('seleium').'@nowhere.net'; $user['password'] = sha1(uniqid('password')); $user['name'] = $names[array_rand($names)].$names[array_rand($names)].uniqid(); //register account self::url('https://www.tumblr.com/'); self::waitForText('Explore Tumblr'); self::byName("user[email]")->value($user['email']); self::byName("user[password]")->value($user['password']); self::byName("tumblelog[name]")->value($user['name']); self::byId('signup_forms_submit')->click(); //say we are of age self::waitForText('How old are you?'); self::byId("signup_age")->value(rand(18,69)); self::byId('signup_tos')->click(); self::byId('signup_forms_submit')->click(); //stop at captcha step self::waitForText('I am not a robot'); } } https://github.com/matt-land/uitest
20. class ResidualSignupTest extends AbstractSignup { public function testResidualSignup() { $this->_doCode('R3X3'); $this->byId('submit')->click(); $email = $this->_doUserInfo('testResidual'); $this->byId('submit')->click(); if ( Genesis_Service_Feature::isActive(Genesis_Entity_Feature::MYFOOT_RESIDUAL_ESIGNATURE)) { $this->_doDocuSign(); $this->byId('submit')->click(); } $this->_doTerms(); $this->byId('submit')->click(); $this->_doBilling($email); // skip billing info while netsuite is down if (\Genesis_Service_Feature::isActive(\Genesis_Entity_Feature::MYFOOT_TEST_NETSUITE_SANDBOX) $this->byId('submit')->click(); } else { $this->byId('do_this_later')->click(); } $this->_assertWeReachedWelcome(); } }
21. Intro Over
22. jQuery Exploring
23. …speeds Test Development • ‘Fail’ faster • Compliments Selenium selectors almost 1:1 • Translates simply back and forth • Can be loaded into any page from the console • Allows running a test to the fail point, call sleep(), break (cntl-c). Switch to the browser console and continue.
24. Loading into the Dom Open the console and paste: var jq = document.createElement('script');
 jq.src = "https://ajax.googleapis.com/ajax/libs/jquery/1/ jquery.min.js";
 doument.getElementsByTagName('head') [0].appendChild(jq);
 Test with version number $().jquery; https://github.com/matt-land/uitest
25. Selector Compliments self::byId(‘idName’); self::byTag(‘body’); self::byName(‘email’); self::byCssSelector(‘.nav .link’); $(‘#idName’); $(‘body’); $(‘input[name=email]’); $(‘.nav .link’);
26. Manipulator Compliments $element.click(); $element.val(‘new val’); $().trigger({type: ‘keypress’, which: $key, keyCode: $key}); $object->click(); $object->value(‘new val’); self::keys($keyPressed);
27. Attribute Compliments $object->displayed(); $object->text(); $object->value(); $object->attribute(‘checked’); $element.is(‘:visible’); $element.text(); $element.val(); $element.attr(‘checked’);
28. XPath Selectors //*[@id = 'bid-locale-picker']//*[contains(@class, ‘bid-by-zip')]/a //div[contains(@class, 'right') and contains(@class, 'menu')]//div[contains(@class, 'ui') and contains(@class, 'dropdown') and contains(@class, ‘item')] //tr[@id={$unitNumber}]/td[contains(@class, 'availability-checkbox')]/ div[contains(@class, 'checkbox') and contains(@class, ‘ui')] //td[@id='action-$confirmationCode']//*[contains(@class,'confirm-button') and contains(@class, 'active')]
29. Method Chaining self::byXPath( “//*[@id=‘site-fac-dropdown’]//*[@class='search']"); ") $(‘#site-fac-dropdown’).find(‘.search’); self::byId(‘menu’)->byTag(‘a’)->click(); $(‘#menu’).find(‘a’)->click();
30. Notable Difference • jQuery returns an array of all matches on selector • Selenium returns first match. • Call the same Selenium selector again to iterate.
31. Crazy While Loops
32. */ use PHPUnit_Extensions_Selenium2TestCase; class WhileLoops extends MyAbstract { public function waitForElementDisplay($id) { $time = time(); while(! $this->byId($id)->displayed()) { usleep(10000); if (time() > $time + 15) { throw new \Exception('element ' . $id . ' failed to display in time'); } } } public function testSomething() { self::url('/myPage'); self::waitForText('Please enter how much rent the customer paid you during'); $value = round(rand(100,9999)/100, 2, PHP_ROUND_HALF_DOWN); self::byId('rent-tenant-other')->value($value); self::byId('edit-tenant-rent-submit')->click(); //wait for client side jquery event to finish while(self::byId('edit-tenant-rent-collected-title')->displayed()) { sleep(1); } $this->assertEquals('$' . $value, self::byId('rent')); } } https://github.com/matt-land/uitest
33. Feature Toggles
34. Feature Toggles Your new feature is crap if: • the toggle for it is off in the CI environment • CI tests never ran against it before it was flipped on in Production Surrender control of Feature Toggles in the testing environment to the tests. Deliberately consider what toggles a test should be setting for itself. Iterate over the variants in CI
35. class ExampleFeatureToggleTest extends MyAbstract { public function testMyFeature() { try { ToggleService::setFeature('featureName', $enabled = 0); $this->_testMyFeature(); ToggleService::setFeature('featureName', $enabled = 1); $this->_testMyFeature(); } catch (\Exception $e) { $this->fail( "With featureName set to " . $enabled . PHP_EOL . $e->getMessage() . PHP_EOL . "Line:".$e->getLine() . PHP_EOL } } private function _testMyFeature() { /** * Do some UI test here */ } } https://github.com/matt-land/uitest
36. Paratest • A wrapper to run multiple threads of phpunit parallel • Cut times by 50% - ? • add to you composer.json and spend a half hour grokking command line args • Must break all singleton dependencies in your app • Must create separate shell accounts to test auth areas
37. Race Conditions self::byId(‘20340ff’)->value(); PHPUnit_Extensions_Selenium2TestCase_WebDriverException Element is no longer attached to the DOM For documentation on this error, please visit: http://seleniumhq.org/exceptions/ stale_element_reference.html PHPUnit_Extensions_Selenium2TestCase_WebDriverException Element not found in the cache - perhaps the page has changed since it was looked up For documentation on this error, please visit: http://seleniumhq.org/exceptions/ stale_element_reference.htmlt
38. <?php namespace UiTest; /** * Created by IntelliJ IDEA. * User: blitzcat * Date: 5/14/15 * Time: 2:53 PM */ class InjectJavascriptTest extends MyAbstract { protected function _testFireJqueryUiDatepicker($inputId) { self::execute([ 'script' => '$("#' . $inputId . '").datepicker("show"); console.log("datepick 'args' => array() ]); } public function testInjectJavascript() { self::url('/'); self::waitForText('privacy'); self::_testFireJqueryUiDatepicker('mypicker'); } } Headless Considerations https://github.com/matt-land/uitest
39. IFrames • Dealing with iframes means more than just one • One way trip down on traversal • Set root node null to exit
40. class IframeExampleTest extends MyAbstract { public function testTraverseIframe() { //switch to the iframe self::frame($this->byId('docusign-iframe')); self::waitForText('Please review the documents below.'); //switch to child iframe self::frame($this->byId('nav-buttons')); // Do STUFF HERE //go back to the parent self::frame(null); //back into the outer iframe self::frame($this->byId('docusign-iframe')); //go back to the parent self::frame(null); } } https://github.com/matt-land/uitest
41. UI Tests Are Expensive!
42. Push Everything Down manual tests
43. Put MVC controllers on a diet and move to MVCS. Automate testing even locally, especially as team grows. You have a node app with microservice endpoints? Its still MVC. Test the services endpoints separately. Push Everything Down
44. • Don’t test everything. • Test whats valuable. • test > no test Code Samples: github.com/matt-land/uitest Slides: speakerdeck.com/mattland/advanced-ui-testing
45. Bonus Round
46. Metainfo • One test per class/file pattern • Define separate test suites in phpunit.xml • Fail on error, or don’t • Code coverage is useless • Smoke tests • This is NOT performance testing
47. Headless Vagrant Strategy • Run on same ports in both OS • Including mysql • PDO connect by IP, not localhost • Vagrant virtual frame buffer and custom firefox