Testing the untestable

When you need to fix or extend code that is 'far down the rabbit hole', most of the work becomes setting up the complex conditions to push the code to a certain point in the application flow. Instead, open classes up directly using testing harnesses, reflection classes and functional programming. Phpunit will be used to demonstrate code coverage before and after.

Transcript

1. Testing the untestable Lone Star PHP 2015
2. We help people get storage. Move in and move on.
3. Matt Land Code Samples: github.com/matt-land Slides: speakerdeck.com/mattland/ linkedin.com/in/matthewland CERTIFIED A R C H I T E CT 2 https://joind.in/13567
4. Testing the Untestable Create the World, and Poke it With a Stick Do I Need CI? Real Fake Data Ice Cream Cones and BDD Intermittent Failure To Launch My Worst UI Test Sticky Bits Its Not Me, Its the Environment
5. (Day 17, 2:45 at SpareFoot) “It doesn’t matter if it works or not. Just make the tests pass so we can deploy.”
6. Poke it with a stick
7. The First Test Noise: • PSR Autoloaders • Composer • Namespaces • Coding Standards • Continuous Integration • Git & Repositories • Best Practices • Mocks • … Not Noise: • Writing Tests • Passing Tests
9. PROTECTED PRIVATE PRIVATE PRIVATE PRIVATE PROTECTED PRIVATE PRIVATE PROTECTED PRIVATE
11. • Sample application: • http://localhost:63342/apps/test-samples/index.php? sort=author • https://github.com/matt-land/test-samples
12. <?php /** * Created by IntelliJ IDEA. * User: mland * Date: 4/17/15 * Time: 4:33 PM */ require_once __DIR__ . '/vendor/autoload.php'; use Samples\Render; $dsnBooks = json_encode([ 'category' => 'Young Adult', 'records' => 5, 'meta' => [ 'next' => '/whatever?q=345w5', 'prev' => '/whatever?q=92342' ], 'books' => [ [ 'title' => 'The Crossover' , 'author' => 'Kwame Alexander', 'publisher' => 'Houghton', 'price' => '16.99', 'isbn' => '9780544107717' ], [ 'title' => 'The Carnival at Bray' , 'author' => 'Jessie Ann Foley', 'publisher' => 'Elephant Rock', 'price' => '12.95', 'isbn' => '97809895115597' ], [ 'title' => 'I\'ll Give You the Sun' , 'author' => 'Jandy Nelson', 'publisher' => 'Dial', 'price' => '17.99', 'publisher' => 'Houghton', 'price' => '16.99', 'isbn' => '9780544107717' ], [ 'title' => 'The Carnival at Bray' , 'author' => 'Jessie Ann Foley', 'publisher' => 'Elephant Rock', 'price' => '12.95', 'isbn' => '97809895115597' ], [ 'title' => 'I\'ll Give You the Sun' , 'author' => 'Jandy Nelson', 'publisher' => 'Dial', 'price' => '17.99', 'isbn' => '9780803734968' ], [ 'title' => 'Jackaby' , 'author' => 'William Ritter', 'publisher' => 'William Ritter', 'price' => '16.95', 'isbn' => '9781616203535' ], [ 'title' => 'The Gospel of Winter' , 'author' => 'Brendan Kiely', 'publisher' => 'Simon & Schuster/Margaret K. McElderry', 'price' => '17.99', 'isbn' => '9781442484894' ], ] ]); $render = new Render($dsnBooks); $sortOrder = isset($_REQUEST['sort']) ? $_REQUEST['sort'] : null; echo $render->toHtml($sortOrder); Continues
13. <h3>Total Records: ' . $this->data->records . '</h3> <a href="' . $this->data->meta->prev . '">Prev</a> <a href="' . $this->data->meta->next . '">Next</a>'; return $html; } private function sortBooksByAuthor() { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $this->books; usort($sortedBooks, function($a, $b) { /** * sort by author last name, so first get the last name * or last segment of the name. Will not work with JR/III suffixes */ $authorA = explode(" ", trim($a->author)); $authorB = explode(" ", trim($b->author)); return strcmp($authorA[count($authorA)-1], $authorB[count($authorB)-1]); }); return $sortedBooks; } private static function sortBooksByPrice($books) { //usort is going to modify $books by reference, //so lets copy it to prevent side effects
14. { $renderObj = new Render(json_encode($this->jsonBody)); $rendering = $renderObj->toHtml(); $this->fail('This is complex: ' . PHP_EOL . $rendering . " Lets test the methods in } /** * reflecting a method, where we have to set the value in the class first */ public function testSortBooksByAuthor() { $reflectedClass = new ReflectionClass('\\Samples\\Render'); $realClass = $reflectedClass->newInstance(); $booksProperty = $reflectedClass->getProperty('books'); // $rea $booksProperty->setAccessible(1); // $rea $booksProperty->setValue($realClass, $this->jsonBody->books); // $rea $sortBooksByAuthorMethod = $reflectedClass->getMethod('sortBooksByAuthor'); //$real $sortBooksByAuthorMethod->setAccessible(true); //$real $sortedBooks = $sortBooksByAuthorMethod->invoke($realClass); //$real $this->assertEquals('Kwame Alexander', $sortedBooks[0]->author); $this->assertEquals('William Ritter', $sortedBooks[4]->author); } /** * this time written statically/no object oriented dependency
16. return strcmp($authorA[count($authorA)-1], $authorB[count($authorB)- }); return $sortedBooks; } private static function sortBooksByPrice($books) { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $books; usort($sortedBooks, function($a, $b) { /** * compare decimal price */ return $a->price >= $b->price; }); return $sortedBooks; } public static function sortBooksByIsbn($books) { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $books; usort($sortedBooks, function($a, $b) { /** * sort by ISBN string which might being with zero Changed to static Parameter is passed in Gave up $this, but still have self::
17. $sortBooksByAuthorMethod->setAccessible(true); //$realC $sortedBooks = $sortBooksByAuthorMethod->invoke($realClass); //$realC $this->assertEquals('Kwame Alexander', $sortedBooks[0]->author); $this->assertEquals('William Ritter', $sortedBooks[4]->author); } /** * this time written statically/no object oriented dependency * Much easier */ public function testSortBooksByPrice() { $realClass = new Render(); $sortBooksByPriceMethod = new ReflectionMethod($realClass, 'sortBooksByPrice'); $sortBooksByPriceMethod->setAccessible(true); $sortedBooks = $sortBooksByPriceMethod->invoke($realClass, $this->jsonBody->books); $this->assertEquals(12.95, $sortedBooks[0]->price); $this->assertEquals(17.99, $sortedBooks[4]->price); } /** * the easiest yet. now that its static and public, we can just use it */ public function testSortBooksByISBN()
19. */ return strcmp($a->isbn, $b->isbn); }); return $sortedBooks; } public static function sortBooksByTitle($books) { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $books; usort($sortedBooks, function($a, $b) { /** * remove articles and sort titles */ return strcmp( str_ireplace(["a ", "an ", "the "], '', $a->title), str_ireplace(["a ", "an ", "the "], '', $b->title) ); }); return $sortedBooks; } } Changed to public No side effect risk
20. { $sortedBooks = Render::sortBooksByIsbn($this->jsonBody->books); $this->assertEquals('9780544107717', $sortedBooks[0]->isbn); $this->assertEquals('9781616203535', $sortedBooks[4]->isbn); } /** * just throw this one in too */ public function testSortBooksByTitle() { $sortedBooks = Render::sortBooksByTitle($this->jsonBody->books); $this->assertEquals('The Carnival at Bray', $sortedBooks[0]->title); $this->assertEquals('Jackaby', $sortedBooks[4]->title); } }
22. Do I need CI?
23. With the addition of CI, each test becomes a contract, between the feature and the maintainers of the application. It says, ‘This feature was important enough to have a test written for it. This contract forbids you from breaking said feature.’
24. • One person: I have self discipline. • Two people: I know who to blame. • Three people: Things were fine until the snake came.
25. Earn Sweet GitHub README badges!
28. Dot the i’s and cross the t’s • Best practice really helps with CI setup • Now make the investment in Composer, PRS namespaces, and a phpunit.xml file • Build tool like ant helps glue together with vagrant (shell exec)
29. <?xml version="1.0" encoding="UTF-8"?> <phpunit colors="true" stopOnFailure="false" syntaxCheck="false" verbose="true" convertErrorsToExceptions="true" convertNoticesToExceptions="true" convertWarningsToExceptions="true" processIsolation="false" bootstrap="tests/Sparefoot/bootstrap.php"> <logging> <log type="junit" target="junit.xml"/> <log type="json" target="results.json"/> <log type="coverage-html" target="codeCoverage" charset="UTF-8" yui="true" highlight="true" lowUpperBound="50" highLowerBound="80"/> </logging> <testsuites> <testsuite name="MyFoot_Test_Suite"> <directory>tests</directory> </testsuite> <testsuite name="MyFoot_SmokeTest"> <file>tests/Sparefoot/Myfoot/LoginTest.php</file> <file>tests/Sparefoot/Myfoot/Facility/DetailsTest.php</file> <file>tests/Sparefoot/Myfoot/Signup/ResidualSignupTest.php</file> <file>tests/Sparefoot/Myfoot/Settings/PaymentTest.php</file> <file>tests/Sparefoot/Myfoot/Statement/CpaDefaultsTest.php</file> <file>tests/Sparefoot/Myfoot/FacilityTest.php</file> <file>tests/Sparefoot/Myfoot/UsersTest.php</file>
30. Game of Life code coverage https://github.com/matt-land/Game-of-Life http://localhost:63342/apps/Game-of-Life/codeCoverage/index.html
31. Real Fake Data
32. public function setUp() { $this->customer = self::mockCustomer(); } public static function mockCustomer() { //build up a test object $customer = new \stdClass(); $customer->name = uniqid('John Test'); $customer->email = uniqid('test').'@test.com'; $customer->address = rand(1,5000).' test street'; $cities = ['Roswell', 'Austin', 'Memphis', 'San Jose']; $customer->city = $cities[array_rand($cities)]; $states = ['NC', 'TX', 'VA', 'CT']; $customer->state = $states[array_rand($states)]; $customer->zip = rand(10000,70000); return $customer; } public function testUpdateCustomer() { return ; Pass & set FK relationships for related models
33. */ public static function generateValidCreditCardNumber($leader = 4) { //set to amex 15 digit if prefix is 3. otherwise its 16 $totalDigits = (substr($leader, 0, 1) == 3) ? 14 : 15; $forwardCardDigits = str_split($leader); // build out the rest of the digits, except for checksum (last digit) for ($i = 0; $i < $totalDigits - strlen($leader); $i++) { $forwardCardDigits[] = rand(0,9); } $reverseCardDigits = array_reverse($forwardCardDigits); $checksum = 0; foreach ($reverseCardDigits as $key => $value) { if ($key % 2 === 0) { $value = 2 * $value; } if ($value > 9) { $checksum += $value - 9; } else { $checksum += $value; } } //get the least significant digit for the checksum $tmp = str_split($checksum); $lsd = (int) end($tmp); $checksumCompliment = $lsd === 0 ? 0 : 10 - $lsd; return implode(null, $forwardCardDigits) . $checksumCompliment; }
34. Ice Cream Cones & Behavior Driven Development
37. • Setup the environment in persistence • Load the UI • Manipulate the UI • Bounce the UI, because last time was just jQuery • Verify the side effects in persistence
38. • Setup the environment in persistence • Load the UI • Manipulate the UI • Bounce the UI, because the first time was just jQuery • Verify the side effects in persistence Tech Debt The test is doing too much or test the service separately Because our logic is everywhere
39. Intermittent Failure To Launch
40. • 1:45pm - click build again • 2:00pm - notice haven’t passed yet today • 2:10pm - bounce selenium grid for luck • 2:30pm - in hip chat debug room for screenshots • 2:40pm - last chance to build and go green on time • 2:55pm - curse violently. Devbot mocks you • 3:10pm - bow out so other teams can deploy • 4:05pm - finally a green build. !Freeze • 6:45pm - DevOps back from kid’s swim practice. Successfully Deploy!
41. Intermittent test failure smell • Blame the test • Blame the CI environment performance • Blame the JavaScript plugins (jQuery whatever) • Realize your code is doing too damn much. • Actually fails a large percentage of the time in prod.
42. My Worst UI Test
43. Watch a robot sign a contract
44. All the brittle things • Send some emails on account creation • Step 3 in a sign up flow • Create an envelope via API (docusign) • Redirect via JS • Simulate Abandoning signup and coming back • ‘Sign’ using a third party service • Upload a PDF copy to a different API (salesforce) • Send a ticket to sales via API (zendesk) • Continue signup flow for 3 more steps
45. All the things I learned • Selenium environment is brittle. Use a launcher script to minimize debug time. • Run and expose mysql on 3307 in host and vagrant. • Keep VM selenium ‘trapped’ • No manual config changes • No dependency on injected Envars
46. Sticky Bits
47. Sticky Bits I run tests both ways. In Vagrant (headless) and in host OS (interactive) Both are super useful to my workflow: • In the host OS for writing and debugging a broken condition • In the VM for running a suite before opening my pull request
48. Sticky Bits (2) Multiple Ant targets (smoke vs full vs regression) balance • speed of quick feedback • full regression coverage • (don’t trust anyone) Code coverage plug-ins are invalid in the scope for integration level tests.
49. • IFRAME->IFRAME is one way trip • Write a dedicated testsuite API for Datepicker() • Expected, Actual, Whoops-Message • Explicitly wait for _everything_ • Disable tests just like C++ by underscore prefixing • Fail as fast as possible (Temporarily disable long tests higher in the class) Sticky Bits (3)
50. • Feature Toggle Iteration • Cookie up with 404 • add controller/action/behavior ids to the dom • Use identical IDs when FF’ing so the test can follow • Request/Factory/Response Mock Factory • Watch for _test nerving in pull requests Sticky Bits (4)
51. Our launcher… https://github.com/matt-land/upstart
52. Its not me, its the environment
53. It Starts With One Test Code Samples: github.com/matt-land Slides: speakerdeck.com/mattland/ linkedin.com/in/matthewland https://joind.in/13567