Share
Embed
Testing the untestable
When you need to fix or extend code that is 'far down the rabbit hole', most of the work becomes setting up the complex conditions to push the code to a certain point in the application flow. Instead, open classes up directly using testing harnesses, reflection classes and functional programming. Phpunit will be used to demonstrate code coverage before and after.
Matt Land
April 18, 2015
Transcript
53 slides
-
Matt Land Code Samples: github.com/matt-land Slides: speakerdeck.com/mattland/ linkedin.com/in/matthewland CERTIFIED A
R C H I T E CT 2 https://joind.in/13567 -
Testing the Untestable Create the World, and Poke it With
a Stick Do I Need CI? Real Fake Data Ice Cream Cones and BDD Intermittent Failure To Launch My Worst UI Test Sticky Bits Its Not Me, Its the Environment -
(Day 17, 2:45 at SpareFoot) “It doesn’t matter if it
works or not. Just make the tests pass so we can deploy.” -
The First Test Noise: • PSR Autoloaders • Composer •
Namespaces • Coding Standards • Continuous Integration • Git & Repositories • Best Practices • Mocks • … Not Noise: • Writing Tests • Passing Tests -
<?php /** * Created by IntelliJ IDEA. * User: mland
* Date: 4/17/15 * Time: 4:33 PM */ require_once __DIR__ . '/vendor/autoload.php'; use Samples\Render; $dsnBooks = json_encode([ 'category' => 'Young Adult', 'records' => 5, 'meta' => [ 'next' => '/whatever?q=345w5', 'prev' => '/whatever?q=92342' ], 'books' => [ [ 'title' => 'The Crossover' , 'author' => 'Kwame Alexander', 'publisher' => 'Houghton', 'price' => '16.99', 'isbn' => '9780544107717' ], [ 'title' => 'The Carnival at Bray' , 'author' => 'Jessie Ann Foley', 'publisher' => 'Elephant Rock', 'price' => '12.95', 'isbn' => '97809895115597' ], [ 'title' => 'I\'ll Give You the Sun' , 'author' => 'Jandy Nelson', 'publisher' => 'Dial', 'price' => '17.99', 'publisher' => 'Houghton', 'price' => '16.99', 'isbn' => '9780544107717' ], [ 'title' => 'The Carnival at Bray' , 'author' => 'Jessie Ann Foley', 'publisher' => 'Elephant Rock', 'price' => '12.95', 'isbn' => '97809895115597' ], [ 'title' => 'I\'ll Give You the Sun' , 'author' => 'Jandy Nelson', 'publisher' => 'Dial', 'price' => '17.99', 'isbn' => '9780803734968' ], [ 'title' => 'Jackaby' , 'author' => 'William Ritter', 'publisher' => 'William Ritter', 'price' => '16.95', 'isbn' => '9781616203535' ], [ 'title' => 'The Gospel of Winter' , 'author' => 'Brendan Kiely', 'publisher' => 'Simon & Schuster/Margaret K. McElderry', 'price' => '17.99', 'isbn' => '9781442484894' ], ] ]); $render = new Render($dsnBooks); $sortOrder = isset($_REQUEST['sort']) ? $_REQUEST['sort'] : null; echo $render->toHtml($sortOrder); Continues -
<h3>Total Records: ' . $this->data->records . '</h3> <a href="' .
$this->data->meta->prev . '">Prev</a> <a href="' . $this->data->meta->next . '">Next</a>'; return $html; } private function sortBooksByAuthor() { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $this->books; usort($sortedBooks, function($a, $b) { /** * sort by author last name, so first get the last name * or last segment of the name. Will not work with JR/III suffixes */ $authorA = explode(" ", trim($a->author)); $authorB = explode(" ", trim($b->author)); return strcmp($authorA[count($authorA)-1], $authorB[count($authorB)-1]); }); return $sortedBooks; } private static function sortBooksByPrice($books) { //usort is going to modify $books by reference, //so lets copy it to prevent side effects -
{ $renderObj = new Render(json_encode($this->jsonBody)); $rendering = $renderObj->toHtml(); $this->fail('This is
complex: ' . PHP_EOL . $rendering . " Lets test the methods in } /** * reflecting a method, where we have to set the value in the class first */ public function testSortBooksByAuthor() { $reflectedClass = new ReflectionClass('\\Samples\\Render'); $realClass = $reflectedClass->newInstance(); $booksProperty = $reflectedClass->getProperty('books'); // $rea $booksProperty->setAccessible(1); // $rea $booksProperty->setValue($realClass, $this->jsonBody->books); // $rea $sortBooksByAuthorMethod = $reflectedClass->getMethod('sortBooksByAuthor'); //$real $sortBooksByAuthorMethod->setAccessible(true); //$real $sortedBooks = $sortBooksByAuthorMethod->invoke($realClass); //$real $this->assertEquals('Kwame Alexander', $sortedBooks[0]->author); $this->assertEquals('William Ritter', $sortedBooks[4]->author); } /** * this time written statically/no object oriented dependency -
return strcmp($authorA[count($authorA)-1], $authorB[count($authorB)- }); return $sortedBooks; } private static function
sortBooksByPrice($books) { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $books; usort($sortedBooks, function($a, $b) { /** * compare decimal price */ return $a->price >= $b->price; }); return $sortedBooks; } public static function sortBooksByIsbn($books) { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $books; usort($sortedBooks, function($a, $b) { /** * sort by ISBN string which might being with zero Changed to static Parameter is passed in Gave up $this, but still have self:: -
$sortBooksByAuthorMethod->setAccessible(true); //$realC $sortedBooks = $sortBooksByAuthorMethod->invoke($realClass); //$realC $this->assertEquals('Kwame Alexander', $sortedBooks[0]->author); $this->assertEquals('William
Ritter', $sortedBooks[4]->author); } /** * this time written statically/no object oriented dependency * Much easier */ public function testSortBooksByPrice() { $realClass = new Render(); $sortBooksByPriceMethod = new ReflectionMethod($realClass, 'sortBooksByPrice'); $sortBooksByPriceMethod->setAccessible(true); $sortedBooks = $sortBooksByPriceMethod->invoke($realClass, $this->jsonBody->books); $this->assertEquals(12.95, $sortedBooks[0]->price); $this->assertEquals(17.99, $sortedBooks[4]->price); } /** * the easiest yet. now that its static and public, we can just use it */ public function testSortBooksByISBN() -
*/ return strcmp($a->isbn, $b->isbn); }); return $sortedBooks; } public static
function sortBooksByTitle($books) { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $books; usort($sortedBooks, function($a, $b) { /** * remove articles and sort titles */ return strcmp( str_ireplace(["a ", "an ", "the "], '', $a->title), str_ireplace(["a ", "an ", "the "], '', $b->title) ); }); return $sortedBooks; } } Changed to public No side effect risk -
{ $sortedBooks = Render::sortBooksByIsbn($this->jsonBody->books); $this->assertEquals('9780544107717', $sortedBooks[0]->isbn); $this->assertEquals('9781616203535', $sortedBooks[4]->isbn); } /**
* just throw this one in too */ public function testSortBooksByTitle() { $sortedBooks = Render::sortBooksByTitle($this->jsonBody->books); $this->assertEquals('The Carnival at Bray', $sortedBooks[0]->title); $this->assertEquals('Jackaby', $sortedBooks[4]->title); } } -
With the addition of CI, each test becomes a contract,
between the feature and the maintainers of the application. It says, ‘This feature was important enough to have a test written for it. This contract forbids you from breaking said feature.’ -
• One person: I have self discipline. • Two people:
I know who to blame. • Three people: Things were fine until the snake came. -
Dot the i’s and cross the t’s • Best practice
really helps with CI setup • Now make the investment in Composer, PRS namespaces, and a phpunit.xml file • Build tool like ant helps glue together with vagrant (shell exec) -
<?xml version="1.0" encoding="UTF-8"?> <phpunit colors="true" stopOnFailure="false" syntaxCheck="false" verbose="true" convertErrorsToExceptions="true" convertNoticesToExceptions="true"
convertWarningsToExceptions="true" processIsolation="false" bootstrap="tests/Sparefoot/bootstrap.php"> <logging> <log type="junit" target="junit.xml"/> <log type="json" target="results.json"/> <log type="coverage-html" target="codeCoverage" charset="UTF-8" yui="true" highlight="true" lowUpperBound="50" highLowerBound="80"/> </logging> <testsuites> <testsuite name="MyFoot_Test_Suite"> <directory>tests</directory> </testsuite> <testsuite name="MyFoot_SmokeTest"> <file>tests/Sparefoot/Myfoot/LoginTest.php</file> <file>tests/Sparefoot/Myfoot/Facility/DetailsTest.php</file> <file>tests/Sparefoot/Myfoot/Signup/ResidualSignupTest.php</file> <file>tests/Sparefoot/Myfoot/Settings/PaymentTest.php</file> <file>tests/Sparefoot/Myfoot/Statement/CpaDefaultsTest.php</file> <file>tests/Sparefoot/Myfoot/FacilityTest.php</file> <file>tests/Sparefoot/Myfoot/UsersTest.php</file> -
public function setUp() { $this->customer = self::mockCustomer(); } public static
function mockCustomer() { //build up a test object $customer = new \stdClass(); $customer->name = uniqid('John Test'); $customer->email = uniqid('test').'@test.com'; $customer->address = rand(1,5000).' test street'; $cities = ['Roswell', 'Austin', 'Memphis', 'San Jose']; $customer->city = $cities[array_rand($cities)]; $states = ['NC', 'TX', 'VA', 'CT']; $customer->state = $states[array_rand($states)]; $customer->zip = rand(10000,70000); return $customer; } public function testUpdateCustomer() { return ; Pass & set FK relationships for related models -
*/ public static function generateValidCreditCardNumber($leader = 4) { //set to
amex 15 digit if prefix is 3. otherwise its 16 $totalDigits = (substr($leader, 0, 1) == 3) ? 14 : 15; $forwardCardDigits = str_split($leader); // build out the rest of the digits, except for checksum (last digit) for ($i = 0; $i < $totalDigits - strlen($leader); $i++) { $forwardCardDigits[] = rand(0,9); } $reverseCardDigits = array_reverse($forwardCardDigits); $checksum = 0; foreach ($reverseCardDigits as $key => $value) { if ($key % 2 === 0) { $value = 2 * $value; } if ($value > 9) { $checksum += $value - 9; } else { $checksum += $value; } } //get the least significant digit for the checksum $tmp = str_split($checksum); $lsd = (int) end($tmp); $checksumCompliment = $lsd === 0 ? 0 : 10 - $lsd; return implode(null, $forwardCardDigits) . $checksumCompliment; } -
• Setup the environment in persistence • Load the UI
• Manipulate the UI • Bounce the UI, because last time was just jQuery • Verify the side effects in persistence -
• Setup the environment in persistence • Load the UI
• Manipulate the UI • Bounce the UI, because the first time was just jQuery • Verify the side effects in persistence Tech Debt The test is doing too much or test the service separately Because our logic is everywhere -
• 1:45pm - click build again • 2:00pm - notice
haven’t passed yet today • 2:10pm - bounce selenium grid for luck • 2:30pm - in hip chat debug room for screenshots • 2:40pm - last chance to build and go green on time • 2:55pm - curse violently. Devbot mocks you • 3:10pm - bow out so other teams can deploy • 4:05pm - finally a green build. !Freeze • 6:45pm - DevOps back from kid’s swim practice. Successfully Deploy! -
Intermittent test failure smell • Blame the test • Blame
the CI environment performance • Blame the JavaScript plugins (jQuery whatever) • Realize your code is doing too damn much. • Actually fails a large percentage of the time in prod. -
All the brittle things • Send some emails on account
creation • Step 3 in a sign up flow • Create an envelope via API (docusign) • Redirect via JS • Simulate Abandoning signup and coming back • ‘Sign’ using a third party service • Upload a PDF copy to a different API (salesforce) • Send a ticket to sales via API (zendesk) • Continue signup flow for 3 more steps -
All the things I learned • Selenium environment is brittle.
Use a launcher script to minimize debug time. • Run and expose mysql on 3307 in host and vagrant. • Keep VM selenium ‘trapped’ • No manual config changes • No dependency on injected Envars -
Sticky Bits I run tests both ways. In Vagrant (headless)
and in host OS (interactive) Both are super useful to my workflow: • In the host OS for writing and debugging a broken condition • In the VM for running a suite before opening my pull request -
Sticky Bits (2) Multiple Ant targets (smoke vs full vs
regression) balance • speed of quick feedback • full regression coverage • (don’t trust anyone) Code coverage plug-ins are invalid in the scope for integration level tests. -
• IFRAME->IFRAME is one way trip • Write a dedicated
testsuite API for Datepicker() • Expected, Actual, Whoops-Message • Explicitly wait for _everything_ • Disable tests just like C++ by underscore prefixing • Fail as fast as possible (Temporarily disable long tests higher in the class) Sticky Bits (3) -
• Feature Toggle Iteration • Cookie up with 404 •
add controller/action/behavior ids to the dom • Use identical IDs when FF’ing so the test can follow • Request/Factory/Response Mock Factory • Watch for _test nerving in pull requests Sticky Bits (4) -
It Starts With One Test Code Samples: github.com/matt-land Slides: speakerdeck.com/mattland/
linkedin.com/in/matthewland https://joind.in/13567