Slide 1

Slide 1 text

Testing the Untestable ZendCon 2015

Slide 2

Slide 2 text

Matt Land Code Samples: github.com/matt-land Slides: speakerdeck.com/mattland/ linkedin.com/in/matthewland CERTIFIED A R C H I T E CT 2 https://joind.in/15548

Slide 3

Slide 3 text

Testing the Untestable • Getting Started • Reflection & Static • Database Coupling • Network Services • Do I Need CI • Code Coverage • Ice Cream Cones and BDD • Intermittent Failure To Launch • My Worst UI Test • Sticky Bits • Its Not Me, Its the Environment

Slide 4

Slide 4 text

Getting Started: Just Do It Noise: • PHPUnit • Asserts • Composer/Autoloaders • Namespaces • Coding Standards • Continuous Integration • Best Practices • Mocks Not Noise: • Writing Tests • Passing Tests

Slide 5

Slide 5 text

OO Classes

Slide 6

Slide 6 text

OO Classes

Slide 7

Slide 7 text

PROTECTED PRIVATE PRIVATE PRIVATE PRIVATE PROTECTED PRIVATE PRIVATE PROTECTED PRIVATE OO Classes

Slide 8

Slide 8 text

OO Classes

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

• Sample application: • https://github.com/matt-land/test-samples

Slide 11

Slide 11 text

Continues { "category": "Young Adult", "records": 5, "meta": { "next": "\/whatever?q=345w5", "prev": "\/whatever?q=92342" }, "books": [ { "title": "The Carnival at Bray", "author": "Jessie Ann Foley", "publisher": "Elephant Rock", "price": "12.95", "isbn": "97809895115597" }, { "title": "The Crossover", "author": "Kwame Alexander", "publisher": "Houghton", "price": "16.99", "isbn": "9780544107717" }, { "title": "The Gospel of Winter", "author": "Brendan Kiely", "publisher": "Simon & Schuster\/Margaret K. McElderry", "price": "17.99", "isbn": "9781442484894" }, { "title": "I\u2019ll Give You the Sun", "author": "Jandy Nelson", "publisher": "Dial", "price": "17.99", "isbn": "9780803734968" }, { "title": "Jackaby", "author": "William Ritter", "publisher": "William Ritter", "price": "16.95", "isbn": "9781616203535" } ] } Sample Application

Slide 12

Slide 12 text

public function __construct($jsonString = '[]') { $this->pageDataForView = json_decode($jsonString); $this->books = isset($this->pageDataForView->books) ? $this->pageDataForView- >books : null ; } /** * The first method I want to test */ private function sortBooksByAuthor() { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $this->books; usort($sortedBooks, function($a, $b) { /** * sort by author last name, so first get the last name * or last segment of the name. Will not work with JR/III suffixes */ $authorA = explode(" ", trim($a->author)); $authorB = explode(" ", trim($b->author)); return strcmp($authorA[count($authorA)-1], $authorB[count($authorB)-1]); }); return $sortedBooks; } /** * @param $books * @return mixed * The second method I want to test Private Method I want to test

Slide 13

Slide 13 text

Test My Method #1 • Push a dataset into my database • Write a selenium test • Navigate the Dom tree • Validate results • Test a lot of other things • Slow down the test suite for one test

Slide 14

Slide 14 text

Test My Method #2 • Push a dataset into my database • Add code die(print_r($result,1)) in my app • Look at the code • Comment out the code • Never test it again • Hope no one ever breaks the working function • Hope I never accidentally check in debug code

Slide 15

Slide 15 text

Test My Method #3 • Make fake data and results • Have PHP make a copy of the class • Have PHP make a copy of the function • Change the copied function to public • Call the function with my data • Verify results

Slide 16

Slide 16 text

OO Classes

Slide 17

Slide 17 text

jsonBody = json_decode( file_get_contents(__DIR__ . '/pageDataForView.json') ); } public function _testRenderer() { Setup $this->jsonBody for all tests

Slide 18

Slide 18 text

} public function _testRenderer() { $renderObj = new Render(json_encode($this->jsonBody)); $rendering = $renderObj->toHtml(); $this->fail('This is complex: ' . PHP_EOL . $rendering . " Lets test the methods instead."); } /** * reflecting a method, where we have to set the value in the class first */ public function testSortBooksByAuthor() { $reflectedClass = new ReflectionClass('\\Samples\\Render'); $realClass = $reflectedClass->newInstance(); $booksProperty = $reflectedClass->getProperty('books'); $booksProperty->setAccessible(1); $booksProperty->setValue($realClass, $this->jsonBody->books); $sortByAuthorMethod = $reflectedClass->getMethod('sortBooksByAuthor'); $sortByAuthorMethod->setAccessible(true); $sortedBooks = $sortByAuthorMethod->invoke($realClass); $this->assertEquals('Kwame Alexander', $sortedBooks[0]->author); $this->assertEquals('William Ritter', $sortedBooks[4]->author); } /** * this time written statically/no object oriented dependency * Much easier */ Test that uses reflection 10 LOC

Slide 19

Slide 19 text

private function sortBooksByAuthor() { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $this->books; usort($sortedBooks, function ($a, $b) { /** * sort by author last name, so first get the last name * or last segment of the name. Will not work with JR/III suffixes */ $authorA = explode(" ", trim($a->author)); $authorB = explode(" ", trim($b->author)); return strcmp($authorA[count($authorA) - 1], $authorB[count($authorB) - 1]); }); return $sortedBooks; } public function testSortBooksByAuthor() { $reflectedClass = new ReflectionClass('\\Samples\\Render'); $realClass = $reflectedClass->newInstance(); $booksProperty = $reflectedClass->getProperty('books'); $booksProperty->setAccessible(1); $booksProperty->setValue($realClass, $this->jsonBody->books); $sortByAuthorMethod = $reflectedClass->getMethod('sortBooksByAuthor'); $sortByAuthorMethod->setAccessible(true); $sortedBooks = $sortByAuthorMethod->invoke($realClass); $this->assertEquals('Kwame Alexander', $sortedBooks[0]->author); $this->assertEquals('William Ritter', $sortedBooks[4]->author); }

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

/** * @param $books * @return mixed * The second method I want to test */ private static function sortBooksByPrice($books) { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $books; usort($sortedBooks, function($a, $b) { /** * compare decimal price */ return $a->price >= $b->price; }); return $sortedBooks; } public function testSortBooksByPrice() { $realClass = new Render(); Second Method I want to test Changed to static Parameter is passed in Gave up $this, but still have self::

Slide 22

Slide 22 text

*/ return $a->price >= $b->price; }); return $sortedBooks; } public function testSortBooksByPrice() { $realClass = new Render(); $sortByPriceMethod = new ReflectionMethod( $realClass, 'sortBooksByPrice' ); $sortByPriceMethod->setAccessible(true); $sortedBooks = $sortByPriceMethod->invoke( $realClass, $this->jsonBody->books ); $this->assertEquals(12.95, $sortedBooks[0]->price); $this->assertEquals(17.99, $sortedBooks[4]->price); } } Testing Second Method with Reflection 6 LOC

Slide 23

Slide 23 text

* The second method I want to test */ private static function sortBooksByPrice($books) { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $books; usort($sortedBooks, function($a, $b) { /** * compare decimal price */ return $a->price >= $b->price; }); return $sortedBooks; } public function testSortBooksByPrice() { $realClass = new Render(); $sortByPriceMethod = new ReflectionMethod( $realClass, 'sortBooksByPrice' ); $sortByPriceMethod->setAccessible(true); $sortedBooks = $sortByPriceMethod->invoke( $realClass, $this->jsonBody->books ); $this->assertEquals(12.95, $sortedBooks[0]->price); $this->assertEquals(17.99, $sortedBooks[4]->price); }

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

private $books, $jsonBody; public static function sortBooksByIsbn($books) { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $books; usort($sortedBooks, function($a, $b) { /** * sort by ISBN string which might being with zero */ return strcmp($a->isbn, $b->isbn); }); return $sortedBooks; } public function testSortBooksByISBN() { $sortedBooks = Render::sortBooksByIsbn($this->jsonBody- >books); $this->assertEquals('9780544107717', $sortedBooks[0]->isbn); $this->assertEquals('9781616203535', $sortedBooks[4]->isbn); Third Method I want to test Changed to public No side effect risk

Slide 26

Slide 26 text

//so lets copy it to prevent side effects $sortedBooks = $books; usort($sortedBooks, function($a, $b) { /** * sort by ISBN string which might being with zero */ return strcmp($a->isbn, $b->isbn); }); return $sortedBooks; } public function testSortBooksByISBN() { $sortedBooks = Render::sortBooksByIsbn($this->jsonBody->books); $this->assertEquals('9780544107717', $sortedBooks[0]->isbn); $this->assertEquals('9781616203535', $sortedBooks[4]->isbn); } } Testing Third Method without Reflection 3 LOC

Slide 27

Slide 27 text

public static function sortBooksByIsbn($books) { //usort is going to modify $books by reference, //so lets copy it to prevent side effects $sortedBooks = $books; usort($sortedBooks, function($a, $b) { /** * sort by ISBN string which might being with zero */ return strcmp($a->isbn, $b->isbn); }); return $sortedBooks; } public function testSortBooksByISBN() { $sortedBooks = Render::sortBooksByIsbn($this->jsonBody->books); $this->assertEquals('9780544107717', $sortedBooks[0]->isbn); $this->assertEquals('9781616203535', $sortedBooks[4]->isbn); } }

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

Visibility • Transformation cases • Can remove or replace $this for self in method • Not hitting the database • No side effects or • Calculated side effects • Add object type hints • Can’t do it another way Reflection Use Both

Slide 30

Slide 30 text

Solving Database Coupling • Each test does not have to have an empty DB: • Refresh from DB from master as often as ‘reasonable’ • unique() will prevent column restrictions and hash collisions • Hacks are OK, at the level “some tests > no tests”

Slide 31

Slide 31 text

class ServiceTest extends \PHPUnit_Framework_TestCase { private $customer; public function setUp() { $this->customer = self::mockCustomer(); } public static function mockCustomer() { //build up a test object $customer = new \stdClass(); $customer->name = uniqid('John Test'); $customer->email = uniqid('test').'@test.com'; $customer->address = rand(1,5000).' test street'; $cities = ['Roswell', 'Austin', 'Memphis', 'San Jose']; $customer->city = $cities[array_rand($cities)]; $states = ['NC', 'TX', 'VA', 'CT']; $customer->state = $states[array_rand($states)]; $customer->zip = rand(10000,70000); return $customer; } public function testUpdateCustomer() { return ; $service = new Service($sandboxKey = 1212312, $sandboxPassword = 5555 $id = $service->updateCustomer($this->customer); $this->assertGreaterThan(0, $id); Fixes Hash problems Makes debug easier for relational records Strings should have different shapes in UI Semi-unique response for each request

Slide 32

Slide 32 text

* @return string */ public static function generateValidCreditCardNumber($leader = 4) { //set to amex 15 digit if prefix is 3. otherwise its 16 $totalDigits = (substr($leader, 0, 1) == 3) ? 14 : 15; $forwardCardDigits = str_split($leader); // build out the rest of the digits, except for checksum (last digit) for ($i = 0; $i < $totalDigits - strlen($leader); $i++) { $forwardCardDigits[] = rand(0,9); } $reverseCardDigits = array_reverse($forwardCardDigits); $checksum = 0; foreach ($reverseCardDigits as $key => $value) { if ($key % 2 === 0) { $value = 2 * $value; } if ($value > 9) { $checksum += $value - 9; } else { $checksum += $value; } } //get the least significant digit for the checksum $tmp = str_split($checksum); $lsd = (int) end($tmp); $checksumCompliment = $lsd === 0 ? 0 : 10 - $lsd; return implode(null, $forwardCardDigits) . $checksumCompliment; }

Slide 33

Slide 33 text

Mock* Factories • Build a mock factory for every model • Build factories that chain mock object relationships together • Test complex things, start small User Payment Method Account Location Booking Statement Batch Statement Sales Order Invoice

Slide 34

Slide 34 text

DB Coupling: Hacks DB’s don’t have to be refreshed between every test. Some things can be left in the DB by the test. Test code can be written to minimize leftovers.

Slide 35

Slide 35 text

{ private $books, $jsonBody; public function testFindRecordLeavesAMess() { $insertIds = []; $this->pdo->insert('some record'); $insertIds[] = $this->pdo->getLastInsertId(); $this->pdo->insert('some record2'); $insertIds[] = $this->pdo->getLastInsertId(); ... $records = My\Service::findRecord('my report'); $found = false; foreach ($records as $record) { if ($record->someCondition() !== false) { $found = true; } } if (! $found) { $this->fail('did not find the record we expected'); } foreach ($insertIds as $id) { $this->pdo->delete($id); } } Test will leave records any time it fails

Slide 36

Slide 36 text

{ private $books, $jsonBody; public function testFindRecordTeardownFirst() { $insertIds = []; $this->pdo->insert('some record'); $insertIds[] = $this->pdo->getLastInsertId(); $this->pdo->insert('some record2'); $insertIds[] = $this->pdo->getLastInsertId(); ... $records = My\Service::findRecord('my report'); $found = false; foreach ($records as $record) { if ($record->someCondition() !== false) { $found = true; } } foreach ($insertIds as $id) { $this->pdo->delete($id); } if (! $found) { $this->fail('did not find the record we expected'); } } This test will clean up before failing

Slide 37

Slide 37 text

Network Services • Wiremock & Wiremock PHP • Works at the network layer like Man-in-middle attack • Intercepts service requests • Returns mock data or routes to a service provider, based on your pre-defined logic • http://wiremock.org/ • https://github.com/rowanhill/wiremock-php

Slide 38

Slide 38 text

Do I need CI?

Slide 39

Slide 39 text

switch (“Do I need CI?”) { case Agile Team: return ‘Yes’; case count(Team) < ~n: return ‘No’; case count(Team) = ~n: return ‘Maybe’; case count(Team) > ~n: return ‘Yes’; default: return ‘Yes’; }

Slide 40

Slide 40 text

How many people is n? • One person: I have self discipline. • Two people: I know to blame the other one. • Three people: Things were fine until a snake entered the garden.

Slide 41

Slide 41 text

… feature speed increases. … regressions can stop. … iteration and feedback cycles can increase. In exchange for a one-time, but massive PIA setup… Talk on the how tomorrow at 11am

Slide 42

Slide 42 text

With the addition of CI, each test becomes a contract, between the feature and the maintainers of the application. It says, ‘This feature was important enough to have a test written for it. This contract forbids you from (accidentally) breaking said feature.’ Regressions

Slide 43

Slide 43 text

“It doesn’t matter if it works or not. Just make the tests pass so we can deploy.” Then Again… …its not for everyone.

Slide 44

Slide 44 text

Try CI with training wheels first. This is Travis CI

Slide 45

Slide 45 text

A real project in Bamboo

Slide 46

Slide 46 text

Information Radiators are a part of CI

Slide 47

Slide 47 text

Code Coverage

Slide 48

Slide 48 text

tests Load xdebug extension and add this block to phpunit.xml https://github.com/matt-land/test-samples results appear in /codeCoverage

Slide 49

Slide 49 text

Ice Cream Cones & Behavior Driven Development

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

How does it get this way?

Slide 53

Slide 53 text

• Store the data in persistence • Load the UI • Manipulate the UI • Bounce the UI, since first time was just jQuery • Verify the side effects in persistence Typical UI Test

Slide 54

Slide 54 text

• Store the data in persistence • Load the UI • Manipulate the UI • Bounce the UI, since first time was just jQuery • Verify the side effects in persistence Tech Debt The test is doing too much, or test the service separately Because our logic is everywhere Because the DB is coupled Because our logic is everywhere Typical UI Test

Slide 55

Slide 55 text

• Store the data in persistence • Load the UI • Manipulate the UI • Bounce the UI, since first time was just jQuery • Verify the side effects in persistence Tech Debt The test is doing too much, or test the service separately Because our logic is everywhere Because the DB is coupled Because our logic is everywhere The result is each test is very slow. Typical UI Test

Slide 56

Slide 56 text

• Store the data in persistence • Load the UI • Manipulate the UI • Bounce the UI, since first time was just jQuery • Verify the side effects in persistence Tech Debt The test is doing too much, or test the service separately Because our logic is everywhere Because the DB is coupled Because our logic is everywhere Its OK! Tests > No Tests, and tech debt is fixable. But its only OK up to a point. Typical UI Test

Slide 57

Slide 57 text

Intermittent Failure To Launch

Slide 58

Slide 58 text

• 1:45pm - start the build again • 2:00pm - notice build hasn’t passed yet today • 2:10pm - bounce selenium grid for luck • 2:30pm - look in hip chat debug room for screenshots • 2:40pm - deploy time is coming. Last chance to go green in time • 2:55pm - build failure. curse violently. Devbot mocks me • 3:10pm - bow out of the deployment • 4:05pm - finally a green build. !Freeze the build, indefinitely • 6:45pm - DevOps is back from kid’s swim practice • Successfully Deploy!

Slide 59

Slide 59 text

Intermittent test failure smell • Blame the test • Blame the CI environment • Blame CI environment performance • Blame the JavaScript plugins (jQuery whatever) • Your MVC code is doing too damn much • Code actually fails a large percentage of the time in prod. You just hadn’t noticed.

Slide 60

Slide 60 text

My Worst UI Test

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

Learn from this • Selenium is a whole other level of difficulty • Test on hotel Wifi, see if you can pass • Remove side effects from controllers • Sending Emails • Uploads to S3 • Calls to push data to other SaaS’s

Slide 63

Slide 63 text

Its not me, its the environment

Slide 64

Slide 64 text

Testing the Untestable Code Samples: github.com/matt-land Slides: speakerdeck.com/mattland/ linkedin.com/in/matthewland https://joind.in/15548

Slide 65

Slide 65 text

Bonus

Slide 66

Slide 66 text

Sticky Bits I run tests both ways. In Vagrant (headless) and in host OS (interactive) Both are super useful to my workflow: • In the host OS for writing and debugging a broken condition • In the VM for running a suite before opening my pull request

Slide 67

Slide 67 text

Sticky Bits (2) Multiple Ant targets (smoke vs full vs regression) balance • speed of quick feedback • full regression coverage • (don’t trust anyone) Code coverage plug-ins are invalid in the scope for integration level tests.

Slide 68

Slide 68 text

Sticky Bits (3) • IFRAME->IFRAME is one way trip • Write a dedicated testsuite API for Datepicker() • Expected, Actual, Whoops-Message • Explicitly wait for _everything_ • Disable tests just like C++ by underscore prefixing • Fail as fast as possible (Temporarily disable long tests higher in the class)

Slide 69

Slide 69 text

Sticky Bits (4) • Feature Toggle Iteration • Cookie up with 404 • add controller/action/behavior ids to the dom • Use identical IDs when FF’ing so the test can follow • Request/Factory/Response Mock Factory • Watch for _test nerving in pull requests