Software Testing: Past, Present, Future William Durand ‑ October 23rd, 2014

Software Testing is the process of executing a program or system with the intent of finding errors. However, "testing shows the presence, not the absence of bugs" (Dijkstra).

Past /pɑːst/

Testing was... Expensive ($$$) Time consuming Complicated

Tools, tools, tools

PHPUnit by Sebastian Bergmann © ® sebastianbergmann/phpunit (Alternative: ) atoum/atoum

Extend It! ® etsy/phpunit‑extensions

Mockery class MockeryTest extends \PHPUnit_Framework_TestCase { public function testGetsAverageTemperatureFromThreeServiceReadings() { $service = \Mockery::mock('service'); $service ->shouldReceive('readTemp') ->times(3) ->andReturn(10, 12, 14); $temperature = new Temperature($service); $this->assertEquals(12, $temperature->average()); } } ® padraic/mockery

Prophecy $prophet = new Prophecy\Prophet(); $prophecy = $prophet->prophesize(); $prophecy->willExtend('stdClass'); $dummy = $prophecy->reveal(); // fake $prophecy->read(Argument::type('string')); $fake = $prophecy->reveal(); // stub $prophecy->read('123')->willReturn('value'); $stub = $prophecy->reveal(); // mock $prophecy->read()->shouldBeCalled(); // spy $prophecy->read()->shouldHaveBeenCalled(); ® phpspec/prophecy

More? Wanna use PHPUnit with Prophecy and Atoum asserters? ® hautelook/frankenstein

vfsStream Mocking file system since 2007. ® mikey179/vfsStream

Example class VfsStreamTest extends \PHPUnit_Framework_TestCase { protected function setUp() { $this->cacheDir = vfsStream::setup('cache'); $this->repository = new YamlUserRepository( vfsStream::url('cache/users.yml') ); } public testAddSerializesUserIntoYaml() { $this->repository->add(new User('John', 'Doe')); $this->assertEquals( '- { first_name: "John", last_name: "Doe" }', $this->cacheDir->getChild('users.yml')->getContent() ); } }

Tests As Documentation

TestDox class EmailParserTest extends \PHPUnit_Framework_TestCase { public function testReadsSimpleBody() { ... } public function testRecognizesDateStringAboveQuote() { ... } public function testDoesNotModifyInputString() { ... } public function testDealsWithMultilineReplyHeaders() { ... } } y $ phpunit --testdox EmailReplyParser\Tests\Parser\EmailParser [x] Reads simple body [x] Recognizes date string above quote [x] Does not modify input string [x] Deals with multiline reply headers F , Dan North (2006) Introducing BDD

Behat Story BDD since . 2010 Feature: ls In order to see the directory structure As a UNIX user I need to be able to list the current directory's contents Scenario: List 2 files in a directory Given I am in a directory "test" And I have a file named "foo" And I have a file named "bar" When I run "ls" Then I should get: """ bar foo """ ® Behat/Behat

phpspec Spec BDD since , yep! 2007 class MarkdownSpec extends ObjectBehavior { function it_converts_plain_text_to_html_paragraphs() { $this->toHtml("Hi, there")->shouldReturn("

Hi, there

"); } } ® phpspec/phpspec

Hooray!? Not really.

Present /ˈpreznt/

Testing is... Slow Manual But, trending (hype?)

Testing, In Parallel ® (Ruby tool!) ® ® grosser/parallel brianium/paratest liuggio/fastest

Continuous Integration

Record & Replay

PHP•VCR class PhpVcrTest extends \PHPUnit_Framework_TestCase { /** * @vcr unittest_annotation_test */ public function testInterceptsWithAnnotations() { // Requests are intercepted and stored into: // `tests/fixtures/unittest_annotation_test` $content = file_get_contents('http://google.com'); $this->assertEquals('some content', $content); // VCR is automatically turned on and off } } ® php‑vcr/php‑vcr

Gor HTTP traffic replication tool, written in Go. ® buger/gor

It is not only about testing your code.

But rather, testing everything.

Security Testing , automatic SQL injection tool , security scanner sqlmap Zapr SensioLabs Security Checker

Web Acceptance Testing

Mink $driver = new \Behat\Mink\Driver\GoutteDriver(); $session = new \Behat\Mink\Session($driver); $session->start(); $session->visit('http://williamdurand.fr'); echo "Status code: ". $session->getStatusCode() . "\n"; echo "Current URL: ". $session->getCurrentUrl() . "\n"; $page = $session->getPage(); $anchorEl = $page->find('css', 'h3 a.title'); ® Behat/Mink

® jlipps/sausage

Fuzzy Testing

gremlins.js ® marmelab/gremlins.js

But also Infrastructure Testing Smoke Testing ( ) Robustness Testing Actually,  Testing smoke.sh

Future /ˈfjuːtʃəʳ/

Testing will be... Automated! More formal

Disclaimer The following techniques are not really "new" from a research point of view.

Model‑based Testing

No content

1 Model w N Implementations

Model ¿ yEd Graph Editor

Code Generation public interface Login { public void e_Init(); public void v_ClientNotRunning(); public void e_StartClient(); public void v_LoginPrompted(); public void e_ValidPremiumCredentials(); public void v_Browse(); } ¿ GraphWalker

Interaction With The SUT public class SimpleTest extends ExecutionContext implements Login { @Override public void e_Init() { ... } ... @Test public void runSmokeTest() { ... } @Test public void runFunctionalTest() { ... } } ¿ Java/JUnit, Robotium, Selenium, Appium, etc.

Sikuli y F http://www.sikuli.org/

Results

Automated Model Generation Story of my life.

Contract‑based Testing

Praspel The missing specification language for PHP. /** * @requires t : array([to integer()],boundinteger(5,10)) and * i : integer(); * @ensures \result : boolean(); * @throwable FooException */ public function find($t, $i) { // ... } ® hoaproject/Praspel

Recap' Congrats for understanding the need for testing Testing is more than using existing tools Test automation is part of your future, start now!

Thank You. Questions? è ¾ ® ¬ joind.in/11953 williamdurand.fr github.com/willdurand twitter.com/couac