Dealing with Testing Fatigue

Slide 1

Slide 1 text

Dealing With Testing Fatigue MAGE TEST FEST Fabian Schmengler

Slide 2

Slide 2 text

Fabian Schmengler Developer and partner at integer_net Testing Magento since 2011 Magento Master 2017 @fschmengler

Slide 3

Slide 3 text

Did you know? 3 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 4

Slide 4 text

What I'm hearing a lot: “ I'd really like to write more tests but I don't get any practical value from them. It takes too much time, or it is too hard. Therefore I don't enjoy it or stopped doing it. (I totally understand that!) 4 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 5

Slide 5 text

Another popular opinion: ‘ Integration tests are superior to unit tests. Do not waste your time with unit tests. (at least they are writing tests!) 5 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 6

Slide 6 text

6 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 7

Slide 7 text

Spoiler: A healthy mix of tests is best Things are mostly ne 07:32 - 21. Mai 2016 36 5.588 5.472 The Practical Dev @ThePracticalDev 7 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 8

Slide 8 text

Why all the frustration? 8 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 9

Slide 9 text

http://xunitpatterns.com/ "Test smells" Project smells Behavior smells Code smells 9 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 10

Slide 10 text

Why all the frustration? The project smells Buggy tests High test maintenance cost Production bugs Developers not writing tests 10 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 11

Slide 11 text

Why all the frustration? The behavior smells Fragile Test Erratic Test Slow Test Frequent Debugging Manual Intervention 11 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 12

Slide 12 text

I've been through it, too... 12 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 13

Slide 13 text

...but did I give up? 13 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 14

Slide 14 text

...but did I give up? NOPE There's a way out! 13 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 15

Slide 15 text

(It's not easy!) 14 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 16

Slide 16 text

First and foremost: Learn! Testing is a different skill than coding 15 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 17

Slide 17 text

Test the right things! No, "everything" is not the answer 16 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 18

Slide 18 text

Write testable code! If code is hard to test, make it easy to test 17 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 19

Slide 19 text

Write testable code Dependency Injection 18 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 20

Slide 20 text

Write testable code Dependency Inversion 19 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 21

Slide 21 text

Write testable code Ports & Adapters 20 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 22

Slide 22 text

21 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 23

Slide 23 text

22 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 24

Slide 24 text

Improve test architecture Know your (anti-)patterns 23 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 25

Slide 25 text

Test phases 1. Setup set up xture (instantiate objects) 2. Exercise interact with subject under test (SUT) 3. Verify assertions 4. Teardown clean up xture 24 / 62

Slide 26

Slide 26 text

Fixture setup strategy Prebuilt xture Shared xture Fresh xture 25 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 27

Slide 27 text

Fixture setup strategy Prebuilt xture e.g. Magento integration test database Shared xture Fresh xture 26 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 28

Slide 28 text

Fixture setup strategy Prebuilt xture Shared xture shared by whole test suite e.g. Magento integration test object manager, Magento unit test code generation shared by test case setUpBeforeClass(), or lazy setup Fresh xture 27 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 29

Slide 29 text

Fixture setup strategy Prebuilt xture Shared xture Fresh xture Inline in test method Implicitly in setUp() 28 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 30

Slide 30 text

Testing smell Copy and paste Duplicated logic in test code dev/tests/integration/testsuite/Magento/Customer/_files/customer.php dev/tests/integration/testsuite/Magento/Customer/_files/two_customers.php dev/tests/integration/testsuite/Magento/Customer/_files/three_customers.php dev/tests/integration/testsuite/Magento/Customer/_files/twenty_one_customers.php 29 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 31

Slide 31 text

Problem increasing cost of changes Possible solutions Extract method refactoring In setup: Creation method for fresh xture Finder method for shared xture In veri cation: Custom assertion method 30 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 32

Slide 32 text

Problem increasing cost of changes Possible solutions Extract class refactoring In setup: Test data builder In veri cation: Custom constraint 31 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 33

Slide 33 text

Testing pattern Constraint class IsMeaningOfLife extends \PHPUnit\Framework\Constraint { protected function matches($other) { return $other === 42; } protected function failureDescription($other) { return sprintf( "%s is not the meaning of life, the universe and everything", $other ); } } 32 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 34

Slide 34 text

Testing pattern Constraint Usage $this->assertThat(42, new IsMeaningOfLife()); 33 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 35

Slide 35 text

Testing pattern Test data builder Creates xture, reusable across tests 34 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 36

Slide 36 text

Test data builder Example CustomerBuilder::aCustomer() ->withEmail('[email protected]') ->withCustomAttributes( [ 'my_custom_attribute' => 42 ] ) ->build() https://github.com/tddwizard/magento2- xtures 35 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 37

Slide 37 text

Testing smell Irrelevant information 36 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 38

Slide 38 text

$customer->setWebsiteId( 1 )->setId( 1 )->setEntityTypeId( 1 )->setAttributeSetId( 1 )->setEmail( '[email protected]' )->setPassword( 'password' )->setGroupId( 1 )->setStoreId( 1 )->setIsActive( 1 )->setFirstname( 'Firstname' )->setLastname( 'Lastname' )->setDefaultBilling( 1 )->setDefaultShipping( 1 )->setRpToken( '8ed8677e6c79e68b94e61658bd756ea5' )->setRpTokenCreatedAt( date('Y-m-d H:i:s') ); 37 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 39

Slide 39 text

Problem Hard to read test as documentation Possible solutions Extract creation methods or assertions with relevant parameters For xtures: Test data builder 38 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 40

Slide 40 text

Testing Smell Hard-coded test data public function testGetTranslationFileTimestamp() { $this->fileManagerMock->expects($this->once()) ->method('getTranslationFileTimestamp') ->willReturn(1445736974); $this->assertEquals(1445736974, $this->model->getTranslationFileTimestamp()); } 39 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 41

Slide 41 text

Problem Hard to understand cause and effect Possible solutions Introduce variables or constants 40 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 42

Slide 42 text

Testing smell Mystery guest Set up or veri cation relies on information that's not visible in the test Filename Database record General xture 41 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 43

Slide 43 text

Mystery Guest Filename public function testBundleImport() { // import data from CSV file $pathToFile = __DIR__ . '/../../_files/import_bundle.csv'; // ... } 42 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 44

Slide 44 text

Mystery Guest Database record /** * @magentoAppArea adminhtml * @magentoDataFixture Magento/Reports/_files/viewed_products.php */ public function testExecute() { $this->dispatch('backend/admin/dashboard/productsViewed/'); $this->assertEquals(200, $this->getResponse()->getHttpResponseCode()); $actual = $this->getResponse()->getBody(); $this->assertContains('Simple Product', $actual); } 43 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 45

Slide 45 text

Problem Hard to understand cause and effect Risk of somebody else editing external source without knowing how it's used Possible solutions Fresh xture for each test Test data builders instead of xture scripts Create les in test Accurately named nder methods 44 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 46

Slide 46 text

General Fixture Using attribute $this->assertTrue($this->product->isNew(), 'Product should be new'); Using nder method $this->assertTrue( $this->getRecentlyAddedProduct()->isNew(), 'Product should be new' ); private function getRecentlyAddedProduct() { return $this->product; } 45 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 47

Slide 47 text

Testing smell Eager test $savedStockItem->setQty(null); $savedStockItem->save(); $savedStockItem->setQty(2); $savedStockItem->save(); $this->assertEquals('2.0000', $savedStockItem->load($savedStockItemId)->getQty()); $savedStockItem->setQty(0); $savedStockItem->save(); $this->assertEquals('0.0000', $savedStockItem->load($savedStockItemId)->getQty()); $savedStockItem->setQty(null); $savedStockItem->save(); $this->assertEquals(null, $savedStockItem->load($savedStockItemId)->getQty()); 46 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 48

Slide 48 text

Problem Hard to tell apart set up, exercise and veri cation Harder to localize defects Possible solutions Single condition tests Create one test case (class) per case, not per subject 47 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 49

Slide 49 text

Testing smell Excessive Mocking $baseCurrency->expects()->method('getCode')->willReturn('USD'); $baseCurrency->expects()->method('getCurrencySymbol')->willReturn('$'); $baseCurrency->expects()->method('getRate')->with('AUD')->willReturn('0.80'); $store->expects()->method('getBaseCurrency')->willReturn($baseCurrency); $store->expects()->method('getDefaultCurrency')->willReturn($baseCurrency); $store->expects()->method('getAvailableCurrencyCodes')->willReturn(['AUD']); $this->storeManager->expects()->method('getStore')->willReturn($store); 48 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 50

Slide 50 text

Problem Coupling test to implementation Test is hard to read Possible solutions Use real domain objects where possible Refactor production code Use custom stub or fake objects with simpler setup 49 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 51

Slide 51 text

Testing pattern Fake object Test speci c implementation without external dependencies 50 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 52

Slide 52 text

Fake object interface CustomerSession { public function isLoggedIn() : bool; } class FakeSession implements CustomerSession { private $loggedIn = false; public function isLoggedIn() : bool { return $this->loggedIn; } // test methods: public function login() { $this->loggedIn = true; } public function logout() { $this->loggedIn = false; } } 51 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 53

Slide 53 text

Fake object PHPUnit stub: $session = $this->createMock(CustomerSession::class); $session->method('isLoggedIn')->willReturn(true); Custom fake: $session = new FakeSession(); $session->login(); more succinct reusable accross tests 52 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 54

Slide 54 text

Testing Smell Meaningless test names public function testProcessValidStoreCodeCase1() { // ... } public function testProcessValidStoreCodeCase2() { // ... } public function testProcessValidStoreCodeCase3() { // ... } 53 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 55

Slide 55 text

Problem Failures in test results are unclear Possible solutions Write meaningful test names Use named keys in data provider arrays return [ 'meaningful description of case 1' => [ ... ], 'meaningful description of case 2' => [ ... ], ] 54 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 56

Slide 56 text

Testing Smell Meaningless assertion messages Failed asserting that 0 is 1. Failed asserting that false is true. 55 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 57

Slide 57 text

Problem Failures in test results are unclear Possible solutions use the $message parameter in assert methods to add custom message 56 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 58

Slide 58 text

Testing smell Conditional test logic public function testSend($configValue, $forceSyncMode, $isComment, $sendingResult) { // ... if (!$isComment) { // ... } // ... if (!$configValue || $forceSyncMode) { // ... if ($sendingResult) { // ... } else { // ... } } else { // ... } } 57 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 59

Slide 59 text

Problem Harder to understand the test Harder to write the test correctly Possible solutions Write separate test cases, extract duplicated logic Make sure, tests are deterministic 58 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 60

Slide 60 text

More smells Test logic in production General xture Using resources (DB, HTTP) in unit tests Indirect testing "Inappropiate intimacy" 59 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 61

Slide 61 text

Conclusion Maintainability and stability of tests can be greatly improved by taking care of the test architecture. Watch out for these typical problems. Get rid of them. MAKE TESTING FUN 60 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 62

Slide 62 text

Conclusion Maintainability and stability of tests can be greatly improved by taking care of the test architecture. Watch out for these typical problems. Get rid of them. MAKE TESTING FUN AGAIN 60 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 63

Slide 63 text

THANK YOU 61 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 64

Slide 64 text

THANK YOU Coming soon: Test Driven Magento by Example http://tddwizard.com/ @fschmengler 61 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

Slide 65

Slide 65 text

Image sources Head in sand: tropical.pete @Flickr CC-BY-SA 2.0 62 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler