Dealing with Testing Fatigue

Transcript

1. Dealing With Testing Fatigue MAGE TEST FEST Fabian Schmengler

2. Fabian Schmengler Developer and partner at integer_net Testing Magento since

   2011 Magento Master 2017 @fschmengler

3. Did you know? 3 / 62 Fabian Schmengler /> Dealing

   With Testing Fatigue @fschmengler

4. What I'm hearing a lot: “ I'd really like to

   write more tests but I don't get any practical value from them. It takes too much time, or it is too hard. Therefore I don't enjoy it or stopped doing it. (I totally understand that!) 4 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

5. Another popular opinion: ‘ Integration tests are superior to unit

   tests. Do not waste your time with unit tests. (at least they are writing tests!) 5 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

6. 6 / 62 Fabian Schmengler /> Dealing With Testing Fatigue

   @fschmengler

7. Spoiler: A healthy mix of tests is best Things are

   mostly ne 07:32 - 21. Mai 2016 36 5.588 5.472 The Practical Dev @ThePracticalDev 7 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

8. Why all the frustration? 8 / 62 Fabian Schmengler />

   Dealing With Testing Fatigue @fschmengler

9. http://xunitpatterns.com/ "Test smells" Project smells Behavior smells Code smells 9

   / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

10. Why all the frustration? The project smells Buggy tests High

    test maintenance cost Production bugs Developers not writing tests 10 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

11. Why all the frustration? The behavior smells Fragile Test Erratic

    Test Slow Test Frequent Debugging Manual Intervention 11 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

12. I've been through it, too... 12 / 62 Fabian Schmengler

    /> Dealing With Testing Fatigue @fschmengler

13. ...but did I give up? 13 / 62 Fabian Schmengler

    /> Dealing With Testing Fatigue @fschmengler

14. ...but did I give up? NOPE There's a way out!

    13 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

15. (It's not easy!) 14 / 62 Fabian Schmengler /> Dealing

    With Testing Fatigue @fschmengler

16. First and foremost: Learn! Testing is a different skill than

    coding 15 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

17. Test the right things! No, "everything" is not the answer

    16 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

18. Write testable code! If code is hard to test, make

    it easy to test 17 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

19. Write testable code Dependency Injection 18 / 62 Fabian Schmengler

    /> Dealing With Testing Fatigue @fschmengler

20. Write testable code Dependency Inversion 19 / 62 Fabian Schmengler

    /> Dealing With Testing Fatigue @fschmengler

21. Write testable code Ports & Adapters 20 / 62 Fabian

    Schmengler /> Dealing With Testing Fatigue @fschmengler

22. 21 / 62 Fabian Schmengler /> Dealing With Testing Fatigue

    @fschmengler

23. 22 / 62 Fabian Schmengler /> Dealing With Testing Fatigue

    @fschmengler

24. Improve test architecture Know your (anti-)patterns 23 / 62 Fabian

    Schmengler /> Dealing With Testing Fatigue @fschmengler

25. Test phases 1. Setup set up xture (instantiate objects) 2.

    Exercise interact with subject under test (SUT) 3. Verify assertions 4. Teardown clean up xture 24 / 62

26. Fixture setup strategy Prebuilt xture Shared xture Fresh xture 25

    / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

27. Fixture setup strategy Prebuilt xture e.g. Magento integration test database

    Shared xture Fresh xture 26 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

28. Fixture setup strategy Prebuilt xture Shared xture shared by whole

    test suite e.g. Magento integration test object manager, Magento unit test code generation shared by test case setUpBeforeClass(), or lazy setup Fresh xture 27 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

29. Fixture setup strategy Prebuilt xture Shared xture Fresh xture Inline

    in test method Implicitly in setUp() 28 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

30. Testing smell Copy and paste Duplicated logic in test code

    dev/tests/integration/testsuite/Magento/Customer/_files/customer.php dev/tests/integration/testsuite/Magento/Customer/_files/two_customers.php dev/tests/integration/testsuite/Magento/Customer/_files/three_customers.php dev/tests/integration/testsuite/Magento/Customer/_files/twenty_one_customers.php 29 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

31. Problem increasing cost of changes Possible solutions Extract method refactoring

    In setup: Creation method for fresh xture Finder method for shared xture In veri cation: Custom assertion method 30 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

32. Problem increasing cost of changes Possible solutions Extract class refactoring

    In setup: Test data builder In veri cation: Custom constraint 31 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

33. Testing pattern Constraint class IsMeaningOfLife extends \PHPUnit\Framework\Constraint { protected function

    matches($other) { return $other === 42; } protected function failureDescription($other) { return sprintf( "%s is not the meaning of life, the universe and everything", $other ); } } 32 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

34. Testing pattern Constraint Usage $this->assertThat(42, new IsMeaningOfLife()); 33 / 62

    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

35. Testing pattern Test data builder Creates xture, reusable across tests

    34 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

36. Test data builder Example CustomerBuilder::aCustomer() ->withEmail('test@example.com') ->withCustomAttributes( [ 'my_custom_attribute' =>

    42 ] ) ->build() https://github.com/tddwizard/magento2- xtures 35 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

37. Testing smell Irrelevant information 36 / 62 Fabian Schmengler />

    Dealing With Testing Fatigue @fschmengler

38. $customer->setWebsiteId( 1 )->setId( 1 )->setEntityTypeId( 1 )->setAttributeSetId( 1 )->setEmail( 'customer@example.com'

    )->setPassword( 'password' )->setGroupId( 1 )->setStoreId( 1 )->setIsActive( 1 )->setFirstname( 'Firstname' )->setLastname( 'Lastname' )->setDefaultBilling( 1 )->setDefaultShipping( 1 )->setRpToken( '8ed8677e6c79e68b94e61658bd756ea5' )->setRpTokenCreatedAt( date('Y-m-d H:i:s') ); 37 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

39. Problem Hard to read test as documentation Possible solutions Extract

    creation methods or assertions with relevant parameters For xtures: Test data builder 38 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

40. Testing Smell Hard-coded test data public function testGetTranslationFileTimestamp() { $this->fileManagerMock->expects($this->once())

    ->method('getTranslationFileTimestamp') ->willReturn(1445736974); $this->assertEquals(1445736974, $this->model->getTranslationFileTimestamp()); } 39 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

41. Problem Hard to understand cause and effect Possible solutions Introduce

    variables or constants 40 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

42. Testing smell Mystery guest Set up or veri cation relies

    on information that's not visible in the test Filename Database record General xture 41 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

43. Mystery Guest Filename public function testBundleImport() { // import data

    from CSV file $pathToFile = __DIR__ . '/../../_files/import_bundle.csv'; // ... } 42 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

44. Mystery Guest Database record /** * @magentoAppArea adminhtml * @magentoDataFixture

    Magento/Reports/_files/viewed_products.php */ public function testExecute() { $this->dispatch('backend/admin/dashboard/productsViewed/'); $this->assertEquals(200, $this->getResponse()->getHttpResponseCode()); $actual = $this->getResponse()->getBody(); $this->assertContains('Simple Product', $actual); } 43 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

45. Problem Hard to understand cause and effect Risk of somebody

    else editing external source without knowing how it's used Possible solutions Fresh xture for each test Test data builders instead of xture scripts Create les in test Accurately named nder methods 44 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

46. General Fixture Using attribute $this->assertTrue($this->product->isNew(), 'Product should be new'); Using

    nder method $this->assertTrue( $this->getRecentlyAddedProduct()->isNew(), 'Product should be new' ); private function getRecentlyAddedProduct() { return $this->product; } 45 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

47. Testing smell Eager test $savedStockItem->setQty(null); $savedStockItem->save(); $savedStockItem->setQty(2); $savedStockItem->save(); $this->assertEquals('2.0000', $savedStockItem->load($savedStockItemId)->getQty());

    $savedStockItem->setQty(0); $savedStockItem->save(); $this->assertEquals('0.0000', $savedStockItem->load($savedStockItemId)->getQty()); $savedStockItem->setQty(null); $savedStockItem->save(); $this->assertEquals(null, $savedStockItem->load($savedStockItemId)->getQty()); 46 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

48. Problem Hard to tell apart set up, exercise and veri

    cation Harder to localize defects Possible solutions Single condition tests Create one test case (class) per case, not per subject 47 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

49. Testing smell Excessive Mocking $baseCurrency->expects()->method('getCode')->willReturn('USD'); $baseCurrency->expects()->method('getCurrencySymbol')->willReturn('$'); $baseCurrency->expects()->method('getRate')->with('AUD')->willReturn('0.80'); $store->expects()->method('getBaseCurrency')->willReturn($baseCurrency); $store->expects()->method('getDefaultCurrency')->willReturn($baseCurrency); $store->expects()->method('getAvailableCurrencyCodes')->willReturn(['AUD']);

    $this->storeManager->expects()->method('getStore')->willReturn($store); 48 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

50. Problem Coupling test to implementation Test is hard to read

    Possible solutions Use real domain objects where possible Refactor production code Use custom stub or fake objects with simpler setup 49 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

51. Testing pattern Fake object Test speci c implementation without external

    dependencies 50 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

52. Fake object interface CustomerSession { public function isLoggedIn() : bool;

    } class FakeSession implements CustomerSession { private $loggedIn = false; public function isLoggedIn() : bool { return $this->loggedIn; } // test methods: public function login() { $this->loggedIn = true; } public function logout() { $this->loggedIn = false; } } 51 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

53. Fake object PHPUnit stub: $session = $this->createMock(CustomerSession::class); $session->method('isLoggedIn')->willReturn(true); Custom fake:

    $session = new FakeSession(); $session->login(); more succinct reusable accross tests 52 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

54. Testing Smell Meaningless test names public function testProcessValidStoreCodeCase1() { //

    ... } public function testProcessValidStoreCodeCase2() { // ... } public function testProcessValidStoreCodeCase3() { // ... } 53 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

55. Problem Failures in test results are unclear Possible solutions Write

    meaningful test names Use named keys in data provider arrays return [ 'meaningful description of case 1' => [ ... ], 'meaningful description of case 2' => [ ... ], ] 54 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

56. Testing Smell Meaningless assertion messages Failed asserting that 0 is

    1. Failed asserting that false is true. 55 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

57. Problem Failures in test results are unclear Possible solutions use

    the $message parameter in assert methods to add custom message 56 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

58. Testing smell Conditional test logic public function testSend($configValue, $forceSyncMode, $isComment,

    $sendingResult) { // ... if (!$isComment) { // ... } // ... if (!$configValue || $forceSyncMode) { // ... if ($sendingResult) { // ... } else { // ... } } else { // ... } } 57 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

59. Problem Harder to understand the test Harder to write the

    test correctly Possible solutions Write separate test cases, extract duplicated logic Make sure, tests are deterministic 58 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

60. More smells Test logic in production General xture Using resources

    (DB, HTTP) in unit tests Indirect testing "Inappropiate intimacy" 59 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

61. Conclusion Maintainability and stability of tests can be greatly improved

    by taking care of the test architecture. Watch out for these typical problems. Get rid of them. MAKE TESTING FUN 60 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

62. Conclusion Maintainability and stability of tests can be greatly improved

    by taking care of the test architecture. Watch out for these typical problems. Get rid of them. MAKE TESTING FUN AGAIN 60 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

63. THANK YOU 61 / 62 Fabian Schmengler /> Dealing With

    Testing Fatigue @fschmengler

64. THANK YOU Coming soon: Test Driven Magento by Example http://tddwizard.com/

    @fschmengler 61 / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

65. Image sources Head in sand: tropical.pete @Flickr CC-BY-SA 2.0 62

    / 62 Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler