Dealing with Testing Fatigue

Slides from MageTestFest

Transcript

1. Dealing With Testing
   Fatigue
   MAGE
   TEST
   FEST
   Fabian Schmengler
   

   View Slide

2. Fabian Schmengler
   Developer and partner at integer_net
   Testing Magento since 2011
   Magento Master 2017
   @fschmengler
   

   View Slide

3. Did you know?
   3 / 62
   Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
   

   View Slide

4. What I'm hearing a lot:
   “ I'd really like to write more tests but I don't get any
   practical value from them. It takes too much time, or
   it is too hard. Therefore I don't enjoy it or stopped
   doing it.
   (I totally understand that!)
   4 / 62
   Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
   

   View Slide

5. Another popular opinion:
   ‘ Integration tests are superior to unit tests. Do not
   waste your time with unit tests.
   (at least they are writing tests!)
   5 / 62
   Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
   

   View Slide

6. 6 / 62
   Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
   

   View Slide

7. Spoiler: A healthy mix of tests is best
   Things are mostly ne
   07:32 - 21. Mai 2016
   36 5.588 5.472
   The Practical Dev
   @ThePracticalDev
   7 / 62
   Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
   

   View Slide

8. Why all the
   frustration?
   8 / 62
   Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
   

   View Slide

9. http://xunitpatterns.com/
   "Test smells"
   Project smells
   Behavior smells
   Code smells
   9 / 62
   Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
   

   View Slide

10. Why all the frustration?
    The project smells
    Buggy tests
    High test maintenance cost
    Production bugs
    Developers not writing tests
    10 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

11. Why all the frustration?
    The behavior smells
    Fragile Test
    Erratic Test
    Slow Test
    Frequent Debugging
    Manual Intervention
    11 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

12. I've been through it, too...
    12 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

13. ...but did I give up?
    13 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

14. ...but did I give up?
    NOPE
    There's a way out!
    13 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

15. (It's not easy!)
    14 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

16. First and foremost:
    Learn!
    Testing is a different skill than coding
    15 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

17. Test the right things!
    No, "everything" is not the answer
    16 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

18. Write testable code!
    If code is hard to test, make it easy to
    test
    17 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

19. Write testable code
    Dependency
    Injection
    18 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

20. Write testable code
    Dependency
    Inversion
    19 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

21. Write testable code
    Ports & Adapters
    20 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

22. 21 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

23. 22 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

24. Improve test
    architecture
    Know your (anti-)patterns
    23 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

25. Test phases
    1. Setup
    set up xture (instantiate objects)
    2. Exercise
    interact with subject under test (SUT)
    3. Verify
    assertions
    4. Teardown
    clean up xture
    24 / 62
    

    View Slide

26. Fixture setup strategy
    Prebuilt xture
    Shared xture
    Fresh xture
    25 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

27. Fixture setup strategy
    Prebuilt xture
    e.g. Magento integration test database
    Shared xture
    Fresh xture
    26 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

28. Fixture setup strategy
    Prebuilt xture
    Shared xture
    shared by whole test suite
    e.g. Magento integration test object manager,
    Magento unit test code generation
    shared by test case
    setUpBeforeClass(), or lazy setup
    Fresh xture
    27 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

29. Fixture setup strategy
    Prebuilt xture
    Shared xture
    Fresh xture
    Inline in test method
    Implicitly in setUp()
    28 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

30. Testing smell
    Copy and paste
    Duplicated logic in test code
    dev/tests/integration/testsuite/Magento/Customer/_files/customer.php
    dev/tests/integration/testsuite/Magento/Customer/_files/two_customers.php
    dev/tests/integration/testsuite/Magento/Customer/_files/three_customers.php
    dev/tests/integration/testsuite/Magento/Customer/_files/twenty_one_customers.php
    29 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

31. Problem
    increasing cost of changes
    Possible solutions
    Extract method refactoring
    In setup:
    Creation method for fresh xture
    Finder method for shared xture
    In veri cation:
    Custom assertion method
    30 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

32. Problem
    increasing cost of changes
    Possible solutions
    Extract class refactoring
    In setup:
    Test data builder
    In veri cation:
    Custom constraint
    31 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

33. Testing pattern
    Constraint
    class IsMeaningOfLife extends \PHPUnit\Framework\Constraint
    {
    protected function matches($other)
    {
    return $other === 42;
    }
    protected function failureDescription($other)
    {
    return sprintf(
    "%s is not the meaning of life, the universe and everything", $other
    );
    }
    }
    32 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

34. Testing pattern
    Constraint
    Usage
    $this->assertThat(42, new IsMeaningOfLife());
    33 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

35. Testing pattern
    Test data builder
    Creates xture, reusable across tests
    34 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

36. Test data builder
    Example
    CustomerBuilder::aCustomer()
    ->withEmail('test@example.com')
    ->withCustomAttributes(
    [
    'my_custom_attribute' => 42
    ]
    )
    ->build()
    https://github.com/tddwizard/magento2- xtures
    35 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

37. Testing smell
    Irrelevant information
    36 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

38. $customer->setWebsiteId(
    1
    )->setId(
    1
    )->setEntityTypeId(
    1
    )->setAttributeSetId(
    1
    )->setEmail(
    'customer@example.com'
    )->setPassword(
    'password'
    )->setGroupId(
    1
    )->setStoreId(
    1
    )->setIsActive(
    1
    )->setFirstname(
    'Firstname'
    )->setLastname(
    'Lastname'
    )->setDefaultBilling(
    1
    )->setDefaultShipping(
    1
    )->setRpToken(
    '8ed8677e6c79e68b94e61658bd756ea5'
    )->setRpTokenCreatedAt(
    date('Y-m-d H:i:s')
    );
    37 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

39. Problem
    Hard to read test as documentation
    Possible solutions
    Extract creation methods or assertions with relevant
    parameters
    For xtures: Test data builder
    38 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

40. Testing Smell
    Hard-coded test data
    public function testGetTranslationFileTimestamp()
    {
    $this->fileManagerMock->expects($this->once())
    ->method('getTranslationFileTimestamp')
    ->willReturn(1445736974);
    $this->assertEquals(1445736974, $this->model->getTranslationFileTimestamp());
    }
    39 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

41. Problem
    Hard to understand cause and effect
    Possible solutions
    Introduce variables or constants
    40 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

42. Testing smell
    Mystery guest
    Set up or veri cation relies on information that's not
    visible in the test
    Filename
    Database record
    General xture
    41 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

43. Mystery Guest
    Filename
    public function testBundleImport()
    {
    // import data from CSV file
    $pathToFile = __DIR__ . '/../../_files/import_bundle.csv';
    // ...
    }
    42 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

44. Mystery Guest
    Database record
    /**
    * @magentoAppArea adminhtml
    * @magentoDataFixture Magento/Reports/_files/viewed_products.php
    */
    public function testExecute()
    {
    $this->dispatch('backend/admin/dashboard/productsViewed/');
    $this->assertEquals(200, $this->getResponse()->getHttpResponseCode());
    $actual = $this->getResponse()->getBody();
    $this->assertContains('Simple Product', $actual);
    }
    43 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

45. Problem
    Hard to understand cause and effect
    Risk of somebody else editing external source without
    knowing how it's used
    Possible solutions
    Fresh xture for each test
    Test data builders instead of xture scripts
    Create les in test
    Accurately named nder methods
    44 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

46. General Fixture
    Using attribute
    $this->assertTrue($this->product->isNew(), 'Product should be new');
    Using nder method
    $this->assertTrue(
    $this->getRecentlyAddedProduct()->isNew(),
    'Product should be new'
    );
    private function getRecentlyAddedProduct()
    {
    return $this->product;
    }
    45 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

47. Testing smell
    Eager test
    $savedStockItem->setQty(null);
    $savedStockItem->save();
    $savedStockItem->setQty(2);
    $savedStockItem->save();
    $this->assertEquals('2.0000', $savedStockItem->load($savedStockItemId)->getQty());
    $savedStockItem->setQty(0);
    $savedStockItem->save();
    $this->assertEquals('0.0000', $savedStockItem->load($savedStockItemId)->getQty());
    $savedStockItem->setQty(null);
    $savedStockItem->save();
    $this->assertEquals(null, $savedStockItem->load($savedStockItemId)->getQty());
    46 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

48. Problem
    Hard to tell apart set up, exercise and veri cation
    Harder to localize defects
    Possible solutions
    Single condition tests
    Create one test case (class) per case, not per subject
    47 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

49. Testing smell
    Excessive Mocking
    $baseCurrency->expects()->method('getCode')->willReturn('USD');
    $baseCurrency->expects()->method('getCurrencySymbol')->willReturn('$');
    $baseCurrency->expects()->method('getRate')->with('AUD')->willReturn('0.80');
    $store->expects()->method('getBaseCurrency')->willReturn($baseCurrency);
    $store->expects()->method('getDefaultCurrency')->willReturn($baseCurrency);
    $store->expects()->method('getAvailableCurrencyCodes')->willReturn(['AUD']);
    $this->storeManager->expects()->method('getStore')->willReturn($store);
    48 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

50. Problem
    Coupling test to implementation
    Test is hard to read
    Possible solutions
    Use real domain objects where possible
    Refactor production code
    Use custom stub or fake objects with simpler setup
    49 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

51. Testing pattern
    Fake object
    Test speci c implementation without external
    dependencies
    50 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

52. Fake object
    interface CustomerSession
    {
    public function isLoggedIn() : bool;
    }
    class FakeSession implements CustomerSession
    {
    private $loggedIn = false;
    public function isLoggedIn() : bool
    {
    return $this->loggedIn;
    }
    // test methods:
    public function login()
    {
    $this->loggedIn = true;
    }
    public function logout()
    {
    $this->loggedIn = false;
    }
    }
    51 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

53. Fake object
    PHPUnit stub:
    $session = $this->createMock(CustomerSession::class);
    $session->method('isLoggedIn')->willReturn(true);
    Custom fake:
    $session = new FakeSession();
    $session->login();
    more succinct
    reusable accross tests
    52 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

54. Testing Smell
    Meaningless test names
    public function testProcessValidStoreCodeCase1()
    {
    // ...
    }
    public function testProcessValidStoreCodeCase2()
    {
    // ...
    }
    public function testProcessValidStoreCodeCase3()
    {
    // ...
    }
    53 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

55. Problem
    Failures in test results are unclear
    Possible solutions
    Write meaningful test names
    Use named keys in data provider arrays
    return [
    'meaningful description of case 1' => [ ... ],
    'meaningful description of case 2' => [ ... ],
    ]
    54 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

56. Testing Smell
    Meaningless assertion
    messages
    Failed asserting that 0 is 1.
    Failed asserting that false is true.
    55 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

57. Problem
    Failures in test results are unclear
    Possible solutions
    use the $message parameter in assert methods to add
    custom message
    56 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

58. Testing smell
    Conditional test logic
    public function testSend($configValue, $forceSyncMode, $isComment, $sendingResult)
    {
    // ...
    if (!$isComment) {
    // ...
    }
    // ...
    if (!$configValue || $forceSyncMode) {
    // ...
    if ($sendingResult) {
    // ...
    } else {
    // ...
    }
    } else {
    // ...
    }
    }
    57 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

59. Problem
    Harder to understand the test
    Harder to write the test correctly
    Possible solutions
    Write separate test cases, extract duplicated logic
    Make sure, tests are deterministic
    58 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

60. More smells
    Test logic in production
    General xture
    Using resources (DB, HTTP) in unit tests
    Indirect testing
    "Inappropiate intimacy"
    59 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

61. Conclusion
    Maintainability and stability of tests can be greatly
    improved by taking care of the test architecture.
    Watch out for these typical problems. Get rid of them.
    MAKE
    TESTING
    FUN
    60 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

62. Conclusion
    Maintainability and stability of tests can be greatly
    improved by taking care of the test architecture.
    Watch out for these typical problems. Get rid of them.
    MAKE
    TESTING
    FUN
    AGAIN
    60 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

63. THANK YOU
    61 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

64. THANK YOU
    Coming soon:
    Test Driven Magento by
    Example
    http://tddwizard.com/
    @fschmengler
    61 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide

65. Image sources
    Head in sand: tropical.pete @Flickr CC-BY-SA 2.0
    62 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler
    

    View Slide