Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dealing with Testing Fatigue #mm18pl

Dealing with Testing Fatigue #mm18pl

Slides from Meet Magento Poland 2018

30 Minutes Presentation. Video will be available later, follow https://twitter.com/meetmagentopl

Fabian Schmengler

September 10, 2018
Tweet

More Decks by Fabian Schmengler

Other Decks in Programming

Transcript

  1. Dealing With Testing Fatigue
    Fabian Schmengler

    View full-size slide

  2. Fabian Schmengler
    Developer and partner at integer_net, Germany
    Testing Magento since 2011
    Magento Master 2017-2018
    @fschmengler

    View full-size slide

  3. Did you know?
    3 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  4. What I'm hearing a lot:
    “ I'd really like to write more tests but I don't get any practical value from
    them. It takes too much time, or it is too hard. Therefore I don't enjoy it or
    stopped doing it.
    (I totally understand that!)
    4 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  5. Another popular opinion:
    ‘ Integration tests are superior to unit tests. Do not waste your time with
    unit tests.
    (at least they are writing tests!)
    5 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  6. 6 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  7. 7 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  8. Why all the frustration?
    8 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  9. http://xunitpatterns.com/
    "Test smells"
    Project smells
    Behavior smells
    Code smells
    9 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  10. Why all the frustration?
    The project smells
    Buggy tests
    High test maintenance cost
    Production bugs
    Developers not writing tests
    10 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  11. Why all the frustration?
    The behavior smells
    Fragile Test
    Erratic Test
    Slow Test
    Frequent Debugging
    Manual Intervention
    11 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  12. I've been through it, too...
    12 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  13. ...but did I give up?
    13 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  14. ...but did I give up?
    NOPE
    There's a way out!
    13 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  15. (It's not easy!)
    14 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  16. First and foremost:
    Learn!
    Testing is a different skill than coding
    15 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  17. Test the right things!
    No, "everything" is not the answer
    16 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  18. Write testable code!
    If code is hard to test, make it easy to test
    17 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  19. Write testable code
    Dependency Injection
    18 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  20. Write testable code
    Dependency Inversion
    19 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  21. Write testable code
    Ports & Adapters
    20 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  22. Improve test architecture
    Know your (anti-)patterns
    21 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  23. Test phases
    1. Setup - set up xture (instantiate objects)
    2. Exercise - interact with subject under test (SUT)
    3. Verify - assertions
    4. Teardown - clean up xture
    22 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  24. Fixture setup strategy
    Prebuilt xture
    Shared xture
    Fresh xture
    23 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  25. Fixture setup strategy
    Prebuilt xture
    e.g. Magento integration test database
    Shared xture
    Fresh xture
    24 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  26. Fixture setup strategy
    Prebuilt xture
    Shared xture
    shared by whole test suite
    OR shared by test case
    Fresh xture
    25 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  27. Fixture setup strategy
    Prebuilt xture
    Shared xture
    Fresh xture
    Inline in test method
    Implicitly in setUp()
    26 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  28. Testing smell
    Copy and paste
    Duplicated logic in test code
    dev/tests/integration/testsuite/Magento/Customer/_files/customer.php
    dev/tests/integration/testsuite/Magento/Customer/_files/two_customers.php
    dev/tests/integration/testsuite/Magento/Customer/_files/three_customers.php
    dev/tests/integration/testsuite/Magento/Customer/_files/twenty_one_customers.php
    27 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  29. Problem
    increasing cost of changes
    Possible solutions
    Extract method refactoring
    In setup:
    Creation method for fresh xture
    Finder method for shared xture
    In veri cation:
    Custom assertion method
    28 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  30. Problem
    increasing cost of changes
    Possible solutions
    Extract class refactoring
    In setup:
    Test data builder
    In veri cation:
    Custom constraint
    29 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  31. Testing pattern
    Constraint
    class IsMeaningOfLife extends \PHPUnit\Framework\Constraint
    {
    protected function matches($other)
    {
    return $other === 42;
    }
    protected function failureDescription($other)
    {
    return sprintf(
    "%s is not the meaning of life, the universe and everything", $other
    );
    }
    }
    30 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  32. Testing pattern
    Constraint
    Usage
    $this->assertThat(42, new IsMeaningOfLife());
    31 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  33. Testing pattern
    Test data builder
    Creates xture, reusable across tests
    32 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  34. Test data builder
    Example
    CustomerBuilder::aCustomer()
    ->withEmail('[email protected]')
    ->withCustomAttributes(
    [
    'my_custom_attribute' => 42
    ]
    )
    ->build()
    https://github.com/tddwizard/magento2- xtures
    33 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  35. Testing smell
    Irrelevant information
    34 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  36. $customer->setWebsiteId(
    1
    )->setId(
    1
    )->setEntityTypeId(
    1
    )->setAttributeSetId(
    1
    )->setEmail(
    '[email protected]'
    )->setPassword(
    'password'
    )->setGroupId(
    1
    )->setStoreId(
    1
    )->setIsActive(
    1
    )->setFirstname(
    'Firstname'
    )->setLastname(
    'Lastname'
    )->setDefaultBilling(
    1
    )->setDefaultShipping(
    1
    )->setRpToken(
    '8ed8677e6c79e68b94e61658bd756ea5'
    )->setRpTokenCreatedAt(
    date('Y-m-d H:i:s')
    );
    35 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  37. Problem
    Hard to read test as documentation
    Possible solutions
    Extract creation methods or assertions with relevant parameters
    For xtures: Test data builder
    36 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  38. Testing Smell
    Hard-coded test data
    public function testGetTranslationFileTimestamp()
    {
    $this->fileManagerMock->expects($this->once())
    ->method('getTranslationFileTimestamp')
    ->willReturn(1445736974);
    $this->assertEquals(1445736974, $this->model->getTranslationFileTimestamp());
    }
    37 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  39. Problem
    Hard to understand cause and effect
    Purpose of typical test data (foo, bar, 1, 42) is unclear
    Possible solutions
    Introduce variables or constants
    38 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  40. Testing smell
    Mystery guest
    Set up or veri cation relies on information that's not visible in the test
    Filename
    Database record
    General xture
    39 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  41. Mystery Guest
    Filename
    public function testBundleImport()
    {
    // import data from CSV file
    $pathToFile = __DIR__ . '/../../_files/import_bundle.csv';
    // ...
    }
    40 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  42. Mystery Guest
    Database record / General Fixture
    /**
    * @magentoAppArea adminhtml
    * @magentoDataFixture Magento/Reports/_files/viewed_products.php
    */
    public function testExecute()
    {
    $this->dispatch('backend/admin/dashboard/productsViewed/');
    $this->assertEquals(200, $this->getResponse()->getHttpResponseCode());
    $actual = $this->getResponse()->getBody();
    $this->assertContains('Simple Product', $actual);
    }
    41 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  43. Problem
    Hard to understand cause and effect
    Risk of somebody else editing external source without knowing how it's used
    Possible solutions
    Fresh xture for each test
    Test data builders instead of xture scripts
    Create les in test
    For shared xture: Accurately named nder methods
    42 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  44. Shared Fixture
    Using attribute
    $this->assertTrue($this->product->isNew(), 'Product should be new');
    Using nder method
    $this->assertTrue(
    $this->getRecentlyAddedProduct()->isNew(),
    'Product should be new'
    );
    private function getRecentlyAddedProduct()
    {
    return $this->product;
    }
    43 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  45. Testing smell
    Eager test
    $savedStockItem->setQty(null);
    $savedStockItem->save();
    $savedStockItem->setQty(2);
    $savedStockItem->save();
    $this->assertEquals('2.0000', $savedStockItem->load($savedStockItemId)->getQty());
    $savedStockItem->setQty(0);
    $savedStockItem->save();
    $this->assertEquals('0.0000', $savedStockItem->load($savedStockItemId)->getQty());
    $savedStockItem->setQty(null);
    $savedStockItem->save();
    $this->assertEquals(null, $savedStockItem->load($savedStockItemId)->getQty());
    44 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  46. Problem
    Hard to tell apart setup, exercise and veri cation
    Harder to localize defects
    Possible solutions
    Single condition tests
    45 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  47. Testing pattern
    Given...When...Then...
    $this->given_stock_item_with_qty(2);
    $this->when_stock_item_is_changed_to(0);
    $this->then_loaded_stock_item_qty_should_be('0.0000');
    Readable high level test methods
    Clear phases: setup (given), exercise (when) and veri cation (then)
    Convert variables ($savedStockItem) to properties
    Additional methods can also start with and_
    46 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  48. Testing smell
    Excessive Mocking
    $baseCurrency->expects()->method('getCode')->willReturn('USD');
    $baseCurrency->expects()->method('getCurrencySymbol')->willReturn('$');
    $baseCurrency->expects()->method('getRate')->with('AUD')->willReturn('0.80');
    $store->expects()->method('getBaseCurrency')->willReturn($baseCurrency);
    $store->expects()->method('getDefaultCurrency')->willReturn($baseCurrency);
    $store->expects()->method('getAvailableCurrencyCodes')->willReturn(['AUD']);
    $this->storeManager->expects()->method('getStore')->willReturn($store);
    47 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  49. Problem
    Coupling test to implementation
    Test is hard to read
    Possible solutions
    Use real domain objects where possible
    Refactor production code
    Use custom stub or fake objects with simpler setup
    48 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  50. Testing pattern
    Fake object
    Test speci c implementation without external dependencies
    49 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  51. Fake object
    interface CustomerSession
    {
    public function isLoggedIn() : bool;
    }
    class FakeSession implements CustomerSession
    {
    private $loggedIn = false;
    public function isLoggedIn() : bool
    {
    return $this->loggedIn;
    }
    // test methods:
    public function login()
    {
    $this->loggedIn = true;
    }
    public function logout()
    {
    $this->loggedIn = false;
    }
    }
    50 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  52. Fake object
    PHPUnit stub:
    $session = $this->createMock(CustomerSession::class);
    $session->method('isLoggedIn')->willReturn(true);
    Custom fake:
    $session = new FakeSession();
    $session->login();
    more succinct
    reusable accross tests
    51 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  53. Testing Smell
    Meaningless test names
    public function testProcessValidStoreCodeCase1()
    {
    // ...
    }
    public function testProcessValidStoreCodeCase2()
    {
    // ...
    }
    public function testProcessValidStoreCodeCase3()
    {
    // ...
    }
    52 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  54. Problem
    Failures in test results are unclear
    Possible solutions
    Write meaningful test names
    Use named keys in data provider arrays
    return [
    'meaningful description of case 1' => [ ... ],
    'meaningful description of case 2' => [ ... ],
    ]
    53 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  55. Testing Smell
    Meaningless assertion messages
    Failed asserting that 0 is 1.
    Failed asserting that false is true.
    54 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  56. Problem
    Failures in test results are unclear
    Possible solutions
    use the $message parameter in assert methods to add custom message
    55 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  57. Testing smell
    Conditional test logic
    public function testSend($configValue, $forceSyncMode, $isComment, $sendingResult)
    {
    // ...
    if (!$isComment) {
    // ...
    }
    // ...
    if (!$configValue || $forceSyncMode) {
    // ...
    if ($sendingResult) {
    // ...
    } else {
    // ...
    }
    } else {
    // ...
    }
    }
    56 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  58. Problem
    Harder to understand the test
    Harder to write the test correctly
    Possible solutions
    Write separate test cases, extract duplicated logic
    Make sure, tests are deterministic
    57 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  59. More smells
    Test logic in production
    General xture
    Using resources (DB, HTTP) in unit tests
    Indirect testing
    "Inappropiate intimacy"
    58 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  60. Conclusion
    Maintainability and stability of tests can be greatly improved by taking care of
    the test architecture.
    Watch out for these typical problems. Get rid of them.
    59 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  61. Image sources
    Head in sand: tropical.pete @Flickr CC-BY-SA 2.0
    60 / 60
    #MM18PL
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide