Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dealing with Testing Fatigue

Dealing with Testing Fatigue

Slides from MageTestFest

Fabian Schmengler

November 17, 2017
Tweet

More Decks by Fabian Schmengler

Other Decks in Programming

Transcript

  1. Dealing With Testing
    Fatigue
    MAGE
    TEST
    FEST
    Fabian Schmengler

    View full-size slide

  2. Fabian Schmengler
    Developer and partner at integer_net
    Testing Magento since 2011
    Magento Master 2017
    @fschmengler

    View full-size slide

  3. Did you know?
    3 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  4. What I'm hearing a lot:
    “ I'd really like to write more tests but I don't get any
    practical value from them. It takes too much time, or
    it is too hard. Therefore I don't enjoy it or stopped
    doing it.
    (I totally understand that!)
    4 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  5. Another popular opinion:
    ‘ Integration tests are superior to unit tests. Do not
    waste your time with unit tests.
    (at least they are writing tests!)
    5 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  6. 6 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  7. Spoiler: A healthy mix of tests is best
    Things are mostly ne
    07:32 - 21. Mai 2016
    36 5.588 5.472
    The Practical Dev
    @ThePracticalDev
    7 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  8. Why all the
    frustration?
    8 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  9. http://xunitpatterns.com/
    "Test smells"
    Project smells
    Behavior smells
    Code smells
    9 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  10. Why all the frustration?
    The project smells
    Buggy tests
    High test maintenance cost
    Production bugs
    Developers not writing tests
    10 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  11. Why all the frustration?
    The behavior smells
    Fragile Test
    Erratic Test
    Slow Test
    Frequent Debugging
    Manual Intervention
    11 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  12. I've been through it, too...
    12 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  13. ...but did I give up?
    13 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  14. ...but did I give up?
    NOPE
    There's a way out!
    13 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  15. (It's not easy!)
    14 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  16. First and foremost:
    Learn!
    Testing is a different skill than coding
    15 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  17. Test the right things!
    No, "everything" is not the answer
    16 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  18. Write testable code!
    If code is hard to test, make it easy to
    test
    17 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  19. Write testable code
    Dependency
    Injection
    18 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  20. Write testable code
    Dependency
    Inversion
    19 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  21. Write testable code
    Ports & Adapters
    20 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  22. 21 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  23. 22 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  24. Improve test
    architecture
    Know your (anti-)patterns
    23 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  25. Test phases
    1. Setup
    set up xture (instantiate objects)
    2. Exercise
    interact with subject under test (SUT)
    3. Verify
    assertions
    4. Teardown
    clean up xture
    24 / 62

    View full-size slide

  26. Fixture setup strategy
    Prebuilt xture
    Shared xture
    Fresh xture
    25 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  27. Fixture setup strategy
    Prebuilt xture
    e.g. Magento integration test database
    Shared xture
    Fresh xture
    26 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  28. Fixture setup strategy
    Prebuilt xture
    Shared xture
    shared by whole test suite
    e.g. Magento integration test object manager,
    Magento unit test code generation
    shared by test case
    setUpBeforeClass(), or lazy setup
    Fresh xture
    27 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  29. Fixture setup strategy
    Prebuilt xture
    Shared xture
    Fresh xture
    Inline in test method
    Implicitly in setUp()
    28 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  30. Testing smell
    Copy and paste
    Duplicated logic in test code
    dev/tests/integration/testsuite/Magento/Customer/_files/customer.php
    dev/tests/integration/testsuite/Magento/Customer/_files/two_customers.php
    dev/tests/integration/testsuite/Magento/Customer/_files/three_customers.php
    dev/tests/integration/testsuite/Magento/Customer/_files/twenty_one_customers.php
    29 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  31. Problem
    increasing cost of changes
    Possible solutions
    Extract method refactoring
    In setup:
    Creation method for fresh xture
    Finder method for shared xture
    In veri cation:
    Custom assertion method
    30 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  32. Problem
    increasing cost of changes
    Possible solutions
    Extract class refactoring
    In setup:
    Test data builder
    In veri cation:
    Custom constraint
    31 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  33. Testing pattern
    Constraint
    class IsMeaningOfLife extends \PHPUnit\Framework\Constraint
    {
    protected function matches($other)
    {
    return $other === 42;
    }
    protected function failureDescription($other)
    {
    return sprintf(
    "%s is not the meaning of life, the universe and everything", $other
    );
    }
    }
    32 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  34. Testing pattern
    Constraint
    Usage
    $this->assertThat(42, new IsMeaningOfLife());
    33 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  35. Testing pattern
    Test data builder
    Creates xture, reusable across tests
    34 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  36. Test data builder
    Example
    CustomerBuilder::aCustomer()
    ->withEmail('[email protected]')
    ->withCustomAttributes(
    [
    'my_custom_attribute' => 42
    ]
    )
    ->build()
    https://github.com/tddwizard/magento2- xtures
    35 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  37. Testing smell
    Irrelevant information
    36 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  38. $customer->setWebsiteId(
    1
    )->setId(
    1
    )->setEntityTypeId(
    1
    )->setAttributeSetId(
    1
    )->setEmail(
    '[email protected]'
    )->setPassword(
    'password'
    )->setGroupId(
    1
    )->setStoreId(
    1
    )->setIsActive(
    1
    )->setFirstname(
    'Firstname'
    )->setLastname(
    'Lastname'
    )->setDefaultBilling(
    1
    )->setDefaultShipping(
    1
    )->setRpToken(
    '8ed8677e6c79e68b94e61658bd756ea5'
    )->setRpTokenCreatedAt(
    date('Y-m-d H:i:s')
    );
    37 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  39. Problem
    Hard to read test as documentation
    Possible solutions
    Extract creation methods or assertions with relevant
    parameters
    For xtures: Test data builder
    38 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  40. Testing Smell
    Hard-coded test data
    public function testGetTranslationFileTimestamp()
    {
    $this->fileManagerMock->expects($this->once())
    ->method('getTranslationFileTimestamp')
    ->willReturn(1445736974);
    $this->assertEquals(1445736974, $this->model->getTranslationFileTimestamp());
    }
    39 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  41. Problem
    Hard to understand cause and effect
    Possible solutions
    Introduce variables or constants
    40 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  42. Testing smell
    Mystery guest
    Set up or veri cation relies on information that's not
    visible in the test
    Filename
    Database record
    General xture
    41 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  43. Mystery Guest
    Filename
    public function testBundleImport()
    {
    // import data from CSV file
    $pathToFile = __DIR__ . '/../../_files/import_bundle.csv';
    // ...
    }
    42 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  44. Mystery Guest
    Database record
    /**
    * @magentoAppArea adminhtml
    * @magentoDataFixture Magento/Reports/_files/viewed_products.php
    */
    public function testExecute()
    {
    $this->dispatch('backend/admin/dashboard/productsViewed/');
    $this->assertEquals(200, $this->getResponse()->getHttpResponseCode());
    $actual = $this->getResponse()->getBody();
    $this->assertContains('Simple Product', $actual);
    }
    43 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  45. Problem
    Hard to understand cause and effect
    Risk of somebody else editing external source without
    knowing how it's used
    Possible solutions
    Fresh xture for each test
    Test data builders instead of xture scripts
    Create les in test
    Accurately named nder methods
    44 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  46. General Fixture
    Using attribute
    $this->assertTrue($this->product->isNew(), 'Product should be new');
    Using nder method
    $this->assertTrue(
    $this->getRecentlyAddedProduct()->isNew(),
    'Product should be new'
    );
    private function getRecentlyAddedProduct()
    {
    return $this->product;
    }
    45 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  47. Testing smell
    Eager test
    $savedStockItem->setQty(null);
    $savedStockItem->save();
    $savedStockItem->setQty(2);
    $savedStockItem->save();
    $this->assertEquals('2.0000', $savedStockItem->load($savedStockItemId)->getQty());
    $savedStockItem->setQty(0);
    $savedStockItem->save();
    $this->assertEquals('0.0000', $savedStockItem->load($savedStockItemId)->getQty());
    $savedStockItem->setQty(null);
    $savedStockItem->save();
    $this->assertEquals(null, $savedStockItem->load($savedStockItemId)->getQty());
    46 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  48. Problem
    Hard to tell apart set up, exercise and veri cation
    Harder to localize defects
    Possible solutions
    Single condition tests
    Create one test case (class) per case, not per subject
    47 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  49. Testing smell
    Excessive Mocking
    $baseCurrency->expects()->method('getCode')->willReturn('USD');
    $baseCurrency->expects()->method('getCurrencySymbol')->willReturn('$');
    $baseCurrency->expects()->method('getRate')->with('AUD')->willReturn('0.80');
    $store->expects()->method('getBaseCurrency')->willReturn($baseCurrency);
    $store->expects()->method('getDefaultCurrency')->willReturn($baseCurrency);
    $store->expects()->method('getAvailableCurrencyCodes')->willReturn(['AUD']);
    $this->storeManager->expects()->method('getStore')->willReturn($store);
    48 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  50. Problem
    Coupling test to implementation
    Test is hard to read
    Possible solutions
    Use real domain objects where possible
    Refactor production code
    Use custom stub or fake objects with simpler setup
    49 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  51. Testing pattern
    Fake object
    Test speci c implementation without external
    dependencies
    50 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  52. Fake object
    interface CustomerSession
    {
    public function isLoggedIn() : bool;
    }
    class FakeSession implements CustomerSession
    {
    private $loggedIn = false;
    public function isLoggedIn() : bool
    {
    return $this->loggedIn;
    }
    // test methods:
    public function login()
    {
    $this->loggedIn = true;
    }
    public function logout()
    {
    $this->loggedIn = false;
    }
    }
    51 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  53. Fake object
    PHPUnit stub:
    $session = $this->createMock(CustomerSession::class);
    $session->method('isLoggedIn')->willReturn(true);
    Custom fake:
    $session = new FakeSession();
    $session->login();
    more succinct
    reusable accross tests
    52 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  54. Testing Smell
    Meaningless test names
    public function testProcessValidStoreCodeCase1()
    {
    // ...
    }
    public function testProcessValidStoreCodeCase2()
    {
    // ...
    }
    public function testProcessValidStoreCodeCase3()
    {
    // ...
    }
    53 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  55. Problem
    Failures in test results are unclear
    Possible solutions
    Write meaningful test names
    Use named keys in data provider arrays
    return [
    'meaningful description of case 1' => [ ... ],
    'meaningful description of case 2' => [ ... ],
    ]
    54 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  56. Testing Smell
    Meaningless assertion
    messages
    Failed asserting that 0 is 1.
    Failed asserting that false is true.
    55 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  57. Problem
    Failures in test results are unclear
    Possible solutions
    use the $message parameter in assert methods to add
    custom message
    56 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  58. Testing smell
    Conditional test logic
    public function testSend($configValue, $forceSyncMode, $isComment, $sendingResult)
    {
    // ...
    if (!$isComment) {
    // ...
    }
    // ...
    if (!$configValue || $forceSyncMode) {
    // ...
    if ($sendingResult) {
    // ...
    } else {
    // ...
    }
    } else {
    // ...
    }
    }
    57 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  59. Problem
    Harder to understand the test
    Harder to write the test correctly
    Possible solutions
    Write separate test cases, extract duplicated logic
    Make sure, tests are deterministic
    58 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  60. More smells
    Test logic in production
    General xture
    Using resources (DB, HTTP) in unit tests
    Indirect testing
    "Inappropiate intimacy"
    59 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  61. Conclusion
    Maintainability and stability of tests can be greatly
    improved by taking care of the test architecture.
    Watch out for these typical problems. Get rid of them.
    MAKE
    TESTING
    FUN
    60 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  62. Conclusion
    Maintainability and stability of tests can be greatly
    improved by taking care of the test architecture.
    Watch out for these typical problems. Get rid of them.
    MAKE
    TESTING
    FUN
    AGAIN
    60 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  63. THANK YOU
    61 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  64. THANK YOU
    Coming soon:
    Test Driven Magento by
    Example
    http://tddwizard.com/
    @fschmengler
    61 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide

  65. Image sources
    Head in sand: tropical.pete @Flickr CC-BY-SA 2.0
    62 / 62
    Fabian Schmengler /> Dealing With Testing Fatigue @fschmengler

    View full-size slide