$30 off During Our Annual Pro Sale. View Details »

A smörgåsbord of testing

A smörgåsbord of testing

The controversy over writing tests and TDD is over. The benefits to the design, development and maintenance of your code are real and well known. In 2013, it is simply unprofessional to eschew tests for production code.

Given this, developers need to move beyond the basics of testing and gain a deeper understanding of the art. This entails knowing the patterns and anti-patterns of effective test suites, understanding what to test and how to do it, and a mastery of the many fantastic testing tools in the Python ecosystem.

Bad tests are a genuine menace. A poorly considered test suite pours concrete over your codebase, hindering maintenance and development.

This talk isn't an introduction to testing and TDD. Rather, it is a mixed buffet of tips, anti-patterns, tools and techniques for effective testing.

David Winterbottom

September 20, 2013
Tweet

More Decks by David Winterbottom

Other Decks in Programming

Transcript

  1. smörgåsbord
    testing
    of
    A

    View Slide

  2. David Winterbottom
    @codeinthehole
    /codeinthehole
    Impending sausage

    View Slide

  3. David Winterbottom
    @codeinthehole
    /codeinthehole
    Grange Hill (only British people who grew up in the 1990s will understand this)

    View Slide

  4. Tangent Labs
    @tangent_labs
    /tangentlabs
    See www.tangentlabs.co.uk (PS - we’re hiring!)

    View Slide

  5. django-oscar
    @django_oscar
    /tangentlabs/django-oscar
    Oscar is the most popular python e-commerce software on Github. It’s code is so beautiful,
    it’s been known to make grown men weep.
    oscarcommerce.com

    View Slide

  6. This is Oscar’s test suite - we can see a few things which betray the biases of this talk

    View Slide

  7. Nose
    Nose
    nose.readthedocs.org
    ...like nose...

    View Slide

  8. Django
    ...and Django.

    View Slide

  9. Dexterity
    Readability
    Speed
    The talk covers 3 topics

    View Slide

  10. Who is this? (Hint - it’s not Philip Seymour Hoffman).
    It’s Kent Beck: creator of jUnit, XP etc

    View Slide

  11. “I get paid for code that
    works, not for tests, so
    my philosophy is to test
    as little as possible to
    reach a given level of
    confidence ”
    ...and he’s a pragmatist. Beware of dogma (100% coverage, one assertion per method etc)

    View Slide

  12. Bad tests exist. They can pour concrete over your codebase.
    It’s ok to delete tests.

    View Slide

  13. Dexterity
    Is there a clever Japanese word for the art of slicing?
    Dexterity = How to use your test suite effectively

    View Slide

  14. Basics
    Love thy test-runner

    View Slide

  15. Run ALL the tests

    View Slide

  16. Run tests beneath path
    Can use your shell’s tab completion for speedy folder selection

    View Slide

  17. Run test module

    View Slide

  18. Run test class
    Colons are a pain (if you are trying to write a autocompleter) as they are word delimiters

    View Slide

  19. Run test method
    This is too labour-intensive to type, so we use ...

    View Slide

  20. nosecomplete
    github.com/alonho/nosecomplete
    Someone solved the colon problem!

    View Slide

  21. nosecomplete
    github.com/alonho/nosecomplete
    ...and here it is in action.

    View Slide

  22. Stories

    View Slide

  23. Stories
    Work on a feature
    Doing TDD, working from a spec etc

    View Slide

  24. Here’s a sample test run

    View Slide

  25. https://github.com/asdfasdf/spec
    Custom test runner
    You need a custom test-runner

    View Slide

  26. https://github.com/asdfasdf/spec
    Testing one module
    Run your unit tests as you develop

    View Slide

  27. https://github.com/asdfasdf/spec
    Spec plugin
    https://github.com/bitprophet/spec
    Write your test classes and methods as a spec
    Based on Pinocchio

    View Slide

  28. https://github.com/asdfasdf/spec
    BDD-style output
    And, write your specs to look like this BDD-style output.

    View Slide

  29. class TestASignedInUser(WebTestCase):
    def setUp(self):
    self.user = G(User)
    self.order = factories.create_order(user=self.user)
    def test_can_see_their_email_address_on_the_profile_page(self):
    profile_page = self.app.get(
    reverse('customer:summary'), user=self.user)
    self.assertTrue(self.email in profile_page.content)
    def test_can_update_their_name(self):
    profile_form_page = self.app.get(
    reverse('customer:profile-update'), user=self.user)
    form = profile_form_page.forms['profile_form']
    form['first_name'] = 'Barry'
    form['last_name'] = 'Chuckle'
    response = form.submit()
    self.assertRedirects(response, reverse('customer:summary'))
    # Reload user
    user = User.objects.get(id=self.user.id)
    self.assertEquals("Barry", user.first_name)
    self.assertEquals("Chuckle", user.last_name)
    The Chuckle Brothers are updating their profiles

    View Slide

  30. class TestASignedInUser(WebTestCase):
    def setUp(self):
    self.user = G(User)
    self.order = factoies.create_order(user=self.user)
    def test_can_see_their_email_address_on_the_profile_page(self):
    profile_page = self.app.get(
    reverse('customer:summary'), user=self.user)
    self.assertTrue(self.email in profile_page.content)
    def test_can_update_their_name(self):
    profile_form_page = self.app.get(
    reverse('customer:profile-update'), user=self.user)
    form = profile_form_page.forms['profile_form']
    form['first_name'] = 'Barry'
    form['last_name'] = 'Chuckle'
    response = form.submit()
    self.assertRedirects(response, reverse('customer:summary'))
    # Reload user
    user = User.objects.get(id=self.user.id)
    self.assertEquals("Barry", user.first_name)
    self.assertEquals("Chuckle", user.last_name)
    System under test

    View Slide

  31. class TestASignedInUser(WebTestCase):
    def setUp(self):
    self.user = G(User)
    self.order = factories.create_order(user=self.user)
    def test_can_see_their_email_address_on_the_profile_page(self):
    profile_page = self.app.get(
    reverse('customer:summary'), user=self.user)
    self.assertTrue(self.email in profile_page.content)
    def test_can_update_their_name(self):
    profile_form_page = self.app.get(
    reverse('customer:profile-update'), user=self.user)
    form = profile_form_page.forms['profile_form']
    form['first_name'] = 'Barry'
    form['last_name'] = 'Chuckle'
    response = form.submit()
    self.assertRedirects(response, reverse('customer:summary'))
    # Reload user
    user = User.objects.get(id=self.user.id)
    self.assertEquals("Barry", user.first_name)
    self.assertEquals("Chuckle", user.last_name)
    Specification
    On the road towards a full BDD framwork like Lettuce/Behave, but you can still use Nose and
    all its plugins

    View Slide

  32. Use-cases
    Work on a new feature
    Refactor a feature
    You’re reviewing a pull request

    View Slide

  33. https://github.com/nose attribs/spec
    Run all the shipping tests!

    View Slide

  34. nose.readthedocs.org/en/latest/plugins/attrib.html
    Run tagged tests

    View Slide

  35. from nose.plugins.attrib import attr
    @attr('shipping')
    class TestFreeShippping(TestCase):
    def test_is_free_for_empty_basket(self):
    ...
    class TestFixedPriceShipping(TestCase):
    @attr(slow=True)
    def test_returns_fixed_price_for_basket(self):
    ...

    View Slide

  36. from nose.plugins.attrib import attr
    @attr('shipping')
    class TestFreeShippping(TestCase):
    def test_is_free_for_empty_basket(self):
    ...
    class TestFixedPriceShipping(TestCase):
    @attr(slow=True)
    def test_returns_fixed_price_for_basket(self):
    ...
    Wrap a class

    View Slide

  37. from nose.plugins.attrib import attr
    @attr('shipping')
    class TestFreeShippping(TestCase):
    def test_is_free_for_empty_basket(self):
    ...
    class TestFixedPriceShipping(TestCase):
    @attr(slow=True)
    def test_returns_fixed_price_for_basket(self):
    ...
    Wrap a method
    You can tag slow tests

    View Slide

  38. View Slide

  39. Pattern match tests
    github.com/iElectric/nose-selecttests
    Better for lazy developers

    View Slide

  40. Use-cases
    Work on a new feature
    Refactor a feature
    Fix failing tests

    View Slide

  41. Fail fast

    View Slide

  42. Editor-friendly paths
    github.com/erikrose/nose-progressive
    Other nice features: prints out failures/errors as they occur, progress bar

    View Slide

  43. PDB on failure
    Need to go up the stack to get to test code

    View Slide

  44. Only re-run failed tests
    Needs to be run twice though

    View Slide

  45. Use-cases
    Work on a new feature
    Refactor a feature
    Fix failing tests
    Commit
    Before pushing etc

    View Slide

  46. Run all the tests as fast as possible

    View Slide

  47. No plugins

    View Slide

  48. Parallelise
    if you use tox, then try detox

    View Slide

  49. Speed
    Watch Gary Bernhardt’s talks

    View Slide

  50. 0 4 8 12 16 20 24
    Focus / flow / “zone”
    Test suite time (seconds)

    View Slide

  51. 0 4 8 12 16 20 24
    Focus / flow / “zone”
    Test suite time (seconds)
    Pre-commit hook?
    Everything’s rosy when your test suite runs in 100ms

    View Slide

  52. 0 4 8 12 16 20 24
    Focus / flow / “zone”
    Test suite time (seconds)
    But watch out for the threshold of doom

    View Slide

  53. 0 4 8 12 16 20 24
    Test suite time (seconds)
    Oh no - you’re reading Hacker News!

    View Slide

  54. Tips for speeding up tests will be in another talk
    Oscar distinguishes between unit, integration and functional (acceptance) tests to try and
    steer people towards unit tests

    View Slide

  55. View Slide

  56. Scam?
    blog.thecodewhisperer.com/2010/10/16/integrated-tests-are-a-scam/
    Read this polemic

    View Slide

  57. import mock
    from django.utils.functional import curry
    no_database = curry(
    mock.patch, 'django.db.backends.util.CursorWrapper',
    Mock(side_effect=RuntimeError(
    "Using the database is not permitted")))
    @no_database()
    class AUnitTest(TestCase):
    ...
    http://bit.ly/126vPrt

    View Slide

  58. import mock
    from django.utils.functional import curry
    no_database = curry(
    mock.patch, 'django.db.backends.util.CursorWrapper',
    Mock(side_effect=RuntimeError(
    "Using the database is not permitted")))
    @no_database()
    class AUnitTest(TestCase):
    ...
    http://bit.ly/126vPrt
    Currying mock.patch!
    All my favourite bits of Python in one curio

    View Slide

  59. import mock
    from django.utils.functional import curry
    no_database = curry(
    mock.patch, 'django.db.backends.util.CursorWrapper',
    Mock(side_effect=RuntimeError(
    "Using the database is not permitted")))
    @no_database()
    class AUnitTest(TestCase):
    ...
    http://bit.ly/126vPrt
    Wrap classes/methods

    View Slide

  60. http://bit.ly/156c48x
    Number of tests
    Seconds
    Oscar’s test suite has doubled number of tests while halving the time to run.

    View Slide

  61. PASSWORD_HASHERS =
    ['django.contrib.auth.hashers.MD5PasswordHasher']
    Using this hasher makes a big difference.

    View Slide

  62. --processes=8
    Maik spotted this option for Nose and we all had a cup of tea to celebrate.

    View Slide

  63. http://bit.ly/2gZnhj
    Boy scout rule...
    Always leave the campsite cleaner than how you found it

    View Slide

  64. Readability

    View Slide

  65. http://bit.ly/19ganRJ (Refactoring a Test)
    Book has a great chapter on refactoring a test from dirty to clean

    View Slide

  66. class TestBasket(TestCase):
    def test_add_items(self):
    # Create products
    product_class = ProductClass(
    name="Books", require_shipping=True)
    product1 = Product.objects.create(title="My first book")
    product2 = Product.objects.create(title="My second book")
    partner = Partner.objects.create(name="Book Partner")
    stockrecord1 = StockRecord(
    price_excl_tax=D('12.00'), partner=partner)
    stockrecord2 = StockRecord(
    price_excl_tax=D('14.00'), partner=partner)
    # Add to basket
    basket = Basket()
    basket.add(product1)
    basket.add(product2)
    total = basket.total_excl_tax
    self.assertEqual(D('26.00'), total)
    self.assertEqual(basket.num_lines, 2)
    self.assertFalse(basket.is_empty)
    This is a bad test

    View Slide

  67. class TestBasket(TestCase):
    def test_add_items(self):
    # Create products
    product_class = ProductClass(
    name="Books", require_shipping=True)
    product1 = Product.objects.create(title="My first book")
    product2 = Product.objects.create(title="My second book")
    partner = Partner.objects.create(name="Book Partner")
    stockrecord1 = StockRecord(
    price_excl_tax=D('12.00'), partner=partner)
    stockrecord2 = StockRecord(
    price_excl_tax=D('14.00'), partner=partner)
    # Add to basket
    basket = Basket()
    basket.add(product1)
    basket.add(product2)
    total = basket.total_excl_tax
    self.assertEqual(D('26.00'), total)
    self.assertEqual(basket.num_lines, 2)
    self.assertFalse(basket.is_empty)
    Bad naming

    View Slide

  68. class TestBasket(TestCase):
    def test_add_items(self):
    # Create products
    product_class = ProductClass(
    name="Books", require_shipping=True)
    product1 = Product.objects.create(title="My first book")
    product2 = Product.objects.create(title="My second book")
    partner = Partner.objects.create(name="Book Partner")
    stockrecord1 = StockRecord(
    price_excl_tax=D('12.00'), partner=partner)
    stockrecord2 = StockRecord(
    price_excl_tax=D('14.00'), partner=partner)
    # Add to basket
    basket = Basket()
    basket.add(product1)
    basket.add(product2)
    total = basket.total_excl_tax
    self.assertEqual(D('26.00'), total)
    self.assertEqual(basket.num_lines, 2)
    self.assertFalse(basket.is_empty)
    Heavy set-up

    View Slide

  69. class TestBasket(TestCase):
    def test_add_items(self):
    # Create products
    product_class = ProductClass(
    name="Books", require_shipping=True)
    product1 = Product.objects.create(title="My first book")
    product2 = Product.objects.create(title="My second book")
    partner = Partner.objects.create(name="Book Partner")
    stockrecord1 = StockRecord(
    price_excl_tax=D('12.00'), partner=partner)
    stockrecord2 = StockRecord(
    price_excl_tax=D('14.00'), partner=partner)
    # Add to basket
    basket = Basket()
    basket.add(product1)
    basket.add(product2)
    total = basket.total_excl_tax
    self.assertEqual(D('26.00'), total)
    self.assertEqual(basket.num_lines, 2)
    self.assertFalse(basket.is_empty)
    Noise

    View Slide

  70. class TestBasket(TestCase):
    def test_add_items(self):
    # Create products
    product_class = ProductClass(
    name="Books", require_shipping=True)
    product1 = Product.objects.create(title="My first book")
    product2 = Product.objects.create(title="My second book")
    partner = Partner.objects.create(name="Book Partner")
    stockrecord1 = StockRecord(
    price_excl_tax=D('12.00'), partner=partner)
    stockrecord2 = StockRecord(
    price_excl_tax=D('14.00'), partner=partner)
    # Add to basket
    basket = Basket()
    basket.add(product1)
    basket.add(product2)
    total = basket.total_excl_tax
    self.assertEqual(D('26.00'), total)
    self.assertEqual(basket.num_lines, 2)
    self.assertFalse(basket.is_empty)
    Unclear assertions

    View Slide

  71. class TestBasket(TestCase):
    def test_add_items(self):
    # Create products
    product_class = ProductClass(
    name="Books", require_shipping=True)
    product1 = Product.objects.create(title="My first book")
    product2 = Product.objects.create(title="My second book")
    partner = Partner.objects.create(name="Book Partner")
    stockrecord1 = StockRecord(
    price_excl_tax=D('12.00'), partner=partner)
    stockrecord2 = StockRecord(
    price_excl_tax=D('14.00'), partner=partner)
    # Add to basket
    basket = Basket()
    basket.add(product1)
    basket.add(product2)
    total = basket.total_excl_tax
    self.assertEqual(D('26.00'), total)
    self.assertEqual(basket.num_lines, 2)
    self.assertFalse(basket.is_empty)
    Sloppy coding
    Numbered variable names make baby jesus cry

    View Slide

  72. class TestBasketTotal(TestCase):
    def setUp(self):
    self.basket = Basket()
    def test_is_correct_after_adding_multiple_items(self):
    products = [
    factory.create_product(price_excl_tax=D('12.00')),
    factory.create_product(price_excl_tax=D('14.00')),
    ]
    for product in products:
    self.basket.add_product(product)
    self.assertEqual(
    D('12.00') + D('14.00'), self.basket.total_excl_tax)
    This is the same test, but cleaned up

    View Slide

  73. class TestBasketTotal(TestCase):
    def setUp(self):
    self.basket = Basket()
    def test_is_correct_after_adding_multiple_items(self):
    products = [
    factory.create_product(price_excl_tax=D('12.00')),
    factory.create_product(price_excl_tax=D('14.00')),
    ]
    for product in products:
    self.basket.add_product(product)
    self.assertEqual(
    D('12.00') + D('14.00'), self.basket.total_excl_tax)
    Descriptive names

    View Slide

  74. class TestBasketTotal(TestCase):
    def setUp(self):
    self.basket = Basket()
    def test_is_correct_after_adding_multiple_items(self):
    products = [
    factory.create_product(price_excl_tax=D('12.00')),
    factory.create_product(price_excl_tax=D('14.00')),
    ]
    for product in products:
    self.basket.add_product(product)
    self.assertEqual(
    D('12.00') + D('14.00'), self.basket.total_excl_tax)
    Factory functions
    Tests should only show relevant details

    View Slide

  75. class TestBasketTotal(TestCase):
    def setUp(self):
    self.basket = Basket()
    def test_is_correct_after_adding_multiple_items(self):
    products = [
    factory.create_product(price_excl_tax=D('12.00')),
    factory.create_product(price_excl_tax=D('14.00')),
    ]
    for product in products:
    self.basket.add_product(product)
    self.assertEqual(
    D('12.00') + D('14.00'), self.basket.total_excl_tax)
    Intention revealing
    Dumb example but a good thing to do in general

    View Slide

  76. PyHamcrest
    github.com/hamcrest/PyHamcrest
    >>> from hamcrest import *
    >>> assert_that(
    shipping.charge, is_(equal_to(D('4.99'))))
    >>> assert_that(
    "RESTful".lower(), contains_string("stfu"))
    Hamcrest - more readable but too many parentheses?

    View Slide

  77. should-dsl
    github.com/nsi-iff/should-dsl
    >>> from should_dsl import should, should_not
    >>> {} |should| be_empty
    >>> user.email |should| contain("@")
    >>> basket |should| contain(product)
    Crazy voodoo here, but quite readable ultimately

    View Slide

  78. This was the best image of a Smorgasbord I could find

    View Slide

  79. wiki.python.org/moin/PythonTestingToolsTaxonomy
    For all your python testing needs.

    View Slide