A smörgåsbord of testing

The controversy over writing tests and TDD is over. The benefits to the design, development and maintenance of your code are real and well known. In 2013, it is simply unprofessional to eschew tests for production code.

Given this, developers need to move beyond the basics of testing and gain a deeper understanding of the art. This entails knowing the patterns and anti-patterns of effective test suites, understanding what to test and how to do it, and a mastery of the many fantastic testing tools in the Python ecosystem.

Bad tests are a genuine menace. A poorly considered test suite pours concrete over your codebase, hindering maintenance and development.

This talk isn't an introduction to testing and TDD. Rather, it is a mixed buffet of tips, anti-patterns, tools and techniques for effective testing.

Transcript

1. smörgåsbord
   testing
   of
   A
   

   View Slide

2. David Winterbottom
   @codeinthehole
   /codeinthehole
   Impending sausage
   

   View Slide

3. David Winterbottom
   @codeinthehole
   /codeinthehole
   Grange Hill (only British people who grew up in the 1990s will understand this)
   

   View Slide

4. Tangent Labs
   @tangent_labs
   /tangentlabs
   See www.tangentlabs.co.uk (PS - we’re hiring!)
   

   View Slide

5. django-oscar
   @django_oscar
   /tangentlabs/django-oscar
   Oscar is the most popular python e-commerce software on Github. It’s code is so beautiful,
   it’s been known to make grown men weep.
   oscarcommerce.com
   

   View Slide

6. This is Oscar’s test suite - we can see a few things which betray the biases of this talk
   

   View Slide

7. Nose
   Nose
   nose.readthedocs.org
   ...like nose...
   

   View Slide

8. Django
   ...and Django.
   

   View Slide

9. Dexterity
   Readability
   Speed
   The talk covers 3 topics
   

   View Slide

10. Who is this? (Hint - it’s not Philip Seymour Hoffman).
    It’s Kent Beck: creator of jUnit, XP etc
    

    View Slide

11. “I get paid for code that
    works, not for tests, so
    my philosophy is to test
    as little as possible to
    reach a given level of
    confidence ”
    ...and he’s a pragmatist. Beware of dogma (100% coverage, one assertion per method etc)
    

    View Slide

12. Bad tests exist. They can pour concrete over your codebase.
    It’s ok to delete tests.
    

    View Slide

13. Dexterity
    Is there a clever Japanese word for the art of slicing?
    Dexterity = How to use your test suite effectively
    

    View Slide

14. Basics
    Love thy test-runner
    

    View Slide

15. Run ALL the tests
    

    View Slide

16. Run tests beneath path
    Can use your shell’s tab completion for speedy folder selection
    

    View Slide

17. Run test module
    

    View Slide

18. Run test class
    Colons are a pain (if you are trying to write a autocompleter) as they are word delimiters
    

    View Slide

19. Run test method
    This is too labour-intensive to type, so we use ...
    

    View Slide

20. nosecomplete
    github.com/alonho/nosecomplete
    Someone solved the colon problem!
    

    View Slide

21. nosecomplete
    github.com/alonho/nosecomplete
    ...and here it is in action.
    

    View Slide

22. Stories
    

    View Slide

23. Stories
    Work on a feature
    Doing TDD, working from a spec etc
    

    View Slide

24. Here’s a sample test run
    

    View Slide

25. https://github.com/asdfasdf/spec
    Custom test runner
    You need a custom test-runner
    

    View Slide

26. https://github.com/asdfasdf/spec
    Testing one module
    Run your unit tests as you develop
    

    View Slide

27. https://github.com/asdfasdf/spec
    Spec plugin
    https://github.com/bitprophet/spec
    Write your test classes and methods as a spec
    Based on Pinocchio
    

    View Slide

28. https://github.com/asdfasdf/spec
    BDD-style output
    And, write your specs to look like this BDD-style output.
    

    View Slide

29. class TestASignedInUser(WebTestCase):
    def setUp(self):
    self.user = G(User)
    self.order = factories.create_order(user=self.user)
    def test_can_see_their_email_address_on_the_profile_page(self):
    profile_page = self.app.get(
    reverse('customer:summary'), user=self.user)
    self.assertTrue(self.email in profile_page.content)
    def test_can_update_their_name(self):
    profile_form_page = self.app.get(
    reverse('customer:profile-update'), user=self.user)
    form = profile_form_page.forms['profile_form']
    form['first_name'] = 'Barry'
    form['last_name'] = 'Chuckle'
    response = form.submit()
    self.assertRedirects(response, reverse('customer:summary'))
    # Reload user
    user = User.objects.get(id=self.user.id)
    self.assertEquals("Barry", user.first_name)
    self.assertEquals("Chuckle", user.last_name)
    The Chuckle Brothers are updating their profiles
    

    View Slide

30. class TestASignedInUser(WebTestCase):
    def setUp(self):
    self.user = G(User)
    self.order = factoies.create_order(user=self.user)
    def test_can_see_their_email_address_on_the_profile_page(self):
    profile_page = self.app.get(
    reverse('customer:summary'), user=self.user)
    self.assertTrue(self.email in profile_page.content)
    def test_can_update_their_name(self):
    profile_form_page = self.app.get(
    reverse('customer:profile-update'), user=self.user)
    form = profile_form_page.forms['profile_form']
    form['first_name'] = 'Barry'
    form['last_name'] = 'Chuckle'
    response = form.submit()
    self.assertRedirects(response, reverse('customer:summary'))
    # Reload user
    user = User.objects.get(id=self.user.id)
    self.assertEquals("Barry", user.first_name)
    self.assertEquals("Chuckle", user.last_name)
    System under test
    

    View Slide

31. class TestASignedInUser(WebTestCase):
    def setUp(self):
    self.user = G(User)
    self.order = factories.create_order(user=self.user)
    def test_can_see_their_email_address_on_the_profile_page(self):
    profile_page = self.app.get(
    reverse('customer:summary'), user=self.user)
    self.assertTrue(self.email in profile_page.content)
    def test_can_update_their_name(self):
    profile_form_page = self.app.get(
    reverse('customer:profile-update'), user=self.user)
    form = profile_form_page.forms['profile_form']
    form['first_name'] = 'Barry'
    form['last_name'] = 'Chuckle'
    response = form.submit()
    self.assertRedirects(response, reverse('customer:summary'))
    # Reload user
    user = User.objects.get(id=self.user.id)
    self.assertEquals("Barry", user.first_name)
    self.assertEquals("Chuckle", user.last_name)
    Specification
    On the road towards a full BDD framwork like Lettuce/Behave, but you can still use Nose and
    all its plugins
    

    View Slide

32. Use-cases
    Work on a new feature
    Refactor a feature
    You’re reviewing a pull request
    

    View Slide

33. https://github.com/nose attribs/spec
    Run all the shipping tests!
    

    View Slide

34. nose.readthedocs.org/en/latest/plugins/attrib.html
    Run tagged tests
    

    View Slide

35. from nose.plugins.attrib import attr
    @attr('shipping')
    class TestFreeShippping(TestCase):
    def test_is_free_for_empty_basket(self):
    ...
    class TestFixedPriceShipping(TestCase):
    @attr(slow=True)
    def test_returns_fixed_price_for_basket(self):
    ...
    

    View Slide

36. from nose.plugins.attrib import attr
    @attr('shipping')
    class TestFreeShippping(TestCase):
    def test_is_free_for_empty_basket(self):
    ...
    class TestFixedPriceShipping(TestCase):
    @attr(slow=True)
    def test_returns_fixed_price_for_basket(self):
    ...
    Wrap a class
    

    View Slide

37. from nose.plugins.attrib import attr
    @attr('shipping')
    class TestFreeShippping(TestCase):
    def test_is_free_for_empty_basket(self):
    ...
    class TestFixedPriceShipping(TestCase):
    @attr(slow=True)
    def test_returns_fixed_price_for_basket(self):
    ...
    Wrap a method
    You can tag slow tests
    

    View Slide

38. View Slide

39. Pattern match tests
    github.com/iElectric/nose-selecttests
    Better for lazy developers
    

    View Slide

40. Use-cases
    Work on a new feature
    Refactor a feature
    Fix failing tests
    

    View Slide

41. Fail fast
    

    View Slide

42. Editor-friendly paths
    github.com/erikrose/nose-progressive
    Other nice features: prints out failures/errors as they occur, progress bar
    

    View Slide

43. PDB on failure
    Need to go up the stack to get to test code
    

    View Slide

44. Only re-run failed tests
    Needs to be run twice though
    

    View Slide

45. Use-cases
    Work on a new feature
    Refactor a feature
    Fix failing tests
    Commit
    Before pushing etc
    

    View Slide

46. Run all the tests as fast as possible
    

    View Slide

47. No plugins
    

    View Slide

48. Parallelise
    if you use tox, then try detox
    

    View Slide

49. Speed
    Watch Gary Bernhardt’s talks
    

    View Slide

50. 0 4 8 12 16 20 24
    Focus / flow / “zone”
    Test suite time (seconds)
    

    View Slide

51. 0 4 8 12 16 20 24
    Focus / flow / “zone”
    Test suite time (seconds)
    Pre-commit hook?
    Everything’s rosy when your test suite runs in 100ms
    

    View Slide

52. 0 4 8 12 16 20 24
    Focus / flow / “zone”
    Test suite time (seconds)
    But watch out for the threshold of doom
    

    View Slide

53. 0 4 8 12 16 20 24
    Test suite time (seconds)
    Oh no - you’re reading Hacker News!
    

    View Slide

54. Tips for speeding up tests will be in another talk
    Oscar distinguishes between unit, integration and functional (acceptance) tests to try and
    steer people towards unit tests
    

    View Slide

55. View Slide

56. Scam?
    blog.thecodewhisperer.com/2010/10/16/integrated-tests-are-a-scam/
    Read this polemic
    

    View Slide

57. import mock
    from django.utils.functional import curry
    no_database = curry(
    mock.patch, 'django.db.backends.util.CursorWrapper',
    Mock(side_effect=RuntimeError(
    "Using the database is not permitted")))
    @no_database()
    class AUnitTest(TestCase):
    ...
    http://bit.ly/126vPrt
    

    View Slide

58. import mock
    from django.utils.functional import curry
    no_database = curry(
    mock.patch, 'django.db.backends.util.CursorWrapper',
    Mock(side_effect=RuntimeError(
    "Using the database is not permitted")))
    @no_database()
    class AUnitTest(TestCase):
    ...
    http://bit.ly/126vPrt
    Currying mock.patch!
    All my favourite bits of Python in one curio
    

    View Slide

59. import mock
    from django.utils.functional import curry
    no_database = curry(
    mock.patch, 'django.db.backends.util.CursorWrapper',
    Mock(side_effect=RuntimeError(
    "Using the database is not permitted")))
    @no_database()
    class AUnitTest(TestCase):
    ...
    http://bit.ly/126vPrt
    Wrap classes/methods
    

    View Slide

60. http://bit.ly/156c48x
    Number of tests
    Seconds
    Oscar’s test suite has doubled number of tests while halving the time to run.
    

    View Slide

61. PASSWORD_HASHERS =
    ['django.contrib.auth.hashers.MD5PasswordHasher']
    Using this hasher makes a big difference.
    

    View Slide

62. --processes=8
    Maik spotted this option for Nose and we all had a cup of tea to celebrate.
    

    View Slide

63. http://bit.ly/2gZnhj
    Boy scout rule...
    Always leave the campsite cleaner than how you found it
    

    View Slide

64. Readability
    

    View Slide

65. http://bit.ly/19ganRJ (Refactoring a Test)
    Book has a great chapter on refactoring a test from dirty to clean
    

    View Slide

66. class TestBasket(TestCase):
    def test_add_items(self):
    # Create products
    product_class = ProductClass(
    name="Books", require_shipping=True)
    product1 = Product.objects.create(title="My first book")
    product2 = Product.objects.create(title="My second book")
    partner = Partner.objects.create(name="Book Partner")
    stockrecord1 = StockRecord(
    price_excl_tax=D('12.00'), partner=partner)
    stockrecord2 = StockRecord(
    price_excl_tax=D('14.00'), partner=partner)
    # Add to basket
    basket = Basket()
    basket.add(product1)
    basket.add(product2)
    total = basket.total_excl_tax
    self.assertEqual(D('26.00'), total)
    self.assertEqual(basket.num_lines, 2)
    self.assertFalse(basket.is_empty)
    This is a bad test
    

    View Slide

67. class TestBasket(TestCase):
    def test_add_items(self):
    # Create products
    product_class = ProductClass(
    name="Books", require_shipping=True)
    product1 = Product.objects.create(title="My first book")
    product2 = Product.objects.create(title="My second book")
    partner = Partner.objects.create(name="Book Partner")
    stockrecord1 = StockRecord(
    price_excl_tax=D('12.00'), partner=partner)
    stockrecord2 = StockRecord(
    price_excl_tax=D('14.00'), partner=partner)
    # Add to basket
    basket = Basket()
    basket.add(product1)
    basket.add(product2)
    total = basket.total_excl_tax
    self.assertEqual(D('26.00'), total)
    self.assertEqual(basket.num_lines, 2)
    self.assertFalse(basket.is_empty)
    Bad naming
    

    View Slide

68. class TestBasket(TestCase):
    def test_add_items(self):
    # Create products
    product_class = ProductClass(
    name="Books", require_shipping=True)
    product1 = Product.objects.create(title="My first book")
    product2 = Product.objects.create(title="My second book")
    partner = Partner.objects.create(name="Book Partner")
    stockrecord1 = StockRecord(
    price_excl_tax=D('12.00'), partner=partner)
    stockrecord2 = StockRecord(
    price_excl_tax=D('14.00'), partner=partner)
    # Add to basket
    basket = Basket()
    basket.add(product1)
    basket.add(product2)
    total = basket.total_excl_tax
    self.assertEqual(D('26.00'), total)
    self.assertEqual(basket.num_lines, 2)
    self.assertFalse(basket.is_empty)
    Heavy set-up
    

    View Slide

69. class TestBasket(TestCase):
    def test_add_items(self):
    # Create products
    product_class = ProductClass(
    name="Books", require_shipping=True)
    product1 = Product.objects.create(title="My first book")
    product2 = Product.objects.create(title="My second book")
    partner = Partner.objects.create(name="Book Partner")
    stockrecord1 = StockRecord(
    price_excl_tax=D('12.00'), partner=partner)
    stockrecord2 = StockRecord(
    price_excl_tax=D('14.00'), partner=partner)
    # Add to basket
    basket = Basket()
    basket.add(product1)
    basket.add(product2)
    total = basket.total_excl_tax
    self.assertEqual(D('26.00'), total)
    self.assertEqual(basket.num_lines, 2)
    self.assertFalse(basket.is_empty)
    Noise
    

    View Slide

70. class TestBasket(TestCase):
    def test_add_items(self):
    # Create products
    product_class = ProductClass(
    name="Books", require_shipping=True)
    product1 = Product.objects.create(title="My first book")
    product2 = Product.objects.create(title="My second book")
    partner = Partner.objects.create(name="Book Partner")
    stockrecord1 = StockRecord(
    price_excl_tax=D('12.00'), partner=partner)
    stockrecord2 = StockRecord(
    price_excl_tax=D('14.00'), partner=partner)
    # Add to basket
    basket = Basket()
    basket.add(product1)
    basket.add(product2)
    total = basket.total_excl_tax
    self.assertEqual(D('26.00'), total)
    self.assertEqual(basket.num_lines, 2)
    self.assertFalse(basket.is_empty)
    Unclear assertions
    

    View Slide

71. class TestBasket(TestCase):
    def test_add_items(self):
    # Create products
    product_class = ProductClass(
    name="Books", require_shipping=True)
    product1 = Product.objects.create(title="My first book")
    product2 = Product.objects.create(title="My second book")
    partner = Partner.objects.create(name="Book Partner")
    stockrecord1 = StockRecord(
    price_excl_tax=D('12.00'), partner=partner)
    stockrecord2 = StockRecord(
    price_excl_tax=D('14.00'), partner=partner)
    # Add to basket
    basket = Basket()
    basket.add(product1)
    basket.add(product2)
    total = basket.total_excl_tax
    self.assertEqual(D('26.00'), total)
    self.assertEqual(basket.num_lines, 2)
    self.assertFalse(basket.is_empty)
    Sloppy coding
    Numbered variable names make baby jesus cry
    

    View Slide

72. class TestBasketTotal(TestCase):
    def setUp(self):
    self.basket = Basket()
    def test_is_correct_after_adding_multiple_items(self):
    products = [
    factory.create_product(price_excl_tax=D('12.00')),
    factory.create_product(price_excl_tax=D('14.00')),
    ]
    for product in products:
    self.basket.add_product(product)
    self.assertEqual(
    D('12.00') + D('14.00'), self.basket.total_excl_tax)
    This is the same test, but cleaned up
    

    View Slide

73. class TestBasketTotal(TestCase):
    def setUp(self):
    self.basket = Basket()
    def test_is_correct_after_adding_multiple_items(self):
    products = [
    factory.create_product(price_excl_tax=D('12.00')),
    factory.create_product(price_excl_tax=D('14.00')),
    ]
    for product in products:
    self.basket.add_product(product)
    self.assertEqual(
    D('12.00') + D('14.00'), self.basket.total_excl_tax)
    Descriptive names
    

    View Slide

74. class TestBasketTotal(TestCase):
    def setUp(self):
    self.basket = Basket()
    def test_is_correct_after_adding_multiple_items(self):
    products = [
    factory.create_product(price_excl_tax=D('12.00')),
    factory.create_product(price_excl_tax=D('14.00')),
    ]
    for product in products:
    self.basket.add_product(product)
    self.assertEqual(
    D('12.00') + D('14.00'), self.basket.total_excl_tax)
    Factory functions
    Tests should only show relevant details
    

    View Slide

75. class TestBasketTotal(TestCase):
    def setUp(self):
    self.basket = Basket()
    def test_is_correct_after_adding_multiple_items(self):
    products = [
    factory.create_product(price_excl_tax=D('12.00')),
    factory.create_product(price_excl_tax=D('14.00')),
    ]
    for product in products:
    self.basket.add_product(product)
    self.assertEqual(
    D('12.00') + D('14.00'), self.basket.total_excl_tax)
    Intention revealing
    Dumb example but a good thing to do in general
    

    View Slide

76. PyHamcrest
    github.com/hamcrest/PyHamcrest
    >>> from hamcrest import *
    >>> assert_that(
    shipping.charge, is_(equal_to(D('4.99'))))
    >>> assert_that(
    "RESTful".lower(), contains_string("stfu"))
    Hamcrest - more readable but too many parentheses?
    

    View Slide

77. should-dsl
    github.com/nsi-iff/should-dsl
    >>> from should_dsl import should, should_not
    >>> {} |should| be_empty
    >>> user.email |should| contain("@")
    >>> basket |should| contain(product)
    Crazy voodoo here, but quite readable ultimately
    

    View Slide

78. This was the best image of a Smorgasbord I could find
    

    View Slide

79. wiki.python.org/moin/PythonTestingToolsTaxonomy
    For all your python testing needs.
    

    View Slide