A smörgåsbord of testing

A smörgåsbord of testing

The controversy over writing tests and TDD is over. The benefits to the design, development and maintenance of your code are real and well known. In 2013, it is simply unprofessional to eschew tests for production code.

Given this, developers need to move beyond the basics of testing and gain a deeper understanding of the art. This entails knowing the patterns and anti-patterns of effective test suites, understanding what to test and how to do it, and a mastery of the many fantastic testing tools in the Python ecosystem.

Bad tests are a genuine menace. A poorly considered test suite pours concrete over your codebase, hindering maintenance and development.

This talk isn't an introduction to testing and TDD. Rather, it is a mixed buffet of tips, anti-patterns, tools and techniques for effective testing.

52d39c7b27386ca98bc016119d95b8b8?s=128

David Winterbottom

September 20, 2013
Tweet

Transcript

  1. smörgåsbord testing of A

  2. David Winterbottom @codeinthehole /codeinthehole Impending sausage

  3. David Winterbottom @codeinthehole /codeinthehole Grange Hill (only British people who

    grew up in the 1990s will understand this)
  4. Tangent Labs @tangent_labs /tangentlabs See www.tangentlabs.co.uk (PS - we’re hiring!)

  5. django-oscar @django_oscar /tangentlabs/django-oscar Oscar is the most popular python e-commerce

    software on Github. It’s code is so beautiful, it’s been known to make grown men weep. oscarcommerce.com
  6. This is Oscar’s test suite - we can see a

    few things which betray the biases of this talk
  7. Nose Nose nose.readthedocs.org ...like nose...

  8. Django ...and Django.

  9. Dexterity Readability Speed The talk covers 3 topics

  10. Who is this? (Hint - it’s not Philip Seymour Hoffman).

    It’s Kent Beck: creator of jUnit, XP etc
  11. “I get paid for code that works, not for tests,

    so my philosophy is to test as little as possible to reach a given level of confidence ” ...and he’s a pragmatist. Beware of dogma (100% coverage, one assertion per method etc)
  12. Bad tests exist. They can pour concrete over your codebase.

    It’s ok to delete tests.
  13. Dexterity Is there a clever Japanese word for the art

    of slicing? Dexterity = How to use your test suite effectively
  14. Basics Love thy test-runner

  15. Run ALL the tests

  16. Run tests beneath path Can use your shell’s tab completion

    for speedy folder selection
  17. Run test module

  18. Run test class Colons are a pain (if you are

    trying to write a autocompleter) as they are word delimiters
  19. Run test method This is too labour-intensive to type, so

    we use ...
  20. nosecomplete github.com/alonho/nosecomplete Someone solved the colon problem!

  21. nosecomplete github.com/alonho/nosecomplete ...and here it is in action.

  22. Stories

  23. Stories Work on a feature Doing TDD, working from a

    spec etc
  24. Here’s a sample test run

  25. https://github.com/asdfasdf/spec Custom test runner You need a custom test-runner

  26. https://github.com/asdfasdf/spec Testing one module Run your unit tests as you

    develop
  27. https://github.com/asdfasdf/spec Spec plugin https://github.com/bitprophet/spec Write your test classes and methods

    as a spec Based on Pinocchio
  28. https://github.com/asdfasdf/spec BDD-style output And, write your specs to look like

    this BDD-style output.
  29. class TestASignedInUser(WebTestCase): def setUp(self): self.user = G(User) self.order = factories.create_order(user=self.user)

    def test_can_see_their_email_address_on_the_profile_page(self): profile_page = self.app.get( reverse('customer:summary'), user=self.user) self.assertTrue(self.email in profile_page.content) def test_can_update_their_name(self): profile_form_page = self.app.get( reverse('customer:profile-update'), user=self.user) form = profile_form_page.forms['profile_form'] form['first_name'] = 'Barry' form['last_name'] = 'Chuckle' response = form.submit() self.assertRedirects(response, reverse('customer:summary')) # Reload user user = User.objects.get(id=self.user.id) self.assertEquals("Barry", user.first_name) self.assertEquals("Chuckle", user.last_name) The Chuckle Brothers are updating their profiles
  30. class TestASignedInUser(WebTestCase): def setUp(self): self.user = G(User) self.order = factoies.create_order(user=self.user)

    def test_can_see_their_email_address_on_the_profile_page(self): profile_page = self.app.get( reverse('customer:summary'), user=self.user) self.assertTrue(self.email in profile_page.content) def test_can_update_their_name(self): profile_form_page = self.app.get( reverse('customer:profile-update'), user=self.user) form = profile_form_page.forms['profile_form'] form['first_name'] = 'Barry' form['last_name'] = 'Chuckle' response = form.submit() self.assertRedirects(response, reverse('customer:summary')) # Reload user user = User.objects.get(id=self.user.id) self.assertEquals("Barry", user.first_name) self.assertEquals("Chuckle", user.last_name) System under test
  31. class TestASignedInUser(WebTestCase): def setUp(self): self.user = G(User) self.order = factories.create_order(user=self.user)

    def test_can_see_their_email_address_on_the_profile_page(self): profile_page = self.app.get( reverse('customer:summary'), user=self.user) self.assertTrue(self.email in profile_page.content) def test_can_update_their_name(self): profile_form_page = self.app.get( reverse('customer:profile-update'), user=self.user) form = profile_form_page.forms['profile_form'] form['first_name'] = 'Barry' form['last_name'] = 'Chuckle' response = form.submit() self.assertRedirects(response, reverse('customer:summary')) # Reload user user = User.objects.get(id=self.user.id) self.assertEquals("Barry", user.first_name) self.assertEquals("Chuckle", user.last_name) Specification On the road towards a full BDD framwork like Lettuce/Behave, but you can still use Nose and all its plugins
  32. Use-cases Work on a new feature Refactor a feature You’re

    reviewing a pull request
  33. https://github.com/nose attribs/spec Run all the shipping tests!

  34. nose.readthedocs.org/en/latest/plugins/attrib.html Run tagged tests

  35. from nose.plugins.attrib import attr @attr('shipping') class TestFreeShippping(TestCase): def test_is_free_for_empty_basket(self): ...

    class TestFixedPriceShipping(TestCase): @attr(slow=True) def test_returns_fixed_price_for_basket(self): ...
  36. from nose.plugins.attrib import attr @attr('shipping') class TestFreeShippping(TestCase): def test_is_free_for_empty_basket(self): ...

    class TestFixedPriceShipping(TestCase): @attr(slow=True) def test_returns_fixed_price_for_basket(self): ... Wrap a class
  37. from nose.plugins.attrib import attr @attr('shipping') class TestFreeShippping(TestCase): def test_is_free_for_empty_basket(self): ...

    class TestFixedPriceShipping(TestCase): @attr(slow=True) def test_returns_fixed_price_for_basket(self): ... Wrap a method You can tag slow tests
  38. None
  39. Pattern match tests github.com/iElectric/nose-selecttests Better for lazy developers

  40. Use-cases Work on a new feature Refactor a feature Fix

    failing tests
  41. Fail fast

  42. Editor-friendly paths github.com/erikrose/nose-progressive Other nice features: prints out failures/errors as

    they occur, progress bar
  43. PDB on failure Need to go up the stack to

    get to test code
  44. Only re-run failed tests Needs to be run twice though

  45. Use-cases Work on a new feature Refactor a feature Fix

    failing tests Commit Before pushing etc
  46. Run all the tests as fast as possible

  47. No plugins

  48. Parallelise if you use tox, then try detox

  49. Speed Watch Gary Bernhardt’s talks

  50. 0 4 8 12 16 20 24 Focus / flow

    / “zone” Test suite time (seconds)
  51. 0 4 8 12 16 20 24 Focus / flow

    / “zone” Test suite time (seconds) Pre-commit hook? Everything’s rosy when your test suite runs in 100ms
  52. 0 4 8 12 16 20 24 Focus / flow

    / “zone” Test suite time (seconds) But watch out for the threshold of doom
  53. 0 4 8 12 16 20 24 Test suite time

    (seconds) Oh no - you’re reading Hacker News!
  54. Tips for speeding up tests will be in another talk

    Oscar distinguishes between unit, integration and functional (acceptance) tests to try and steer people towards unit tests
  55. None
  56. Scam? blog.thecodewhisperer.com/2010/10/16/integrated-tests-are-a-scam/ Read this polemic

  57. import mock from django.utils.functional import curry no_database = curry( mock.patch,

    'django.db.backends.util.CursorWrapper', Mock(side_effect=RuntimeError( "Using the database is not permitted"))) @no_database() class AUnitTest(TestCase): ... http://bit.ly/126vPrt
  58. import mock from django.utils.functional import curry no_database = curry( mock.patch,

    'django.db.backends.util.CursorWrapper', Mock(side_effect=RuntimeError( "Using the database is not permitted"))) @no_database() class AUnitTest(TestCase): ... http://bit.ly/126vPrt Currying mock.patch! All my favourite bits of Python in one curio
  59. import mock from django.utils.functional import curry no_database = curry( mock.patch,

    'django.db.backends.util.CursorWrapper', Mock(side_effect=RuntimeError( "Using the database is not permitted"))) @no_database() class AUnitTest(TestCase): ... http://bit.ly/126vPrt Wrap classes/methods
  60. http://bit.ly/156c48x Number of tests Seconds Oscar’s test suite has doubled

    number of tests while halving the time to run.
  61. PASSWORD_HASHERS = ['django.contrib.auth.hashers.MD5PasswordHasher'] Using this hasher makes a big difference.

  62. --processes=8 Maik spotted this option for Nose and we all

    had a cup of tea to celebrate.
  63. http://bit.ly/2gZnhj Boy scout rule... Always leave the campsite cleaner than

    how you found it
  64. Readability

  65. http://bit.ly/19ganRJ (Refactoring a Test) Book has a great chapter on

    refactoring a test from dirty to clean
  66. class TestBasket(TestCase): def test_add_items(self): # Create products product_class = ProductClass(

    name="Books", require_shipping=True) product1 = Product.objects.create(title="My first book") product2 = Product.objects.create(title="My second book") partner = Partner.objects.create(name="Book Partner") stockrecord1 = StockRecord( price_excl_tax=D('12.00'), partner=partner) stockrecord2 = StockRecord( price_excl_tax=D('14.00'), partner=partner) # Add to basket basket = Basket() basket.add(product1) basket.add(product2) total = basket.total_excl_tax self.assertEqual(D('26.00'), total) self.assertEqual(basket.num_lines, 2) self.assertFalse(basket.is_empty) This is a bad test
  67. class TestBasket(TestCase): def test_add_items(self): # Create products product_class = ProductClass(

    name="Books", require_shipping=True) product1 = Product.objects.create(title="My first book") product2 = Product.objects.create(title="My second book") partner = Partner.objects.create(name="Book Partner") stockrecord1 = StockRecord( price_excl_tax=D('12.00'), partner=partner) stockrecord2 = StockRecord( price_excl_tax=D('14.00'), partner=partner) # Add to basket basket = Basket() basket.add(product1) basket.add(product2) total = basket.total_excl_tax self.assertEqual(D('26.00'), total) self.assertEqual(basket.num_lines, 2) self.assertFalse(basket.is_empty) Bad naming
  68. class TestBasket(TestCase): def test_add_items(self): # Create products product_class = ProductClass(

    name="Books", require_shipping=True) product1 = Product.objects.create(title="My first book") product2 = Product.objects.create(title="My second book") partner = Partner.objects.create(name="Book Partner") stockrecord1 = StockRecord( price_excl_tax=D('12.00'), partner=partner) stockrecord2 = StockRecord( price_excl_tax=D('14.00'), partner=partner) # Add to basket basket = Basket() basket.add(product1) basket.add(product2) total = basket.total_excl_tax self.assertEqual(D('26.00'), total) self.assertEqual(basket.num_lines, 2) self.assertFalse(basket.is_empty) Heavy set-up
  69. class TestBasket(TestCase): def test_add_items(self): # Create products product_class = ProductClass(

    name="Books", require_shipping=True) product1 = Product.objects.create(title="My first book") product2 = Product.objects.create(title="My second book") partner = Partner.objects.create(name="Book Partner") stockrecord1 = StockRecord( price_excl_tax=D('12.00'), partner=partner) stockrecord2 = StockRecord( price_excl_tax=D('14.00'), partner=partner) # Add to basket basket = Basket() basket.add(product1) basket.add(product2) total = basket.total_excl_tax self.assertEqual(D('26.00'), total) self.assertEqual(basket.num_lines, 2) self.assertFalse(basket.is_empty) Noise
  70. class TestBasket(TestCase): def test_add_items(self): # Create products product_class = ProductClass(

    name="Books", require_shipping=True) product1 = Product.objects.create(title="My first book") product2 = Product.objects.create(title="My second book") partner = Partner.objects.create(name="Book Partner") stockrecord1 = StockRecord( price_excl_tax=D('12.00'), partner=partner) stockrecord2 = StockRecord( price_excl_tax=D('14.00'), partner=partner) # Add to basket basket = Basket() basket.add(product1) basket.add(product2) total = basket.total_excl_tax self.assertEqual(D('26.00'), total) self.assertEqual(basket.num_lines, 2) self.assertFalse(basket.is_empty) Unclear assertions
  71. class TestBasket(TestCase): def test_add_items(self): # Create products product_class = ProductClass(

    name="Books", require_shipping=True) product1 = Product.objects.create(title="My first book") product2 = Product.objects.create(title="My second book") partner = Partner.objects.create(name="Book Partner") stockrecord1 = StockRecord( price_excl_tax=D('12.00'), partner=partner) stockrecord2 = StockRecord( price_excl_tax=D('14.00'), partner=partner) # Add to basket basket = Basket() basket.add(product1) basket.add(product2) total = basket.total_excl_tax self.assertEqual(D('26.00'), total) self.assertEqual(basket.num_lines, 2) self.assertFalse(basket.is_empty) Sloppy coding Numbered variable names make baby jesus cry
  72. class TestBasketTotal(TestCase): def setUp(self): self.basket = Basket() def test_is_correct_after_adding_multiple_items(self): products

    = [ factory.create_product(price_excl_tax=D('12.00')), factory.create_product(price_excl_tax=D('14.00')), ] for product in products: self.basket.add_product(product) self.assertEqual( D('12.00') + D('14.00'), self.basket.total_excl_tax) This is the same test, but cleaned up
  73. class TestBasketTotal(TestCase): def setUp(self): self.basket = Basket() def test_is_correct_after_adding_multiple_items(self): products

    = [ factory.create_product(price_excl_tax=D('12.00')), factory.create_product(price_excl_tax=D('14.00')), ] for product in products: self.basket.add_product(product) self.assertEqual( D('12.00') + D('14.00'), self.basket.total_excl_tax) Descriptive names
  74. class TestBasketTotal(TestCase): def setUp(self): self.basket = Basket() def test_is_correct_after_adding_multiple_items(self): products

    = [ factory.create_product(price_excl_tax=D('12.00')), factory.create_product(price_excl_tax=D('14.00')), ] for product in products: self.basket.add_product(product) self.assertEqual( D('12.00') + D('14.00'), self.basket.total_excl_tax) Factory functions Tests should only show relevant details
  75. class TestBasketTotal(TestCase): def setUp(self): self.basket = Basket() def test_is_correct_after_adding_multiple_items(self): products

    = [ factory.create_product(price_excl_tax=D('12.00')), factory.create_product(price_excl_tax=D('14.00')), ] for product in products: self.basket.add_product(product) self.assertEqual( D('12.00') + D('14.00'), self.basket.total_excl_tax) Intention revealing Dumb example but a good thing to do in general
  76. PyHamcrest github.com/hamcrest/PyHamcrest >>> from hamcrest import * >>> assert_that( shipping.charge,

    is_(equal_to(D('4.99')))) >>> assert_that( "RESTful".lower(), contains_string("stfu")) Hamcrest - more readable but too many parentheses?
  77. should-dsl github.com/nsi-iff/should-dsl >>> from should_dsl import should, should_not >>> {}

    |should| be_empty >>> user.email |should| contain("@") >>> basket |should| contain(product) Crazy voodoo here, but quite readable ultimately
  78. This was the best image of a Smorgasbord I could

    find
  79. wiki.python.org/moin/PythonTestingToolsTaxonomy For all your python testing needs.