A smörgåsbord of testing

The controversy over writing tests and TDD is over. The benefits to the design, development and maintenance of your code are real and well known. In 2013, it is simply unprofessional to eschew tests for production code.

Given this, developers need to move beyond the basics of testing and gain a deeper understanding of the art. This entails knowing the patterns and anti-patterns of effective test suites, understanding what to test and how to do it, and a mastery of the many fantastic testing tools in the Python ecosystem.

Bad tests are a genuine menace. A poorly considered test suite pours concrete over your codebase, hindering maintenance and development.

This talk isn't an introduction to testing and TDD. Rather, it is a mixed buffet of tips, anti-patterns, tools and techniques for effective testing.

Transcript

1. smörgåsbord testing of A
2. David Winterbottom @codeinthehole /codeinthehole Impending sausage
3. David Winterbottom @codeinthehole /codeinthehole Grange Hill (only British people who grew up in the 1990s will understand this)
4. Tangent Labs @tangent_labs /tangentlabs See www.tangentlabs.co.uk (PS - we’re hiring!)
5. django-oscar @django_oscar /tangentlabs/django-oscar Oscar is the most popular python e-commerce software on Github. It’s code is so beautiful, it’s been known to make grown men weep. oscarcommerce.com
6. This is Oscar’s test suite - we can see a few things which betray the biases of this talk
7. Nose Nose nose.readthedocs.org ...like nose...
8. Django ...and Django.
9. Dexterity Readability Speed The talk covers 3 topics
10. Who is this? (Hint - it’s not Philip Seymour Hoffman). It’s Kent Beck: creator of jUnit, XP etc
11. “I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence ” ...and he’s a pragmatist. Beware of dogma (100% coverage, one assertion per method etc)
12. Bad tests exist. They can pour concrete over your codebase. It’s ok to delete tests.
13. Dexterity Is there a clever Japanese word for the art of slicing? Dexterity = How to use your test suite effectively
14. Basics Love thy test-runner
15. Run ALL the tests
16. Run tests beneath path Can use your shell’s tab completion for speedy folder selection
17. Run test module
18. Run test class Colons are a pain (if you are trying to write a autocompleter) as they are word delimiters
19. Run test method This is too labour-intensive to type, so we use ...
20. nosecomplete github.com/alonho/nosecomplete Someone solved the colon problem!
21. nosecomplete github.com/alonho/nosecomplete ...and here it is in action.
22. Stories
23. Stories Work on a feature Doing TDD, working from a spec etc
24. Here’s a sample test run
25. https://github.com/asdfasdf/spec Custom test runner You need a custom test-runner
26. https://github.com/asdfasdf/spec Testing one module Run your unit tests as you develop
27. https://github.com/asdfasdf/spec Spec plugin https://github.com/bitprophet/spec Write your test classes and methods as a spec Based on Pinocchio
28. https://github.com/asdfasdf/spec BDD-style output And, write your specs to look like this BDD-style output.
29. class TestASignedInUser(WebTestCase): def setUp(self): self.user = G(User) self.order = factories.create_order(user=self.user) def test_can_see_their_email_address_on_the_profile_page(self): profile_page = self.app.get( reverse('customer:summary'), user=self.user) self.assertTrue(self.email in profile_page.content) def test_can_update_their_name(self): profile_form_page = self.app.get( reverse('customer:profile-update'), user=self.user) form = profile_form_page.forms['profile_form'] form['first_name'] = 'Barry' form['last_name'] = 'Chuckle' response = form.submit() self.assertRedirects(response, reverse('customer:summary')) # Reload user user = User.objects.get(id=self.user.id) self.assertEquals("Barry", user.first_name) self.assertEquals("Chuckle", user.last_name) The Chuckle Brothers are updating their profiles
30. class TestASignedInUser(WebTestCase): def setUp(self): self.user = G(User) self.order = factoies.create_order(user=self.user) def test_can_see_their_email_address_on_the_profile_page(self): profile_page = self.app.get( reverse('customer:summary'), user=self.user) self.assertTrue(self.email in profile_page.content) def test_can_update_their_name(self): profile_form_page = self.app.get( reverse('customer:profile-update'), user=self.user) form = profile_form_page.forms['profile_form'] form['first_name'] = 'Barry' form['last_name'] = 'Chuckle' response = form.submit() self.assertRedirects(response, reverse('customer:summary')) # Reload user user = User.objects.get(id=self.user.id) self.assertEquals("Barry", user.first_name) self.assertEquals("Chuckle", user.last_name) System under test
31. class TestASignedInUser(WebTestCase): def setUp(self): self.user = G(User) self.order = factories.create_order(user=self.user) def test_can_see_their_email_address_on_the_profile_page(self): profile_page = self.app.get( reverse('customer:summary'), user=self.user) self.assertTrue(self.email in profile_page.content) def test_can_update_their_name(self): profile_form_page = self.app.get( reverse('customer:profile-update'), user=self.user) form = profile_form_page.forms['profile_form'] form['first_name'] = 'Barry' form['last_name'] = 'Chuckle' response = form.submit() self.assertRedirects(response, reverse('customer:summary')) # Reload user user = User.objects.get(id=self.user.id) self.assertEquals("Barry", user.first_name) self.assertEquals("Chuckle", user.last_name) Specification On the road towards a full BDD framwork like Lettuce/Behave, but you can still use Nose and all its plugins
32. Use-cases Work on a new feature Refactor a feature You’re reviewing a pull request
33. https://github.com/nose attribs/spec Run all the shipping tests!
34. nose.readthedocs.org/en/latest/plugins/attrib.html Run tagged tests
35. from nose.plugins.attrib import attr @attr('shipping') class TestFreeShippping(TestCase): def test_is_free_for_empty_basket(self): ... class TestFixedPriceShipping(TestCase): @attr(slow=True) def test_returns_fixed_price_for_basket(self): ...
36. from nose.plugins.attrib import attr @attr('shipping') class TestFreeShippping(TestCase): def test_is_free_for_empty_basket(self): ... class TestFixedPriceShipping(TestCase): @attr(slow=True) def test_returns_fixed_price_for_basket(self): ... Wrap a class
37. from nose.plugins.attrib import attr @attr('shipping') class TestFreeShippping(TestCase): def test_is_free_for_empty_basket(self): ... class TestFixedPriceShipping(TestCase): @attr(slow=True) def test_returns_fixed_price_for_basket(self): ... Wrap a method You can tag slow tests
39. Pattern match tests github.com/iElectric/nose-selecttests Better for lazy developers
40. Use-cases Work on a new feature Refactor a feature Fix failing tests
41. Fail fast
42. Editor-friendly paths github.com/erikrose/nose-progressive Other nice features: prints out failures/errors as they occur, progress bar
43. PDB on failure Need to go up the stack to get to test code
44. Only re-run failed tests Needs to be run twice though
45. Use-cases Work on a new feature Refactor a feature Fix failing tests Commit Before pushing etc
46. Run all the tests as fast as possible
47. No plugins
48. Parallelise if you use tox, then try detox
49. Speed Watch Gary Bernhardt’s talks
50. 0 4 8 12 16 20 24 Focus / flow / “zone” Test suite time (seconds)
51. 0 4 8 12 16 20 24 Focus / flow / “zone” Test suite time (seconds) Pre-commit hook? Everything’s rosy when your test suite runs in 100ms
52. 0 4 8 12 16 20 24 Focus / flow / “zone” Test suite time (seconds) But watch out for the threshold of doom
53. 0 4 8 12 16 20 24 Test suite time (seconds) Oh no - you’re reading Hacker News!
54. Tips for speeding up tests will be in another talk Oscar distinguishes between unit, integration and functional (acceptance) tests to try and steer people towards unit tests
56. Scam? blog.thecodewhisperer.com/2010/10/16/integrated-tests-are-a-scam/ Read this polemic
57. import mock from django.utils.functional import curry no_database = curry( mock.patch, 'django.db.backends.util.CursorWrapper', Mock(side_effect=RuntimeError( "Using the database is not permitted"))) @no_database() class AUnitTest(TestCase): ... http://bit.ly/126vPrt
58. import mock from django.utils.functional import curry no_database = curry( mock.patch, 'django.db.backends.util.CursorWrapper', Mock(side_effect=RuntimeError( "Using the database is not permitted"))) @no_database() class AUnitTest(TestCase): ... http://bit.ly/126vPrt Currying mock.patch! All my favourite bits of Python in one curio
59. import mock from django.utils.functional import curry no_database = curry( mock.patch, 'django.db.backends.util.CursorWrapper', Mock(side_effect=RuntimeError( "Using the database is not permitted"))) @no_database() class AUnitTest(TestCase): ... http://bit.ly/126vPrt Wrap classes/methods
60. http://bit.ly/156c48x Number of tests Seconds Oscar’s test suite has doubled number of tests while halving the time to run.
61. PASSWORD_HASHERS = ['django.contrib.auth.hashers.MD5PasswordHasher'] Using this hasher makes a big difference.
62. --processes=8 Maik spotted this option for Nose and we all had a cup of tea to celebrate.
63. http://bit.ly/2gZnhj Boy scout rule... Always leave the campsite cleaner than how you found it
64. Readability
65. http://bit.ly/19ganRJ (Refactoring a Test) Book has a great chapter on refactoring a test from dirty to clean
66. class TestBasket(TestCase): def test_add_items(self): # Create products product_class = ProductClass( name="Books", require_shipping=True) product1 = Product.objects.create(title="My first book") product2 = Product.objects.create(title="My second book") partner = Partner.objects.create(name="Book Partner") stockrecord1 = StockRecord( price_excl_tax=D('12.00'), partner=partner) stockrecord2 = StockRecord( price_excl_tax=D('14.00'), partner=partner) # Add to basket basket = Basket() basket.add(product1) basket.add(product2) total = basket.total_excl_tax self.assertEqual(D('26.00'), total) self.assertEqual(basket.num_lines, 2) self.assertFalse(basket.is_empty) This is a bad test
67. class TestBasket(TestCase): def test_add_items(self): # Create products product_class = ProductClass( name="Books", require_shipping=True) product1 = Product.objects.create(title="My first book") product2 = Product.objects.create(title="My second book") partner = Partner.objects.create(name="Book Partner") stockrecord1 = StockRecord( price_excl_tax=D('12.00'), partner=partner) stockrecord2 = StockRecord( price_excl_tax=D('14.00'), partner=partner) # Add to basket basket = Basket() basket.add(product1) basket.add(product2) total = basket.total_excl_tax self.assertEqual(D('26.00'), total) self.assertEqual(basket.num_lines, 2) self.assertFalse(basket.is_empty) Bad naming
68. class TestBasket(TestCase): def test_add_items(self): # Create products product_class = ProductClass( name="Books", require_shipping=True) product1 = Product.objects.create(title="My first book") product2 = Product.objects.create(title="My second book") partner = Partner.objects.create(name="Book Partner") stockrecord1 = StockRecord( price_excl_tax=D('12.00'), partner=partner) stockrecord2 = StockRecord( price_excl_tax=D('14.00'), partner=partner) # Add to basket basket = Basket() basket.add(product1) basket.add(product2) total = basket.total_excl_tax self.assertEqual(D('26.00'), total) self.assertEqual(basket.num_lines, 2) self.assertFalse(basket.is_empty) Heavy set-up
69. class TestBasket(TestCase): def test_add_items(self): # Create products product_class = ProductClass( name="Books", require_shipping=True) product1 = Product.objects.create(title="My first book") product2 = Product.objects.create(title="My second book") partner = Partner.objects.create(name="Book Partner") stockrecord1 = StockRecord( price_excl_tax=D('12.00'), partner=partner) stockrecord2 = StockRecord( price_excl_tax=D('14.00'), partner=partner) # Add to basket basket = Basket() basket.add(product1) basket.add(product2) total = basket.total_excl_tax self.assertEqual(D('26.00'), total) self.assertEqual(basket.num_lines, 2) self.assertFalse(basket.is_empty) Noise
70. class TestBasket(TestCase): def test_add_items(self): # Create products product_class = ProductClass( name="Books", require_shipping=True) product1 = Product.objects.create(title="My first book") product2 = Product.objects.create(title="My second book") partner = Partner.objects.create(name="Book Partner") stockrecord1 = StockRecord( price_excl_tax=D('12.00'), partner=partner) stockrecord2 = StockRecord( price_excl_tax=D('14.00'), partner=partner) # Add to basket basket = Basket() basket.add(product1) basket.add(product2) total = basket.total_excl_tax self.assertEqual(D('26.00'), total) self.assertEqual(basket.num_lines, 2) self.assertFalse(basket.is_empty) Unclear assertions
71. class TestBasket(TestCase): def test_add_items(self): # Create products product_class = ProductClass( name="Books", require_shipping=True) product1 = Product.objects.create(title="My first book") product2 = Product.objects.create(title="My second book") partner = Partner.objects.create(name="Book Partner") stockrecord1 = StockRecord( price_excl_tax=D('12.00'), partner=partner) stockrecord2 = StockRecord( price_excl_tax=D('14.00'), partner=partner) # Add to basket basket = Basket() basket.add(product1) basket.add(product2) total = basket.total_excl_tax self.assertEqual(D('26.00'), total) self.assertEqual(basket.num_lines, 2) self.assertFalse(basket.is_empty) Sloppy coding Numbered variable names make baby jesus cry
72. class TestBasketTotal(TestCase): def setUp(self): self.basket = Basket() def test_is_correct_after_adding_multiple_items(self): products = [ factory.create_product(price_excl_tax=D('12.00')), factory.create_product(price_excl_tax=D('14.00')), ] for product in products: self.basket.add_product(product) self.assertEqual( D('12.00') + D('14.00'), self.basket.total_excl_tax) This is the same test, but cleaned up
73. class TestBasketTotal(TestCase): def setUp(self): self.basket = Basket() def test_is_correct_after_adding_multiple_items(self): products = [ factory.create_product(price_excl_tax=D('12.00')), factory.create_product(price_excl_tax=D('14.00')), ] for product in products: self.basket.add_product(product) self.assertEqual( D('12.00') + D('14.00'), self.basket.total_excl_tax) Descriptive names
74. class TestBasketTotal(TestCase): def setUp(self): self.basket = Basket() def test_is_correct_after_adding_multiple_items(self): products = [ factory.create_product(price_excl_tax=D('12.00')), factory.create_product(price_excl_tax=D('14.00')), ] for product in products: self.basket.add_product(product) self.assertEqual( D('12.00') + D('14.00'), self.basket.total_excl_tax) Factory functions Tests should only show relevant details
75. class TestBasketTotal(TestCase): def setUp(self): self.basket = Basket() def test_is_correct_after_adding_multiple_items(self): products = [ factory.create_product(price_excl_tax=D('12.00')), factory.create_product(price_excl_tax=D('14.00')), ] for product in products: self.basket.add_product(product) self.assertEqual( D('12.00') + D('14.00'), self.basket.total_excl_tax) Intention revealing Dumb example but a good thing to do in general
76. PyHamcrest github.com/hamcrest/PyHamcrest >>> from hamcrest import * >>> assert_that( shipping.charge, is_(equal_to(D('4.99')))) >>> assert_that( "RESTful".lower(), contains_string("stfu")) Hamcrest - more readable but too many parentheses?
77. should-dsl github.com/nsi-iff/should-dsl >>> from should_dsl import should, should_not >>> {} |should| be_empty >>> user.email |should| contain("@") >>> basket |should| contain(product) Crazy voodoo here, but quite readable ultimately
78. This was the best image of a Smorgasbord I could find
79. wiki.python.org/moin/PythonTestingToolsTaxonomy For all your python testing needs.