THE ART OF TESTING IN PYTHON

How do you know whether the code that you wrote is correct? How do you change it without breaking existing functionality? To me, the answer lies in automated testing and I would like to introduce you to this vast topic. We will start by discussing the various ways of testing software and their relative advantages and disadvantages.  We will proceed to develop some simple code that encapsulates the common problems faced while testing and I will show you how to overcome them and engineer your test code using tools in the unittest module. We will also peek behind the curtains to see how a testrunner executes your code. By the end of the talk, I hope you will have a solid understanding of basics of testing and a newfound appreciation of classic paintings.

This talk is mostly aimed at people new to software testing. However, more experienced developers are also welcome there will be plenty of questions for the audience and a good number of charades inspired guessing!

There is an excellent talk by Ned Batchelder on getting started with testing in Python which helped me enormously to organize this presentation: https://nedbatchelder.com/text/test0.html
as Oscar Wilde said: “Imitation is the sincerest form of flattery that mediocrity can pay to greatness.”, Ned should feel very flattered!

Transcript

1. The Art of Testing Python OUCH iamjanhak goo.gl/sFi6H2
2. Goals  Explain the motivation for testing  Show different ways to test and introduce core concepts  Present the correct way to implement them in Python goo.gl/sFi6H2 iamjanhak
3. What is the point of testing?  Verify that your code works  Make development predictable  Better design  Gives you confidence in your code  “Debugging is hard, testing is easy” goo.gl/sFi6H2 iamjanhak
4. How I feel when I am not writing tests!
5. It is hard…  A lot of effort  You don’t want to do it  However: it pays off goo.gl/sFi6H2 iamjanhak
6. Proceed fearlessly with the Lance of tests!
7. What we will do today  How we can test code  Principles of testing  Introduce unittest framework goo.gl/sFi6H2 iamjanhak
8. Finally! Let’s build some tests
9. Cargo ship class class Ship: """Transport naval vessel""" def __init__(self): # cargo is a list of lists: # [[name, quantity, weight], ... ] self.cargo = [] def load(self, name, quantity, weight): # load `name`: `weight` per `quantity` self.cargo.append([name, quantity, weight]) def weight(self): """Total weight of the cargo onboard""" load = 0.0 for name, quantity, weight in self.cargo: load += weight * quantity return load  Cargo  Load  Weight goo.gl/sFi6H2
10. Interactive testing >>> s = Ship() >>> s.weight() 0.0 >>> s.load('Lions', 2, 190) >>> s.weight() 380.0 >>> s.load('African swallows', 20, 0.2) >>> s.weight() 384.0  Good: testing the code  Bad: time consuming  Bad: not repeatable  Bad: is it correct? goo.gl/sFi6H2 iamjanhak
11. Automated testing from ship import Ship s = Ship() print('Empty ship has cargo weighing: %s' % s.weight()) s.load('Elephant', 2, 3500) print('Ship with 2 elephants 3500kg each weighs: %s' % s.weight()) s.load('Lion', 2, 350) print('Add 2 Lions 350kg each weight is: %s' % s.weight()) > python ship_test1.py Empty ship has cargo weighing: 0.0 Ship with 2 elephants 3500kg each weighs: 7000.0 Add 2 Lions 350kg each weight is: 7700.0  Good: testing the code  Better: repeatable  Better: effortless  Bad: is it correct? goo.gl/sFi6H2 iamjanhak
12. Automated testing – explicit results from ship import Ship s = Ship() print('Empty ship: %s, should be 0.0' % s.weight()) s.load('Elephant', 2, 3500) print('Add 2 elephants @ 3500kg: %s, should be 7000kg' % s.weight()) s.load('Lion', 2, 350) print('Add 2 Lions @ 350kg: %s, should be 7700kg' % s.weight()) > python ship_test2.py Empty ship: 0.0, should be 0.0 Add 2 elephants @ 3500kg: 7000.0, should be 7000kg Add 2 Lions @ 350kg: 7700.0, should be 7700kg  Good: repeatable  Good: effortless  Better: explicit results  Bad: checking is manual goo.gl/sFi6H2 iamjanhak
13. Automated testing - assertions from ship import Ship s = Ship() print('Empty ship: %s, should be 0.0' % s.weight()) assert s.weight() == 0.0 s.load('Elephant', 2, 3500) print('Add 2 elephants @ 3500kg: %s, should be 7000kg' % s.weight()) assert s.weight() == 7000 s.load('Lion', 2, 350) print('Add 2 Lions @ 350kg: %s, should be 7700kg' % s.weight()) assert s.weight() == 7700 > python ship_test2.py Empty ship: 0.0, should be 0.0 Add 2 elephants @ 3500kg: 7000.0, should be 7000kg Add 2 Lions @ 350kg: 7700.0, should be 7700kg  Good: repeatable  Good: low effort  Good: explicit results  Good: verifiable goo.gl/sFi6H2
14. What do failures look like? > python ship_test3_broken.py Empty ship: 0.0, should be 0.0 Add 2 elephants @ 3500kg: 7000.0, should be 7000kg Add 2 Lions @ 350kg: 7700.0, should be 7700kg Traceback (most recent call last): File "ship_test3_broken.py", line 11, in <module> assert s.weight() == 8500 AssertionError  Good: repeatable with little effort  Good: explicitly verify the correctness  So so: failures are visible, but among other output  Bad: testing stops at the first failure goo.gl/sFi6H2
15. This will quickly get complicated!  Common problems with testing  Find a consistent way of solving them  We should invent a standard that everyone follows goo.gl/sFi6H2 iamjanhak
16. Principles of testing F.I.R.S.T.  Fast  Informative  Repeatable  Self-verifying  Timely
17. The way to write our tests unittest
18. First unit test import unittest from ship import Ship class ShipTest(unittest.TestCase): def test_load_cargo(self): s = Ship() s.load('Elephant', 2, 3500) self.assertEquals(s.weight(), 7000) > python -m unittest test_ship1 . ---------------------------------------------------------------------- Ran 1 test in 0.000s OK goo.gl/sFi6H2
19. Behind the scenes # unittest runs the test as if: testcase = ShipTest() try: testcase.test_load_cargo() except AssertionError: [record failure] else: [record success]
20. Let’s add some tests import unittest from ship1 import Ship class ShipTest(unittest.TestCase): def test_empty_ship_has_no_cargo(self): s = Ship() self.assertEquals(s.weight(), 0) def test_load_cargo(self): s = Ship() s.load('Elephant', 2, 3500) self.assertEquals(s.weight(), 7000) def test_load_two_cargo(self): s = Ship() s.load('Elephant', 2, 3500) s.load('Lion', 2, 350) self.assertEquals(s.weight(), 7700) > python -m unittest test_ship2 ... ------------------------------------------------------------ ---------- Ran 3 tests in 0.000s OK  Arrange  Act  Assert 8000 goo.gl/sFi6H2
21. Failure >python -m unittest test_ship2_broken .F. ====================================================================== FAIL: test_load_cargo (test_ship2_broken.ShipTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "test_ship2_broken.py", line 12, in test_load_cargo self.assertEquals(s.weight(), 8000) AssertionError: 7000.0 != 8000 ---------------------------------------------------------------------- Ran 3 tests in 0.000s FAILED (failures=1)  Good: expected result is clear  Good: failure didn’t stop execution  Good: output is not cluttered
22. Behind the scenes testcase = ShipTest() try: testcase.empty_ship_has_no_cargo() except AssertionError: [record failure] else: [record success] testcase = ShipTest() try: testcase.test_load_cargo() except AssertionError: [record failure] else: [record success] testcase = ShipTest() try: testcase.test_load_two_cargo() except AssertionError: [record failure] else: [record success]
23. Test Independence  Every test is executed independently  Tests can’t affect each other  Failure doesn’t stop the test runner
24. Customize your base class class ShipTestCase(unittest.TestCase): """Base class for all ship tests.""" def assertWeightEquals(self, s, weight): """Asserts that s.weight() is equal to weight.""" self.assertEquals(s.weight(), weight) class ShipTest(ShipTestCase): def test_empty_ship_has_no_cargo(self): s = Ship() self.assertWeightEquals(s, 0) def test_load_cargo(self): s = Ship() s.load('Elephant', 2) self.assertWeightEquals(s, 7000) def test_load_two_cargo(self): s = Ship() s.load('Elephant', 2, 3500) s.load('Lion', 2, 350) self.assertWeightEquals(s, 7700) goo.gl/sFi6H2
25. Error > python -m unittest test_ship3_broken E.. ====================================================================== ERROR: test_bad_input (test_ship3_broken.ShipTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "test_ship3_broken.py", line 23, in test_bad_input s.load('Elephant', 2) TypeError: load() takes exactly 4 arguments (3 given) ---------------------------------------------------------------------- Ran 3 tests in 0.000s FAILED (errors=1) Test raises an exception other then AssertionError goo.gl/sFi6H2
26. How to test for expected failures def test_bad_input(self): s = Ship() with self.assertRaises(TypeError): s.load('Elephant', 2) > python -m unittest test_ship4 .... ------------------------------------------- Ran 4 tests in 0.000s OK TestCase.assertRaises goo.gl/sFi6H2
27. Ark: .unload() def unload(self, name, quantity): """Unloads some cargo.""" for item in self.cargo: if item[0] == name: if item[1] < quantity: raise ValueError('Not enough cargo on board') item[1] -= quantity return else: raise ValueError('No cargo of this type on board') goo.gl/sFi6H2 iamjanhak
28. Testing unload() class ShipTestUnload(ShipTestCase): def test_unload(self): s = Ship() s.load('Elephant', 2, 3500) s.load('Lion', 2, 350) s.load('T-Rex', 2, 7500) s.unload('T-Rex', 2) self.assertWeightEquals(s, 7700) def test_unload_cargo_when_not_enough(self): s = Ship() s.load('Elephant', 2, 3500) s.load('Lion', 2, 350) s.load('T-Rex', 2, 7500) with self.assertRaises(ValueError): s.unload('T-Rex', 3) def test_unload_cargo_not_on_board(self): s = Ship() s.load('Elephant', 2, 3500) s.load('Lion', 2, 350) s.load('T-Rex', 2, 7500) s.unload('T-Rex', 2) with self.assertRaises(ValueError): s.unload('African Swallow', 20)
29. Test set up - fixtures class ShipTestUnload(ShipTestCase): def setUp(self): self.s = Ship() self.s.load('Elephant', 2, 3500) self.s.load('Lion', 2, 350) self.s.load('T-Rex', 2, 7500) def test_unload(self): self.s.unload('T-Rex', 2) self.assertWeightEquals(self.s, 7700) def test_unload_cargo_when_not_enough(self): with self.assertRaises(ValueError): self.s.unload('T-Rex', 3) def test_unload_cargo_not_on_board(self): with self.assertRaises(ValueError): self.s.unload('African Swallow', 20)
30. Behind the scenes testcase = ShipTest() try: testcase.setUp() except: [record error] else: try: testcase.a_test() except AssertionError: [record failure] else: [record success] finally: try: testcase.tearDown() except: [record error]
31. setUp and tearDown are your friends  Solve common preparation and clean up work  Give isolation even with failures  Promote consistency and conciseness  setUpClass and tearDownClass for time consuming and/or static environment goo.gl/sFi6H2 iamjanhak
32. your test suite  Write helper functions and classes  Come up with suitable setUp and tearDown methods  Use Fixture libraries  Use domain language to promote communication goo.gl/sFi6H2 iamjanhak
33. Testing is rich and complex there is loads to explore. What comes next?
34. Tools to explore  coverage – a way to see how much of your code is tested  hypothesis – property based testing that works  pytest – pythonic way to write tests!  doctest – test code snippets in your comments  selenium – automate your browser goo.gl/sFi6H2 iamjanhak
35. Ideas that take testing further  TDD – Red Green Refactor!  BDD – describe behavior in plain English  Integration testing – testing larger parts of the system  Continuous Integration – run your tests all the time goo.gl/sFi6H2 iamjanhak
36. Resources Getting Started Testing Ned Batchelder Test-Driven Development with Python Harry J. W. Percival
37. Testing is complicated, but important and rewarding Summary
38. Testing is:  Challenging  Important  Craft  Stimulating iamjanhak
39. Questions? iamjanhak