Thomi Richards: A Testing Talk

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Thomi Richards:
A Testing Talk
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2014 - Saturday, 13 Sep 2014 - Track 2
http://kiwi.pycon.org/

**Audience level**

Intermediate

**Description**

This talk will contain my opinionated views on several topics, including, but not limited to:

- Which areas of testing are well catered for, and which are not?
- What sources of bugs are frequent, and very hard to test?
- Should we perhaps start talking about unit tests less, and start talking about other forms of testing more?

**Abstract**

In my day job I spend a lot of time looking at projects that are part of Ubuntu, and trying to find out why they’re being released with bugs. This in turn leads to a lot of dissection of test suites, and a lot of discussion with my colleagues around tests, names of tests, why some tests are better than others, where the common gaps in test coverage are, how applications should be tested, where, when, and why certain test suites should be run, and other subjects too boring to mention.

Having done this for several years, I've started to form a few troubling thoughts about the state of automated testing in software development, and in python specifically. When taken together, these can start to form a rough and ready 'testing philosophy' - a way of looking at code and, by analysing it's structure and visibility, start to find gaps in it's test coverage.

This talk contains no silver bullets - no magical solutions, but does (hopefully) contain some interesting questions.

**YouTube**

https://www.youtube.com/watch?v=Mydv_EOuezs

Transcript

1. A TESTING TALK A TESTING TALK A Testing Talk http://127.0.0.1:8000/?print-pdf 1 of 34 13/09/14 12:20
2. A Testing Talk http://127.0.0.1:8000/?print-pdf 2 of 34 13/09/14 12:20
3. FRUSTRATION FRUSTRATION A Testing Talk http://127.0.0.1:8000/?print-pdf 3 of 34 13/09/14 12:20
4. DISSATISFACTION DISSATISFACTION A Testing Talk http://127.0.0.1:8000/?print-pdf 4 of 34 13/09/14 12:20
5. I HAVE NO SOLUTION I HAVE NO SOLUTION A Testing Talk http://127.0.0.1:8000/?print-pdf 5 of 34 13/09/14 12:20
6. AN ALTERNATIVE APPROACH AN ALTERNATIVE APPROACH A Testing Talk http://127.0.0.1:8000/?print-pdf 6 of 34 13/09/14 12:20
7. SOURCES OF PROBLEMS: ALGORITHMIC CORRECTNESS SOURCES OF PROBLEMS: ALGORITHMIC CORRECTNESS A Testing Talk http://127.0.0.1:8000/?print-pdf 7 of 34 13/09/14 12:20
8. def isPrime(n): for i in range(2, int(n**0.5) + 1): if n % i == 0: return False return True A Testing Talk http://127.0.0.1:8000/?print-pdf 8 of 34 13/09/14 12:20
9. def maybe_handle_error(e): if e.code == 302: raise Retry() elif e.code == 200: return raise Abort() A Testing Talk http://127.0.0.1:8000/?print-pdf 9 of 34 13/09/14 12:20
10. from testtools import TestCase class PrimeTests(TestCase): def test_one_is_not_a_prime(self): self.assertFalse(isPrime(1)) def test_low_primes(self): for p in (2, 3, 5, 7, 11): self.assertTrue(isPrime(p)) # TODO: test -ves, 0, bad input etc. A Testing Talk http://127.0.0.1:8000/?print-pdf 10 of 34 13/09/14 12:20
11. STATE STATE A Testing Talk http://127.0.0.1:8000/?print-pdf 11 of 34 13/09/14 12:20
12. FURTHER READING FURTHER READING "Boundaries" - Gary Bernhardt, PyCon US 2013 A Testing Talk http://127.0.0.1:8000/?print-pdf 12 of 34 13/09/14 12:20
13. MOSTLY-SOLVED PROBLEM MOSTLY-SOLVED PROBLEM Source of bugs: Low A Testing Talk http://127.0.0.1:8000/?print-pdf 13 of 34 13/09/14 12:20
14. API COMPATIBILITY API COMPATIBILITY A Testing Talk http://127.0.0.1:8000/?print-pdf 14 of 34 13/09/14 12:20
15. INTERNAL API INTERNAL API A Testing Talk http://127.0.0.1:8000/?print-pdf 15 of 34 13/09/14 12:20
16. INTERNAL API INTERNAL API def function_a(): result = function_b("Thomi Richards", "The Hulk", "Mowgli") def function_b(name, alter_ego, childhood_nickname): return " A.K.A. ".join((name, alter_ego, childhood_nickname)) A Testing Talk http://127.0.0.1:8000/?print-pdf 16 of 34 13/09/14 12:20
17. INTERNAL API INTERNAL API def function_a(): result = function_b("Thomi Richards", "The Hulk", "Mowgli") def function_b(name, alter_ego): return " A.K.A. ".join((name, alter_ego)) A Testing Talk http://127.0.0.1:8000/?print-pdf 17 of 34 13/09/14 12:20
18. NAMING NAMING A Testing Talk http://127.0.0.1:8000/?print-pdf 18 of 34 13/09/14 12:20
19. ADOPT A NAMING CONVENTION ADOPT A NAMING CONVENTION Public API Internal API Private Code A Testing Talk http://127.0.0.1:8000/?print-pdf 19 of 34 13/09/14 12:20
20. PUBLIC API PUBLIC API Code that other people might call (if you're a library vendor): autopilot.testcase.AutopilotTestCase A Testing Talk http://127.0.0.1:8000/?print-pdf 20 of 34 13/09/14 12:20
21. INTERNAL API INTERNAL API Code that is used elsewhere in your project: autopilot._debug.VerboseDebugProfile A Testing Talk http://127.0.0.1:8000/?print-pdf 21 of 34 13/09/14 12:20
22. PRIVATE CODE PRIVATE CODE Code that is only used within your module: autopilot._glib._import_gdk_modules A Testing Talk http://127.0.0.1:8000/?print-pdf 22 of 34 13/09/14 12:20
23. MODULE DESIGN MODULE DESIGN A Testing Talk http://127.0.0.1:8000/?print-pdf 23 of 34 13/09/14 12:20
24. WHAT ABOUT TESTING? WHAT ABOUT TESTING? Can we use function annotations? A Testing Talk http://127.0.0.1:8000/?print-pdf 24 of 34 13/09/14 12:20
25. Source of Bugs: Medium A Testing Talk http://127.0.0.1:8000/?print-pdf 25 of 34 13/09/14 12:20
26. EXTERNAL API COMPATIBILITY EXTERNAL API COMPATIBILITY But not APIs as you know them... A Testing Talk http://127.0.0.1:8000/?print-pdf 26 of 34 13/09/14 12:20
27. Data read from a file on disk stdout or stderr from some other process Response from a web service A third-party API "Frameworks" (a.k.a bloated APIs) The Python standard library A Testing Talk http://127.0.0.1:8000/?print-pdf 27 of 34 13/09/14 12:20
28. PROTECT YOURSELF! PROTECT YOURSELF! ... by isolating the interaction A Testing Talk http://127.0.0.1:8000/?print-pdf 28 of 34 13/09/14 12:20
29. PROTECT YOURSELF! PROTECT YOURSELF! ... with build-time tests A Testing Talk http://127.0.0.1:8000/?print-pdf 29 of 34 13/09/14 12:20
30. PROTECT YOURSELF! PROTECT YOURSELF! ... with run-time tests A Testing Talk http://127.0.0.1:8000/?print-pdf 30 of 34 13/09/14 12:20
31. Source of Bugs: High A Testing Talk http://127.0.0.1:8000/?print-pdf 31 of 34 13/09/14 12:20
32. FRAMEWORKS FRAMEWORKS A Testing Talk http://127.0.0.1:8000/?print-pdf 32 of 34 13/09/14 12:20
33. A Testing Talk http://127.0.0.1:8000/?print-pdf 33 of 34 13/09/14 12:20
34. QUESTIONS? QUESTIONS? CONTACT ME: CONTACT ME: Twitter: @thomir Email: thomir@gmail.com thomi.richards@canonical.com IRC: thomi on irc.freenode.net A Testing Talk http://127.0.0.1:8000/?print-pdf 34 of 34 13/09/14 12:20