PyConZA 2014: "Practical testing" by Jeremy Thurgood

Automated tests are an important part of any software project, but they're often hard to work with. Good test code should be written with its own set of priorities and is sufficiently different from non-test code that many standard programming practices are unsuitable.

I have spent many years writing (and reading) tests of varying quality and have learned quite a lot from the experience. I hope to share some of that experience and help to remove some of the pain usually associated with writing and maintaining tests.

This talk will cover some of the practical aspects of writing good tests for real code, including:

* Properties of a good test case
* How to manage setup and teardown
* Testing interactions between components
* Some tools and techniques to make testing easier:
-- Helpers
-- Verified fakes
-- Recording doubles

Transcript

1. PRACTICAL TESTING HOW TO WRITE TESTS THAT DON'T SUCK Jeremy Thurgood PyconZA 2014
2. WHAT MAKES A TEST GOOD? Short Long tests are harder to read and more likely to be buggy. Orthogonal A single case or code path should be exercised in each test. Clear Obscurity and complication hides bugs. Documented Every test should explain what it's testing in the docstring.
3. BAD i m p o r t u n i t t e s t f r o m m y c o d e i m p o r t a d d c l a s s T e s t M y C o d e ( u n i t t e s t . T e s t C a s e ) : d e f s e t U p ( s e l f ) : s e l f . o p e r a n d _ 1 = 1 s e l f . o p e r a n d _ 2 = 2 s e l f . o p e r a n d _ 3 = - 2 s e l f . a d d _ r e s u l t _ 1 _ a n d _ 2 = 3 s e l f . a d d _ r e s u l t _ 1 _ a n d _ 3 = - 1 d e f t e s t _ a d d ( s e l f ) : r e s u l t _ 1 = a d d ( s e l f . o p e r a n d _ 1 , s e l f . o p e r a n d _ 2 ) s e l f . a s s e r t E q u a l ( r e s u l t _ 1 , s e l f . a d d _ r e s u l t _ 1 _ a n d _ 2 ) r e s u l t _ 2 = a d d ( s e l f . o p e r a n d _ 1 , s e l f . o p e r a n d _ 3 ) s e l f . a s s e r t E q u a l ( r e s u l t _ 2 , s e l f . a d d _ r e s u l t _ 1 _ a n d _ 3 )
4. GOOD i m p o r t u n i t t e s t f r o m m y c o d e i m p o r t a d d c l a s s T e s t M y C o d e ( u n i t t e s t . T e s t C a s e ) : d e f t e s t _ a d d _ p o s i t i v e ( s e l f ) : " " " A d d i n g t w o p o s i t i v e n u m b e r s r e t u r n s t h e i r s u m . " " " s e l f . a s s e r t E q u a l ( a d d ( 1 , 2 ) , 3 ) d e f t e s t _ a d d _ n e g a t i v e ( s e l f ) : " " " A d d i n g a n e g a t i v e n u m b e r i s e q u i v a l e n t t o s u b t r a c t i o n . " " " s e l f . a s s e r t E q u a l ( a d d ( 1 , - 2 ) , - 1 )
5. TESTS ARE DAMP, NOT DRY Don't Repeat Yourself is a generally useful principle in non- test code, but less applicable to tests. i m p o r t u n i t t e s t f r o m m y c o d e i m p o r t D e f l e c t o r , r e c a l i b r a t e _ d e f l e c t o r c l a s s T e s t M y C o d e ( u n i t t e s t . T e s t C a s e ) : d e f c h e c k _ d e f l e c t o r _ r e c a l i b r a t i o n ( s e l f , p h a s e , d e l t a , e x p e c t e d ) : d e f l e c t o r = D e f l e c t o r ( p h a s e ) r e c a l i b r a t e _ d e f l e c t o r ( d e f l e c t o r , p h a s e _ s h i f t = d e l t a ) s e l f . a s s e r t E q u a l ( d e f l e c t o r . p h a s e , e x p e c t e d ) d e f t e s t _ r e c a l i b r e a t e _ d e f l e c t o r _ p o s i t i v e ( s e l f ) : s e l f . c h e c k _ d e f l e c t o r _ r e c a l i b r a t i o n ( 1 . 0 , 1 . 2 5 , 2 . 2 5 ) d e f t e s t _ r e c a l i b r a t e _ d e f l e c t o r _ n e g a t i v e ( s e l f ) : s e l f . c h e c k _ d e f l e c t o r _ r e c a l i b r a t i o n ( 1 . 0 , - 0 . 7 5 , 0 . 2 5 ) d e f t e s t _ r e c a l i b r a t e _ d e f l e c t o r _ v e r y _ n e g a t i v e ( s e l f ) : s e l f . c h e c k _ d e f l e c t o r _ r e c a l i b r a t i o n ( 1 . 0 , - 1 . 7 5 , - 0 . 7 5 )
6. TESTS ARE DAMP (CTD) i m p o r t u n i t t e s t f r o m m y c o d e i m p o r t D e f l e c t o r , r e c a l i b r a t e _ d e f l e c t o r c l a s s T e s t M y C o d e ( u n i t t e s t . T e s t C a s e ) : d e f t e s t _ r e c a l i b r e a t e _ d e f l e c t o r _ p o s i t i v e ( s e l f ) : d e f l e c t o r = D e f l e c t o r ( 1 . 0 ) r e c a l i b r a t e _ d e f l e c t o r ( d e f l e c t o r , p h a s e _ s h i f t = 1 . 2 5 ) s e l f . a s s e r t E q u a l ( d e f l e c t o r . p h a s e , 2 . 2 5 ) d e f t e s t _ r e c a l i b r a t e _ d e f l e c t o r _ n e g a t i v e ( s e l f ) : d e f l e c t o r = D e f l e c t o r ( 1 . 0 ) r e c a l i b r a t e _ d e f l e c t o r ( d e f l e c t o r , p h a s e _ s h i f t = - 0 . 7 5 ) s e l f . a s s e r t E q u a l ( d e f l e c t o r . p h a s e , 0 . 2 5 ) d e f t e s t _ r e c a l i b r a t e _ d e f l e c t o r _ v e r y _ n e g a t i v e ( s e l f ) : d e f l e c t o r = D e f l e c t o r ( 1 . 0 ) r e c a l i b r a t e _ d e f l e c t o r ( d e f l e c t o r , p h a s e _ s h i f t = - 1 . 7 5 ) s e l f . a s s e r t E q u a l ( d e f l e c t o r . p h a s e , - 0 . 7 5 )
7. WHICH TEST FRAMEWORK? stdlib u n i t t e s t unittest2 py.test nose Twisted trial something else?
8. TEST HELPERS Setup and teardown Monkey patching Matchers (t e s t t o o l s style) More complex helpers u n i t t e s t . T e s t C a s e is much more capable from 2.7 (but I still care about 2.6)
9. SETUP AND TEARDOWN i m p o r t u n i t t e s t f r o m m y c o d e i m p o r t D i a g n o s t i c c l a s s T e s t M y C o d e ( u n i t t e s t . T e s t C a s e ) : _ c l e a n u p _ f u n c s = ( ) # I m m u t a b l e , s o w e d o n ' t m o d i f y t h e c l a s s . d e f a d d _ c l e a n u p ( s e l f , f u n c ) : i f n o t s e l f . _ c l e a n u p _ f u n c s : s e l f . _ c l e a n u p _ f u n c s = [ ] # T o m a k e i t a n i n s t a n c e v a r i a b l e . s e l f . _ c l e a n u p _ f u n c s . a p p e n d ( f u n c ) d e f t e a r D o w n ( s e l f ) : f o r c l e a n u p _ f u n c i n r e v e r s e d ( s e l f . _ c l e a n u p _ f u n c s ) : c l e a n u p _ f u n c ( ) d e f t e s t _ q u e r y _ e n g i n e _ s t a t u s ( s e l f ) : d i a g n o s t i c = D i a g n o s t i c ( ) s e l f . a d d _ c l e a n u p ( d i a g n o s t i c . c l e a n u p ) s e l f . a s s e r t E q u a l ( d i a g n o s t i c . r u n ( ) , " f a i l e d " )
10. MONKEY PATCHING i m p o r t u n i t t e s t f r o m m y c o d e i m p o r t D i a g n o s t i c c l a s s T e s t M y C o d e ( u n i t t e s t . T e s t C a s e ) : _ p a t c h e s = ( ) # I m m u t a b l e , s o w e d o n ' t a c c i d e n t a l l y m o d i f y t h e c l a s s . d e f m o n k e y _ p a t c h ( s e l f , o b j , n a m e , v a l u e ) : i f n o t s e l f . _ p a t c h e s : s e l f . _ p a t c h e s = [ ] # T o m a k e i t a n i n s t a n c e v a r i a b l e . s e l f . _ p a t c h e s . a p p e n d ( ( o b j , n a m e , g e t a t t r ( o b j , n a m e ) ) ) s e t a t t r ( o b j , n a m e , v a l u e ) d e f t e a r D o w n ( s e l f ) : f o r o b j , n a m e , v a l u e i n r e v e r s e d ( s e l f . _ p a t c h e s ) : s e t a t t r ( o b j , n a m e , v a l u e ) d e f t e s t _ q u e r y _ e n g i n e _ s t a t u s ( s e l f ) : s e l f . a s s e r t E q u a l ( D i a g n o s t i c ( ) . r u n ( ) , " f a i l e d " ) s e l f . m o n k e y _ p a t c h ( D i a g n o s t i c , " r u n " , l a m b d a s : " p a s s e d " ) s e l f . a s s e r t E q u a l ( D i a g n o s t i c ( ) . r u n ( ) , " p a s s e d " )
11. MATCHERS (TESTTOOLS) i m p o r t u n i t t e s t f r o m t e s t t o o l s i m p o r t m a t c h e r s , a s s e r t i o n s f r o m m y c o d e i m p o r t D e f l e c t o r c l a s s P h a s e W i t h i n R a n g e ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , m i n i m u m , m a x i m u m ) : s e l f . _ r a n g e = ( m i n i m u m , m a x i m u m ) d e f _ _ s t r _ _ ( s e l f ) : r e t u r n " P h a s e W i t h i n R a n g e ( % s , % s ) " % s e l f . _ r a n g e d e f m a t c h ( s e l f , o b s e r v e d ) : i f n o t ( s e l f . _ r a n g e [ 0 ] < = o b s e r v e d . p h a s e < = s e l f . _ r a n g e [ 1 ] ) : r e t u r n m a t c h e r s . M i s m a t c h ( " E x p e c t e d v a l u e i n ( % s , % s ) , g o t % s . " % ( s e l f . _ m i n , s e l f . _ m a x , o b s e r v e d . p h a s e ) ) c l a s s T e s t M y C o d e ( u n i t t e s t . T e s t C a s e ) : d e f t e s t _ r e c a l i b r a t e _ d e f l e c t o r ( s e l f ) : a s s e r t i o n s . a s s e r t _ t h a t ( D e f l e c t o r ( 1 . 2 ) , P h a s e W i t h i n R a n g e ( 1 . 0 , 2 . 0 ) )
12. MORE COMPLEX HELPERS f r o m m y c o d e i m p o r t D e f l e c t o r c l a s s D e f l e c t o r H e l p e r ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , d e f a u l t _ p h a s e = 1 . 0 ) : s e l f . _ d e f a u l t _ p h a s e = d e f a u l t _ p h a s e s e l f . _ d e f l e c t o r s = [ ] d e f c l e a n u p ( s e l f ) : # N o t h i n g t o d o h e r e , b u t o t h e r t h i n g s m i g h t h a v e c l e a n u p . p a s s d e f g e t _ d e f l e c t o r ( s e l f , p h a s e = N o n e ) : i f p h a s e i s N o n e : p h a s e = s e l f . _ d e f a u l t _ p h a s e d e f l e c t o r = D e f l e c t o r ( p h a s e ) s e l f . _ d e f l e c t o r s . a p p e n d ( d e f l e c t o r ) r e t u r n d e f l e c t o r
13. MORE COMPLEX HELPERS (USAGE) i m p o r t u n i t t e s t f r o m t e s t s . d e f l e c t o r _ h e l p e r i m p o r t D e f l e c t o r H e l p e r c l a s s T e s t M y C o d e ( u n i t t e s t . T e s t C a s e ) : d e f s e t U p ( s e l f ) : s e l f . d e f l e c t o r _ h e l p e r = D e f l e c t o r H e l p e r ( ) s e l f . a d d C l e a n u p ( s e l f . d e f l e c t o r _ h e l p e r . c l e a n u p ) d e f t e s t _ r e c a l i b r a t e _ d e f l e c t o r ( s e l f ) : d e f l e c t o r = s e l f . d e f l e c t o r _ h e l p e r . g e t _ d e f l e c t o r ( ) s e l f . a s s e r t E q u a l ( d e f l e c t o r . p h a s e , 1 . 0 )
14. TEST DOUBLES Some terminology Dummy: placeholder Stub: canned responses Mock: programmable behaviour Fake: alternate implementation Recording double: record and replay
15. DUMMY i m p o r t u n i t t e s t f r o m m y c o d e i m p o r t D e f l e c t o r c l a s s T e s t M y C o d e ( u n i t t e s t . T e s t C a s e ) : d e f t e s t _ c r e a t e _ d e f l e c t o r _ N o n e ( s e l f ) : d e f l e c t o r = D e f l e c t o r ( N o n e ) s e l f . a s s e r t E q u a l ( d e f l e c t o r . p h a s e , N o n e ) d e f t e s t _ c r e a t e _ d e f l e c t o r _ s t r i n g ( s e l f ) : d e f l e c t o r = D e f l e c t o r ( " d u m m y " ) s e l f . a s s e r t E q u a l ( d e f l e c t o r . p h a s e , " d u m m y " ) d e f t e s t _ c r e a t e _ d e f l e c t o r _ o b j e c t ( s e l f ) : p h a s e = o b j e c t ( ) d e f l e c t o r = D e f l e c t o r ( p h a s e ) s e l f . a s s e r t E q u a l ( d e f l e c t o r . p h a s e , p h a s e )
16. STUB i m p o r t u n i t t e s t f r o m m y c o d e i m p o r t q u e r y _ e n g i n e _ s t a t u s c l a s s E n g i n e S t u b ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , s t a t u s ) : s e l f . s t a t u s = s t a t u s c l a s s T e s t M y C o d e ( u n i t t e s t . T e s t C a s e ) : d e f t e s t _ q u e r y _ e n g i n e _ s t a t u s ( s e l f ) : e n g i n e = E n g i n e S t u b ( s t a t u s = " h a p p y " ) s e l f . a s s e r t E q u a l ( q u e r y _ e n g i n e _ s t a t u s ( e n g i n e ) , " h a p p y " )
17. MOCK c l a s s D e f l e c t o r ( o b j e c t ) : d e f s h i f t _ p h a s e ( s e l f , d e l t a ) : # M u m b l e m u m b l e t e c h n o b a b b l e p a s s d e f r e c a l i b r a t e _ d e f l e c t o r ( d e f l e c t o r , p h a s e _ s h i f t ) : d e f l e c t o r . s h i f t _ p h a s e ( p h a s e _ s h i f t ) i m p o r t u n i t t e s t i m p o r t m o c k f r o m m y c o d e i m p o r t r e c a l i b r a t e _ d e f l e c t o r c l a s s T e s t M y C o d e ( u n i t t e s t . T e s t C a s e ) : d e f t e s t _ r e c a l i b r a t e _ d e f l e c t o r ( s e l f ) : m o c k _ d e f l e c t o r = m o c k . M o c k ( ) r e c a l i b r a t e _ d e f l e c t o r ( m o c k _ d e f l e c t o r , p h a s e _ s h i f t = 1 . 2 ) m o c k _ d e f l e c t o r . s h i f t _ p h a s e . a s s e r t _ c a l l e d _ w i t h ( 1 . 2 )
18. FAKE i m p o r t u n i t t e s t f r o m m y c o d e i m p o r t r e c a l i b r a t e _ d e f l e c t o r c l a s s F a k e D e f l e c t o r ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , p h a s e ) : s e l f . p h a s e = p h a s e d e f s h i f t _ p h a s e ( s e l f , d e l t a ) : s e l f . p h a s e + = d e l t a c l a s s T e s t M y C o d e ( u n i t t e s t . T e s t C a s e ) : d e f t e s t _ r e c a l i b r a t e _ d e f l e c t o r ( s e l f ) : d e f l e c t o r = F a k e D e f l e c t o r ( 1 . 0 ) r e c a l i b r a t e _ d e f l e c t o r ( d e f l e c t o r , p h a s e _ s h i f t = 1 . 2 ) s e l f . a s s e r t E q u a l ( d e f l e c t o r . p h a s e , 2 . 2 )
19. VERIFIED FAKE Fake implementation tested alongside real implementation. i m p o r t u n i t t e s t f r o m m y c o d e i m p o r t D e f l e c t o r , r e c a l i b r a t e _ d e f l e c t o r f r o m f a k e _ d e f l e c t o r i m p o r t F a k e D e f l e c t o r c l a s s M y C o d e T e s t M i x i n ( o b j e c t ) : d e f g e t _ d e f l e c t o r ( s e l f , p h a s e ) : r a i s e N o t I m p l e m e n t e d E r r o r ( ) d e f t e s t _ r e c a l i b r a t e _ d e f l e c t o r ( s e l f ) : d e f l e c t o r = s e l f . g e t _ d e f l e c t o r ( 1 . 0 ) r e c a l i b r a t e _ d e f l e c t o r ( d e f l e c t o r , p h a s e _ s h i f t = 1 . 2 ) s e l f . a s s e r t E q u a l ( d e f l e c t o r . p h a s e , 2 . 2 ) c l a s s T e s t M y C o d e F a k e ( u n i t t e s t . T e s t C a s e , M y C o d e T e s t M i x i n ) : d e f g e t _ d e f l e c t o r ( s e l f , p h a s e ) : r e t u r n F a k e D e f l e c t o r ( p h a s e ) c l a s s T e s t M y C o d e R e a l ( u n i t t e s t . T e s t C a s e , M y C o d e T e s t M i x i n ) : d e f g e t _ d e f l e c t o r ( s e l f , p h a s e ) : r e t u r n D e f l e c t o r ( p h a s e )
20. RECORDING DOUBLE Record mode Make real calls to real external systems Record requests and responses Playback mode Match requests to stored data Return stored responses (See real example.)
21. THE END OF THE SLIDES Now you get to ask me hard questions. (Or I can show some more code.)