Slide 1

Slide 1 text

Test Anything and Everything dave dash DJANGOCON 2011 1

Slide 2

Slide 2 text

me (the djangonaut) sr web developer @ mozilla helped move mozilla webdev to Django worked on Firefox Add-ons Technical lead for Firefox Input I work on internal libraries, tools, and some smaller sites. i blog @ davedash.com i email @ [email protected] i tweet @davedash 2

Slide 3

Slide 3 text

mozilla webdev pronounced mo-zilla 40+ people 100+ web sites we happily churn out PEP-8 compliant code Django shop since early 2010 we create and support most web sites *.mozilla.com / *.mozilla.org Mozilla Developer Network Firefox Add-ons Firefox Support Mozilla.org and more... actively hiring, if this sort of thing appeals to you 3

Slide 4

Slide 4 text

What we’ll cover philosophy of testing ./manage.py test Writing TestCases Code coverage Testing difficult things in the django test suite • • • • • 4

Slide 5

Slide 5 text

What we won’t cover Systems testing Failure testing In browser-testing Load testing • • • • 5

Slide 6

Slide 6 text

Follow along Etherpad - http://mzl.la/django-test +1 any interesting topics List any testing issues or general questions you might have • • • 6

Slide 7

Slide 7 text

Introduce yourself Name Involvement with Django Testing experience What you want to get out of this morning • • • • 7

Slide 8

Slide 8 text

Agenda I’ll start a fight Testing @ Mozilla Interactive stuff How Django’s testing works How we can make testing better Testing beyond the DB Gotchas 8

Slide 9

Slide 9 text

please... Ask questions. This is 3 hours... I can’t fill it all with slides. Correct me if I’m wrong. • • 9

Slide 10

Slide 10 text

Questions? 10

Slide 11

Slide 11 text

when to test? 11

Slide 12

Slide 12 text

never 12

Slide 13

Slide 13 text

but... you told us to test 13

Slide 14

Slide 14 text

you convinced us it was good 14

Slide 15

Slide 15 text

“Instead of writing tests I try to be extremely careful in coding, and keep the code size small so I continue to understand it.” -maciej ceglowski, pinboard.in 15

Slide 16

Slide 16 text

it all depends 16

Slide 17

Slide 17 text

what am I doing? 17

Slide 18

Slide 18 text

know your app 18

Slide 19

Slide 19 text

but... 19

Slide 20

Slide 20 text

when to test 20

Slide 21

Slide 21 text

when others use your code 21

Slide 22

Slide 22 text

libraries 22

Slide 23

Slide 23 text

projects? 23

Slide 24

Slide 24 text

big sites 24

Slide 25

Slide 25 text

teams 25

Slide 26

Slide 26 text

pull requests 26

Slide 27

Slide 27 text

no tests, no merge 27

Slide 28

Slide 28 text

failed tests... no merge 28

Slide 29

Slide 29 text

Testing Django @ Mozilla 29

Slide 30

Slide 30 text

2009/2010 30

Slide 31

Slide 31 text

Python Revolution at Mozilla 31

Slide 32

Slide 32 text

previously... 32

Slide 33

Slide 33 text

Banging our heads 33

Slide 34

Slide 34 text

PHP apps with no test framework 34

Slide 35

Slide 35 text

Django’s Test framework == Appealing 35

Slide 36

Slide 36 text

today... 36

Slide 37

Slide 37 text

Firefox Add-ons: 2500+ tests 37

Slide 38

Slide 38 text

Firefox Support: 1100 Tests 38

Slide 39

Slide 39 text

Coverage: 90%+ 39

Slide 40

Slide 40 text

Tons of smaller Django sites 40

Slide 41

Slide 41 text

All expected to be well tested 41

Slide 42

Slide 42 text

At least 80-90% 42

Slide 43

Slide 43 text

Some things aren’t worth testing 43

Slide 44

Slide 44 text

What does 5000 tests buy you? 44

Slide 45

Slide 45 text

Problems 45

Slide 46

Slide 46 text

Too many tests per file 46

Slide 47

Slide 47 text

Tests take too long to run 47

Slide 48

Slide 48 text

Failing tests don’t get fixed 48

Slide 49

Slide 49 text

avoid testing external systems 49

Slide 50

Slide 50 text

Vagrant 50

Slide 51

Slide 51 text

vagrant up

Slide 52

Slide 52 text

regular django testing

Slide 53

Slide 53 text

cd ~/fxinput_nohacks ./manage.py test

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

How testing works in Django 55

Slide 56

Slide 56 text

The Test Runner 56

Slide 57

Slide 57 text

Creates Database 57

Slide 58

Slide 58 text

No Data... to start 58

Slide 59

Slide 59 text

Finds tests 59

Slide 60

Slide 60 text

in models.py and tests.py 60

Slide 61

Slide 61 text

runs the tests 61

Slide 62

Slide 62 text

runs non-Class tests 62

Slide 63

Slide 63 text

runs any doctests 63

Slide 64

Slide 64 text

64

Slide 65

Slide 65 text

Class tests 65

Slide 66

Slide 66 text

Class Setup 66

Slide 67

Slide 67 text

for each test... 67

Slide 68

Slide 68 text

Setup 68

Slide 69

Slide 69 text

Load a fixture 69

Slide 70

Slide 70 text

70

Slide 71

Slide 71 text

Run Test 71

Slide 72

Slide 72 text

Teardown 72

Slide 73

Slide 73 text

after all tests in a class... 73

Slide 74

Slide 74 text

class teardown 74

Slide 75

Slide 75 text

75

Slide 76

Slide 76 text

then we get our report card 76

Slide 77

Slide 77 text

“F” means bad 77

Slide 78

Slide 78 text

“.” means good 78

Slide 79

Slide 79 text

“E” doesn’t mean effort 79

Slide 80

Slide 80 text

Now that we know how it works... 80

Slide 81

Slide 81 text

WE CAN HACK IT 81

Slide 82

Slide 82 text

hacking testing 82

Slide 83

Slide 83 text

2,500 tests is a lot 83

Slide 84

Slide 84 text

Multiple Files 84

Slide 85

Slide 85 text

85

Slide 86

Slide 86 text

be messy! 86

Slide 87

Slide 87 text

87

Slide 88

Slide 88 text

test_utils 88

Slide 89

Slide 89 text

be real messy! 89

Slide 90

Slide 90 text

90

Slide 91

Slide 91 text

91

Slide 92

Slide 92 text

92

Slide 93

Slide 93 text

django-nose + test_utils 93

Slide 94

Slide 94 text

organize your mess 94

Slide 95

Slide 95 text

load a set of fixtures once 95

Slide 96

Slide 96 text

run all the test-cases 96

Slide 97

Slide 97 text

DRY... for fixtures 97

Slide 98

Slide 98 text

98 class TestCase1(TestCase): fixtures = ['a'] class TestCase2(TestCase): fixtures = ['a', 'b'] class TestCase3(TestCase): fixtures = ['a'] class TestCase4(TestCase): fixtures = ['a', 'c'] class TestCase5(TestCase): fixtures = ['a'] class TestCase6(TestCase): fixtures = ['a', 'c']

Slide 99

Slide 99 text

django-nose + test_utils 99

Slide 100

Slide 100 text

fixture misers 100

Slide 101

Slide 101 text

model maker 101

Slide 102

Slide 102 text

Useful testing tools 102

Slide 103

Slide 103 text

nose (via django-nose) 103

Slide 104

Slide 104 text

nose finds test 104

Slide 105

Slide 105 text

105

Slide 106

Slide 106 text

it has nice tools 106

Slide 107

Slide 107 text

raise SkipTest 107

Slide 108

Slide 108 text

108

Slide 109

Slide 109 text

eq_(x, y, “x ain’t y”) 109

Slide 110

Slide 110 text

--failed 110

Slide 111

Slide 111 text

111

Slide 112

Slide 112 text

nose has plugins 112

Slide 113

Slide 113 text

113 ................................S....................................................F................................................................ ====================================================================== FAIL: Verify the "next" pagination link appears and directs the user to the ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/dash/Projects/input/reporter/apps/search/tests/test_dashboard.py", line 57, in test_beta_pagination_link eq_(len(pag_link), 1) File "/Users/dash/Projects/input/reporter/vendor/packages/nose/nose/tools.py", line 31, in eq_ assert a == b, msg or "%r != %r" % (a, b) AssertionError: 0 != 1 ---------------------------------------------------------------------- Ran 150 tests in 32.504s FAILED (SKIP=1, failures=1)

Slide 114

Slide 114 text

nicedots 114

Slide 115

Slide 115 text

115 apps/input/tests/test_middleware.py:MiddlewareTests ........... apps/input/tests/test_redirects.py:RedirectTests ... apps/myadmin/tests.py:ViewTestCase ...... apps/search/tests/test_client.py:SearchTest .............. apps/search/tests/test_client.py .. ====================================================================== FAIL: apps/search/tests/test_dashboard.py:TestDashboard.test_beta_pagination_link ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/dash/Projects/input/reporter/apps/search/tests/test_dashboard.py", line 57, in test_beta_pagination_link eq_(len(pag_link), 1) File "/Users/dash/Projects/input/reporter/vendor/packages/nose/nose/tools.py", line 31, in eq_ assert a == b, msg or "%r != %r" % (a, b) AssertionError: 0 != 1 apps/search/tests/test_dashboard.py:TestDashboard .. apps/search/tests/test_dashboard.py:TestHelpers ........ apps/search/tests/test_dashboard.py:TestMobileDashboard . apps/search/tests/test_elastic.py:TestElastic ..

Slide 116

Slide 116 text

nose-progressive 116

Slide 117

Slide 117 text

117

Slide 118

Slide 118 text

do not mix those two 118

Slide 119

Slide 119 text

coverage 119

Slide 120

Slide 120 text

coverage ./manage.py test 120

Slide 121

Slide 121 text

coverage report -m 121

Slide 122

Slide 122 text

122 Name Stmts Miss Cover Missing ---------------------------------------------------------------- apps/search/__init__ 0 0 100% apps/search/client 193 4 98% 96-97, 134-137 apps/search/context_processors 5 0 100% apps/search/cron 13 13 0% 1-24 apps/search/forms 70 2 97% 131, 136 apps/search/helpers 136 1 99% 154 apps/search/models 0 0 100% apps/search/tasks 8 0 100% apps/search/tests/__init__ 26 1 96% 27 apps/search/tests/test_client 75 0 100% apps/search/tests/test_dashboard 105 1 99% 58 apps/search/tests/test_elastic 15 0 100% apps/search/tests/test_views 193 0 100% apps/search/urls 3 0 100% apps/search/utils 10 0 100% apps/search/views 142 1 99% 166 ---------------------------------------------------------------- TOTAL 994 23 98%

Slide 123

Slide 123 text

jenkins! 123

Slide 124

Slide 124 text

formerly hudson 124

Slide 125

Slide 125 text

jenkins.mozilla.org 125

Slide 126

Slide 126 text

runs after each commit 126

Slide 127

Slide 127 text

Testing Everything 127

Slide 128

Slide 128 text

not 100% coverage 128

Slide 129

Slide 129 text

80-90% is okay 129

Slide 130

Slide 130 text

“everything?” 130

Slide 131

Slide 131 text

Test your entire site 131

Slide 132

Slide 132 text

not just your database 132

Slide 133

Slide 133 text

test your search engines 133

Slide 134

Slide 134 text

your weird REST API 134

Slide 135

Slide 135 text

that kind of “everything” 135

Slide 136

Slide 136 text

good coverage on tricky things 136

Slide 137

Slide 137 text

some coverage on everything 137

Slide 138

Slide 138 text

code coverage isn’t everything 138

Slide 139

Slide 139 text

subclass me 139

Slide 140

Slide 140 text

APIs 140

Slide 141

Slide 141 text

Search 141

Slide 142

Slide 142 text

key-value stores 142

Slide 143

Slide 143 text

anything... 143

Slide 144

Slide 144 text

you can do cool things... 144

Slide 145

Slide 145 text

assert! 145

Slide 146

Slide 146 text

results are in order 146

Slide 147

Slide 147 text

147 class RankingTest(SphinxTestCase): """This test assures that we don't regress our rankings.""" fixtures = ('base/users', 'base/addon_1833_yoono', 'base/addon_9825_fastestfox', 'base/addon_5579', 'base/addon_personas-plus', ) def test_twitter(self): """ Search for twitter should yield Yoono before FastestFox since Yoono has "twitter" in it's name field. """ r = query('twitter') eq_(r[0].id, 1833) eq_(r[1].id, 9825) def test_cool(self): """Search for cool should return CoolIris before PersonasPlus.""" r = query('cool') eq_(r[0].slug, 'cooliris') eq_(r[1].slug, 'personas-plus')

Slide 148

Slide 148 text

FireFox == fire fox 148

Slide 149

Slide 149 text

saves headaches 149

Slide 150

Slide 150 text

Mock 150

Slide 151

Slide 151 text

Mock Redis 151

Slide 152

Slide 152 text

152 if not connections: # don't set this repeatedly for alias, backend in settings.REDIS_BACKENDS.items(): _, server, params = parse_backend_uri(backend) try: socket_timeout = float(params.pop('socket_timeout')) except (KeyError, ValueError): socket_timeout = None password = params.pop('password', None) if ':' in server: host, port = server.split(':') try: port = int(port) except (ValueError, TypeError): port = 6379 else: host = 'localhost' port = 6379 connections[alias] = redislib.Redis(host=host, port=port, db=0, password=password, socket_timeout=socket_timeout) def mock_redis(): ret = dict(connections) for key in connections: connections[key] = MockRedis() return ret

Slide 153

Slide 153 text

Redis <> testing requirement 153

Slide 154

Slide 154 text

the mock client 154

Slide 155

Slide 155 text

doesn’t do everything 155

Slide 156

Slide 156 text

does what I’m testing for 156

Slide 157

Slide 157 text

could get built out more... 157

Slide 158

Slide 158 text

Setup & Teardown 158

Slide 159

Slide 159 text

Django’s setup 159

Slide 160

Slide 160 text

Loads fixtures 160

Slide 161

Slide 161 text

Gives you a clean slate 161

Slide 162

Slide 162 text

make clean slates 162

Slide 163

Slide 163 text

search, ldap and others 163

Slide 164

Slide 164 text

destroy the data-store 164

Slide 165

Slide 165 text

load data-fixtures 165

Slide 166

Slide 166 text

wrap this in a subclass of TestCase 166

Slide 167

Slide 167 text

167 class SphinxTestCase(TestCase): """ This test case type can setUp and tearDown the sphinx daemon. Use this when testing any feature that requires sphinx. """ fixtures = ['users.json', 'search/documents.json', 'posts.json', 'questions.json'] @classmethod def setup_class(cls): super(SphinxTestCase, cls).setup_class() if not settings.SPHINX_SEARCHD or not settings.SPHINX_INDEXER: raise SkipTest() os.environ['DJANGO_ENVIRONMENT'] = 'test' if os.path.exists(settings.TEST_SPHINX_PATH): shutil.rmtree(settings.TEST_SPHINX_PATH) os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'data')) os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'log')) os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'etc')) reindex() start_sphinx() time.sleep(1) @classmethod def teardown_class(cls): stop_sphinx() super(SphinxTestCase, cls).teardown_class()

Slide 168

Slide 168 text

your coworkers will thank you 168

Slide 169

Slide 169 text

“But I don’t work on search” 169

Slide 170

Slide 170 text

raise SkipTest 170

Slide 171

Slide 171 text

comes with nose 171

Slide 172

Slide 172 text

you can sniff settings 172

Slide 173

Slide 173 text

Built this for Sphinx Search 173

Slide 174

Slide 174 text

Elastic Search 174

Slide 175

Slide 175 text

LDAP 175

Slide 176

Slide 176 text

Extras 176

Slide 177

Slide 177 text

fixture-magic 177

Slide 178

Slide 178 text

dj custom-dump addon 3615 178

Slide 179

Slide 179 text

the Model Maker pattern 179

Slide 180

Slide 180 text

be careful of dates 180

Slide 181

Slide 181 text

use PDB for snooping 181

Slide 182

Slide 182 text

use ipython+pdb 182

Slide 183

Slide 183 text

alias i from IPython.Shell import IPShellEmbed as IPSh; IPSh(argv='') () > ~/.pdbrc 183

Slide 184

Slide 184 text

I’m looking for new coworkers Mozilla is a fun place to work It’s more exciting than these slides We are a developer friendly python shop We build things and share them • • • • 184

Slide 185

Slide 185 text

Perks desks computers chairs (if you want) work remotely fantastic coworkers <= seriously • • • • • 185