Functional Test Automation Reporting with Sneeze & Pocket Change

Software development engineer Silas Ray gave this talk to the NYC Python meetup.

• Watch video from Silas's talk.
• Read more about the Sneeze and Pocket Change libraries on the Open blog.
• Get the project code.

Transcript

1. Sneeze and Pocket Change Functional Test Automation Reporting
2. Automation Framework • SUT interfaces • Business data model • Dataset management
3. Nose • Test discovery • Fixtures • Data driven framework plugin • Rudimentary reporting
4. Hudson • Launch tests • Scheduling runs • Build process integration • Parsed xUnit reporting
5. Poor Result Visibility • Job runs:test suites not 1:1 ◦ parameterized jobs ◦ cancelled/incomplete runs
6. Poor Result Visibility • Poor usability of dumped xUnit results ◦ confusing drill down ◦ flat text ◦ doesn’t capture all data ◦ not searchable ◦ poorly archived
7. Poor Result Visibility • Results not responsive ◦ nothing until job completes ◦ cancelled job gives no output
8. Sneeze + Pocket Change
9. Sneeze + Pocket Change • Nose plugin - test state/logging > DB • Web frontend - browse/filter test cycles • Collect test metadata (environment, etc) • Workflow integration - JIRA • Future features ◦ Run from web ◦ Rich result analysis ◦ More integrations ◦ Multimedia logging (screenshots!)
10. Nose Sneeze Pocket Framework/ Tests Logging Reporting DB Pocket Change
11. Examples! import logging, time log = logging.getLogger('repro_steps') def validate(operation, result, message): log.info('verifying {}'.format(operation)) assert result, message def test_abctastic(): '''Doing some math with a b and c.''' a, b, c = 1, 2, 3 log.info('a, b, c = 1, 2, 3') validate('a + b == c', a + b == c, 'expected {}, found {}'.format(a + b, c)) validate('c / b == 1.5', c / b == 1.5, 'expected 1.5, found {}'.format(c / b)) def test_sleepy(): '''A test that logs every .1 second for roughly a minute.''' for i in xrange(600): log.info('Message {}'.format(i)) time.sleep(.1) Some setup code
12. Run It! nosetests --test-cycle-name ‘demo cycle’ --test-cycle-description ‘some tests for demoing sneeze’ test_demo_1.py
13. Get Involved! • repos (click through to github): http://developers. nytimes.com/opensource/sneeze/ • Open to feature requests and bugs • Pull requests welcome