Slide 1

Slide 1 text

Sneeze and Pocket Change Functional Test Automation Reporting

Slide 2

Slide 2 text

Automation Framework ● SUT interfaces ● Business data model ● Dataset management

Slide 3

Slide 3 text

Nose ● Test discovery ● Fixtures ● Data driven framework plugin ● Rudimentary reporting

Slide 4

Slide 4 text

Hudson ● Launch tests ● Scheduling runs ● Build process integration ● Parsed xUnit reporting

Slide 5

Slide 5 text

Poor Result Visibility ● Job runs:test suites not 1:1 ○ parameterized jobs ○ cancelled/incomplete runs

Slide 6

Slide 6 text

Poor Result Visibility ● Poor usability of dumped xUnit results ○ confusing drill down ○ flat text ○ doesn’t capture all data ○ not searchable ○ poorly archived

Slide 7

Slide 7 text

Poor Result Visibility ● Results not responsive ○ nothing until job completes ○ cancelled job gives no output

Slide 8

Slide 8 text

Sneeze + Pocket Change

Slide 9

Slide 9 text

Sneeze + Pocket Change ● Nose plugin - test state/logging > DB ● Web frontend - browse/filter test cycles ● Collect test metadata (environment, etc) ● Workflow integration - JIRA ● Future features ○ Run from web ○ Rich result analysis ○ More integrations ○ Multimedia logging (screenshots!)

Slide 10

Slide 10 text

Nose Sneeze Pocket Framework/ Tests Logging Reporting DB Pocket Change

Slide 11

Slide 11 text

Examples! import logging, time log = logging.getLogger('repro_steps') def validate(operation, result, message): log.info('verifying {}'.format(operation)) assert result, message def test_abctastic(): '''Doing some math with a b and c.''' a, b, c = 1, 2, 3 log.info('a, b, c = 1, 2, 3') validate('a + b == c', a + b == c, 'expected {}, found {}'.format(a + b, c)) validate('c / b == 1.5', c / b == 1.5, 'expected 1.5, found {}'.format(c / b)) def test_sleepy(): '''A test that logs every .1 second for roughly a minute.''' for i in xrange(600): log.info('Message {}'.format(i)) time.sleep(.1) Some setup code

Slide 12

Slide 12 text

Run It! nosetests --test-cycle-name ‘demo cycle’ --test-cycle-description ‘some tests for demoing sneeze’ test_demo_1.py

Slide 13

Slide 13 text

Get Involved! ● repos (click through to github): http://developers. nytimes.com/opensource/sneeze/ ● Open to feature requests and bugs ● Pull requests welcome