Testing Without Assertions

Slide 1

Slide 1 text

TESTING WITHOUT ASSERTIONS IAN FEATHER

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

2016 2017 ANALYTICS OUTAGES

Slide 4

Slide 4 text

WHY ANALYTICS MATTERS TO BUZZFEED SECTION 1

Slide 5

Slide 5 text

CMS Publish Store collect CREATE

Slide 6

Slide 6 text

CMS Publish Store collect CREATE learn

Slide 7

Slide 7 text

ANALYTICS OUTAGES 2016 2017

Slide 8

Slide 8 text

PIXIEDUST

Slide 9

Slide 9 text

WHY IS IT SO HARD TO TEST ANALYTICS? SECTION 2

Slide 10

Slide 10 text

Feature: Quizzes Scenario: Filters When I go to the quiz page

Slide 11

Slide 11 text

Feature: Quizzes Scenario: Filters When I go to the quiz page And I click on the filter

Slide 12

Slide 12 text

Feature: Quizzes Scenario: Filters When I go to the quiz page And I click on the filter Then I see the new results

Slide 13

Slide 13 text

Feature: Quizzes Scenario: Analytics When I go to the quiz page And I click on the filter Then ???

Slide 14

Slide 14 text

Feature: Quizzes Scenario: Analytics When I go to the quiz page And I click on the filter Then I see the event locally

Slide 15

Slide 15 text

flush(events) { } buzzfeed-analytics.js

Slide 16

Slide 16 text

flush(events) { } send(events); buzzfeed-analytics.js

Slide 17

Slide 17 text

flush(events) { } send(events); if (mode == ‘test’) {    } buzzfeed-analytics.js

Slide 18

Slide 18 text

flush(events) { } send(events); if (mode == ‘test’) {    } store(events); buzzfeed-analytics.js

Slide 19

Slide 19 text

TEST ASSERT Session storage

Slide 20

Slide 20 text

THE TESTS WERE REALLY FLAKEY

Slide 21

Slide 21 text

BUGS STILL REACHED PRODUCTION

Slide 22

Slide 22 text

BUGS STILL APPEARED IN PRODUCTION

Slide 23

Slide 23 text

RETHINKING THE PROBLEM SECTION 3

Slide 24

Slide 24 text

NON-FLAKEY E2E TESTS ASSERTING AGAINST THE NETWORK

Slide 25

Slide 25 text

TESTS EXIST TO PROTECT PRODUCTION “

Slide 26

Slide 26 text

GETTING A DIFFERENT PERSPECTIVE

Slide 27

Slide 27 text

TESTS EXIST TO PROTECT PRODUCTION “

Slide 28

Slide 28 text

TESTS EXIST TO GIVE US CONFIDENCE “

Slide 29

Slide 29 text

PRE-PRODUCTION TESTING PRODUCTION MONITORING PRODUCTION TESTING VS VS

Slide 30

Slide 30 text

OUR SOLUTION SECTION 4

Slide 31

Slide 31 text

TEST ASSERT Session storage

Slide 32

Slide 32 text

TEST

Slide 33

Slide 33 text

TEST

Slide 34

Slide 34 text

TEST

Slide 35

Slide 35 text

TEST

Slide 36

Slide 36 text

TEST

Slide 37

Slide 37 text

Pixiedust API TEST

Slide 38

Slide 38 text

Pixiedust API nsq TEST

Slide 39

Slide 39 text

Pixiedust API nsq TEST QR QR QR

Slide 40

Slide 40 text

Pixiedust API nsq TEST QR QR QR DB

Slide 41

Slide 41 text

Pixiedust API nsq TEST QR QR QR DB

Slide 42

Slide 42 text

Pixiedust API nsq TEST QR QR QR DB QR

Slide 43

Slide 43 text

Pixiedust API nsq TEST QR

Slide 44

Slide 44 text

Pixiedust API nsq TEST QR

Slide 45

Slide 45 text

Pixiedust API nsq TEST ASSERT QR

Slide 46

Slide 46 text

Pixiedust API nsq TEST Dashboard ASSERT QR

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

Feature: Quizzes Scenario: Analytics When I go to the quiz page And I click on the filter Then I see the event in SessionStorage

Slide 49

Slide 49 text

Feature: Quizzes Scenario: Analytics When I go to the quiz page And I click on the filter

Slide 50

Slide 50 text

Pixiedust API nsq TEST Dashboard ASSERT QR

Slide 51

Slide 51 text

Pixiedust API nsq TEST Dashboard ASSERT CRON QR

Slide 52

Slide 52 text

Pixiedust API nsq TEST Dashboard ASSERT CRON QR

Slide 53

Slide 53 text

Pixiedust API nsq TEST Dashboard ASSERT CRON QR

Slide 54

Slide 54 text

SLACK ALARMS

Slide 55

Slide 55 text

Pixiedust API nsq TEST Dashboard ASSERT CRON QR

Slide 56

Slide 56 text

LESSONS LEARNED ALONG THE WAY SECTION 5

Slide 57

Slide 57 text

WE FOUND BUGS WE DIDN’T KNOW ABOUT

Slide 58

Slide 58 text

TESTING IN PRODUCTION HAS INTERESTING SIDE EFFECTS

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

TESTING IN PRODUCTION BROUGHT A LOT OF CONFIDENCE

Slide 62

Slide 62 text

WE HAVE SEEN ALMOST ZERO REGRESSIONS

Slide 63

Slide 63 text

SOMETIMES YOU HAVE TO THROW OUT EVERYTHING YOU KNOW

Slide 64

Slide 64 text

THANKS! @IANFEATHER WWW.BUZZFEED.COM/JOBS