Testing Without Assertions

At Buzzfeed, analytics is extremely important as a feedback loop to the content creators. They use a client side JS library to send these analytics events which is well tested; however, in early 2016 there were a series of data outages that prompted the team to revisit their testing strategy. Testing network requests to downstream systems can be tough. There’s only so far that Selenium tests can reach and you can often end up with tests that offer you false confidence in the complete system. In this presentation, Ian Feather will describe how they approached this problem and ultimately the solution they chose: using Sauce Labs to run continuous cross-platform tests against our production systems. Ian will share a real-life case study on how BuzzFeed knows their analytics are working correctly, a general technique for testing network requests to downstream systems, and more.

Transcript

1. TESTING WITHOUT ASSERTIONS IAN FEATHER
3. 2016 2017 ANALYTICS OUTAGES
4. WHY ANALYTICS MATTERS TO BUZZFEED SECTION 1
5. CMS Publish Store collect CREATE
6. CMS Publish Store collect CREATE learn
7. ANALYTICS OUTAGES 2016 2017
8. PIXIEDUST
9. WHY IS IT SO HARD TO TEST ANALYTICS? SECTION 2
10. Feature: Quizzes Scenario: Filters When I go to the quiz page
11. Feature: Quizzes Scenario: Filters When I go to the quiz page And I click on the filter
12. Feature: Quizzes Scenario: Filters When I go to the quiz page And I click on the filter Then I see the new results
13. Feature: Quizzes Scenario: Analytics When I go to the quiz page And I click on the filter Then ???
14. Feature: Quizzes Scenario: Analytics When I go to the quiz page And I click on the filter Then I see the event locally
15. flush(events) { } buzzfeed-analytics.js
16. flush(events) { } send(events); buzzfeed-analytics.js
17. flush(events) { } send(events); if (mode == ‘test’) {
 
 } buzzfeed-analytics.js
18. flush(events) { } send(events); if (mode == ‘test’) {
 
 } store(events); buzzfeed-analytics.js
19. TEST ASSERT Session storage
20. THE TESTS WERE REALLY FLAKEY
21. BUGS STILL REACHED PRODUCTION
22. BUGS STILL APPEARED IN PRODUCTION
23. RETHINKING THE PROBLEM SECTION 3
24. NON-FLAKEY E2E TESTS ASSERTING AGAINST THE NETWORK
25. TESTS EXIST TO PROTECT PRODUCTION “
26. GETTING A DIFFERENT PERSPECTIVE
27. TESTS EXIST TO PROTECT PRODUCTION “
28. TESTS EXIST TO GIVE US CONFIDENCE “
29. PRE-PRODUCTION TESTING PRODUCTION MONITORING PRODUCTION TESTING VS VS
30. OUR SOLUTION SECTION 4
31. TEST ASSERT Session storage
32. TEST
33. TEST
34. TEST
35. TEST
36. TEST
37. Pixiedust API TEST
38. Pixiedust API nsq TEST
39. Pixiedust API nsq TEST QR QR QR
40. Pixiedust API nsq TEST QR QR QR DB
41. Pixiedust API nsq TEST QR QR QR DB
42. Pixiedust API nsq TEST QR QR QR DB QR
43. Pixiedust API nsq TEST QR
44. Pixiedust API nsq TEST QR
45. Pixiedust API nsq TEST ASSERT QR
46. Pixiedust API nsq TEST Dashboard ASSERT QR
48. Feature: Quizzes Scenario: Analytics When I go to the quiz page And I click on the filter Then I see the event in SessionStorage
49. Feature: Quizzes Scenario: Analytics When I go to the quiz page And I click on the filter
50. Pixiedust API nsq TEST Dashboard ASSERT QR
51. Pixiedust API nsq TEST Dashboard ASSERT CRON QR
52. Pixiedust API nsq TEST Dashboard ASSERT CRON QR
53. Pixiedust API nsq TEST Dashboard ASSERT CRON QR
54. SLACK ALARMS
55. Pixiedust API nsq TEST Dashboard ASSERT CRON QR
56. LESSONS LEARNED ALONG THE WAY SECTION 5
57. WE FOUND BUGS WE DIDN’T KNOW ABOUT
58. TESTING IN PRODUCTION HAS INTERESTING SIDE EFFECTS
61. TESTING IN PRODUCTION BROUGHT A LOT OF CONFIDENCE
62. WE HAVE SEEN ALMOST ZERO REGRESSIONS
63. SOMETIMES YOU HAVE TO THROW OUT EVERYTHING YOU KNOW
64. THANKS! @IANFEATHER WWW.BUZZFEED.COM/JOBS