Testing Without Assertions

Testing Without Assertions

At Buzzfeed, analytics is extremely important as a feedback loop to the content creators. They use a client side JS library to send these analytics events which is well tested; however, in early 2016 there were a series of data outages that prompted the team to revisit their testing strategy. Testing network requests to downstream systems can be tough. There’s only so far that Selenium tests can reach and you can often end up with tests that offer you false confidence in the complete system. In this presentation, Ian Feather will describe how they approached this problem and ultimately the solution they chose: using Sauce Labs to run continuous cross-platform tests against our production systems. Ian will share a real-life case study on how BuzzFeed knows their analytics are working correctly, a general technique for testing network requests to downstream systems, and more.

649e3d33ce29a5e6bb4ff3025c6aaffa?s=128

Ianfeather

March 01, 2018
Tweet

Transcript

  1. TESTING WITHOUT ASSERTIONS IAN FEATHER

  2. None
  3. 2016 2017 ANALYTICS OUTAGES

  4. WHY ANALYTICS MATTERS TO BUZZFEED SECTION 1

  5. CMS Publish Store collect CREATE

  6. CMS Publish Store collect CREATE learn

  7. ANALYTICS OUTAGES 2016 2017

  8. PIXIEDUST

  9. WHY IS IT SO HARD TO TEST ANALYTICS? SECTION 2

  10. Feature: Quizzes Scenario: Filters When I go to the quiz

    page
  11. Feature: Quizzes Scenario: Filters When I go to the quiz

    page And I click on the filter
  12. Feature: Quizzes Scenario: Filters When I go to the quiz

    page And I click on the filter Then I see the new results
  13. Feature: Quizzes Scenario: Analytics When I go to the quiz

    page And I click on the filter Then ???
  14. Feature: Quizzes Scenario: Analytics When I go to the quiz

    page And I click on the filter Then I see the event locally
  15. flush(events) { } buzzfeed-analytics.js

  16. flush(events) { } send(events); buzzfeed-analytics.js

  17. flush(events) { } send(events); if (mode == ‘test’) {
 


    } buzzfeed-analytics.js
  18. flush(events) { } send(events); if (mode == ‘test’) {
 


    } store(events); buzzfeed-analytics.js
  19. TEST ASSERT Session storage

  20. THE TESTS WERE REALLY FLAKEY

  21. BUGS STILL REACHED PRODUCTION

  22. BUGS STILL APPEARED IN PRODUCTION

  23. RETHINKING THE PROBLEM SECTION 3

  24. NON-FLAKEY E2E TESTS ASSERTING AGAINST THE NETWORK

  25. TESTS EXIST TO PROTECT PRODUCTION “

  26. GETTING A DIFFERENT PERSPECTIVE

  27. TESTS EXIST TO PROTECT PRODUCTION “

  28. TESTS EXIST TO GIVE US CONFIDENCE “

  29. PRE-PRODUCTION TESTING PRODUCTION MONITORING PRODUCTION TESTING VS VS

  30. OUR SOLUTION SECTION 4

  31. TEST ASSERT Session storage

  32. TEST

  33. TEST

  34. TEST

  35. TEST

  36. TEST

  37. Pixiedust API TEST

  38. Pixiedust API nsq TEST

  39. Pixiedust API nsq TEST QR QR QR

  40. Pixiedust API nsq TEST QR QR QR DB

  41. Pixiedust API nsq TEST QR QR QR DB

  42. Pixiedust API nsq TEST QR QR QR DB QR

  43. Pixiedust API nsq TEST QR

  44. Pixiedust API nsq TEST QR

  45. Pixiedust API nsq TEST ASSERT QR

  46. Pixiedust API nsq TEST Dashboard ASSERT QR

  47. None
  48. Feature: Quizzes Scenario: Analytics When I go to the quiz

    page And I click on the filter Then I see the event in SessionStorage
  49. Feature: Quizzes Scenario: Analytics When I go to the quiz

    page And I click on the filter
  50. Pixiedust API nsq TEST Dashboard ASSERT QR

  51. Pixiedust API nsq TEST Dashboard ASSERT CRON QR

  52. Pixiedust API nsq TEST Dashboard ASSERT CRON QR

  53. Pixiedust API nsq TEST Dashboard ASSERT CRON QR

  54. SLACK ALARMS

  55. Pixiedust API nsq TEST Dashboard ASSERT CRON QR

  56. LESSONS LEARNED ALONG THE WAY SECTION 5

  57. WE FOUND BUGS WE DIDN’T KNOW ABOUT

  58. TESTING IN PRODUCTION HAS INTERESTING SIDE EFFECTS

  59. None
  60. None
  61. TESTING IN PRODUCTION BROUGHT A LOT OF CONFIDENCE

  62. WE HAVE SEEN ALMOST ZERO REGRESSIONS

  63. SOMETIMES YOU HAVE TO THROW OUT EVERYTHING YOU KNOW

  64. THANKS! @IANFEATHER WWW.BUZZFEED.COM/JOBS