Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Flaky Tests

Flaky Tests

Best practices to avoid writing flaky tests.

B77afd7977c10e820f4821acd38a1f09?s=128

Stephanie Viccari

January 07, 2021
Tweet

Transcript

  1. Identify Flaky Tests 1

  2. What is a Flaky Test? A test that sometimes fails

    without any code changes. 2
  3. Problems with flaky tests • They are useless • They

    waste our time • They undermine our trust 3
  4. 4

  5. What can cause Flaky Tests? 5

  6. Leaky State - Global Variables $global_variable = 1 RSpec.describe "mutating

    a global variable" do it "reads from a global variable" do expect($global_variable).to eq(1) end it "updates the global variable" do $global_variable = 2 expect($global_variable).to eq(2) end end # These tests fail when the second test runs first. # Global variables are best avoided because they break encapsulation. 6
  7. Leaky State - ENV Variables context "when an invalid token

    is sent" do it "returns a status 401" do # ENV["API_TOKEN"] defaults to nil get api_path, params: {}, headers: authorized_headers(token: "token") expect(response.status).to eq(401) end end context "when a valid token is sent" do it "returns a status 200" do ENV["API_TOKEN"] = "token" get api_path, params: {}, headers: authorized_headers(token: "token") expect(response.status).to eq(200) end end 7
  8. Leaky State - ENV Variables: Not Thread Safe context "cache

    and reset the env variable" do it "is not thread safe" do cached_api_token = ENV["API_TOKEN"] ENV["API_TOKEN"] = "token" ... exercise code that relies on env variable ... ENV["API_TOKEN"] = cached_api_token end end 8
  9. ENV Variables - Refactor context "stub ENV" do it "does

    not introduce leaky state and is thread-safe" do allow(ENV).to receive(:[]).with("API_TOKEN").and_return("token") ... exercise code that relies on env variable ... end end 9
  10. ENV Variables - Climate Control context "using Climate Control" do

    it "does not stub ENV and is thread-safe" do ClimateControl.modify API_TOKEN: "token" do ... exercise code that relies on env variable ... end end end 10
  11. Order dependency class Book < ApplicationRecord def self.published where.not(publication_date: nil)

    end end Rspec.describe Book do describe ".published" do it "returns only published books" do book1 = create(:book, :published) book2 = create(:book, :published) book3 = create(:book, :draft) expect(Book.published).to eq [book1, book2] end end end 11
  12. Order dependency - Refactor class Book < ApplicationRecord def self.published

    where.not(publication_date: nil) end end Rspec.describe Book do describe ".published" do it "returns only published books" do book1 = create(:book, :published) book2 = create(:book, :published) book3 = create(:book, :draft) expect(Book.published).to match_array [book1, book2] expect(Book.published).to contain_exactly(book1, book2) end end end 12
  13. Time Rspec.describe Venue do describe ".upcoming_concerts" do it "returns concerts

    today or in the future" do todays_concert = create(:concert, start_time: DateTime.current) yesterday_concert = create(:concert, start_time: DateTime.current - 1.day) tomorrows_concert = create(:concert, start_time: DateTime.current + 1.day) upcoming_concerts = Venue.upcoming_concerts expect(upcoming_concerts).to contain_exactly( todays_concert, tomorrows_concert ) end end end 13
  14. Time - Freeze Time with TimeCop Rspec.describe Venue do describe

    ".upcoming_concerts" do it "returns concerts today or in the future" do Timecop.freeze do todays_concert = create(:concert, start_time: DateTime.current) yesterday_concert = create(:concert, start_time: DateTime.current - 1.day) tomorrows_concert = create(:concert, start_time: DateTime.current + 1.day) upcoming_concerts = Venue.upcoming_concerts expect(upcoming_concerts).to contain_exactly( todays_concert, tomorrows_concert ) end end end end 14
  15. Time - Precision # Ruby Time objects maintain greater precision

    than the database. # use the `be_within` matcher to compare timestamps expect(database_record.updated_at).to be_within(1.second).of(ruby_object.timestamp) 15
  16. Resources Testing and Environment Variables - https://thoughtbot.com/blog/testing- and-environment-variables Climate Control

    - Easily manage your env - https://github.com/thoughtbot/ climate_control Timecop - A gem that provides time travel - https://github.com/travisjeffery/timecop It's About Time (Zones): - https://thoughtbot.com/blog/its-about- time-zones 16