Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Flaky Tests

Flaky Tests

Best practices to avoid writing flaky tests.

Stephanie Viccari

January 07, 2021
Tweet

More Decks by Stephanie Viccari

Other Decks in Education

Transcript

  1. Problems with flaky tests • They are useless • They

    waste our time • They undermine our trust 3
  2. 4

  3. Leaky State - Global Variables $global_variable = 1 RSpec.describe "mutating

    a global variable" do it "reads from a global variable" do expect($global_variable).to eq(1) end it "updates the global variable" do $global_variable = 2 expect($global_variable).to eq(2) end end # These tests fail when the second test runs first. # Global variables are best avoided because they break encapsulation. 6
  4. Leaky State - ENV Variables context "when an invalid token

    is sent" do it "returns a status 401" do # ENV["API_TOKEN"] defaults to nil get api_path, params: {}, headers: authorized_headers(token: "token") expect(response.status).to eq(401) end end context "when a valid token is sent" do it "returns a status 200" do ENV["API_TOKEN"] = "token" get api_path, params: {}, headers: authorized_headers(token: "token") expect(response.status).to eq(200) end end 7
  5. Leaky State - ENV Variables: Not Thread Safe context "cache

    and reset the env variable" do it "is not thread safe" do cached_api_token = ENV["API_TOKEN"] ENV["API_TOKEN"] = "token" ... exercise code that relies on env variable ... ENV["API_TOKEN"] = cached_api_token end end 8
  6. ENV Variables - Refactor context "stub ENV" do it "does

    not introduce leaky state and is thread-safe" do allow(ENV).to receive(:[]).with("API_TOKEN").and_return("token") ... exercise code that relies on env variable ... end end 9
  7. ENV Variables - Climate Control context "using Climate Control" do

    it "does not stub ENV and is thread-safe" do ClimateControl.modify API_TOKEN: "token" do ... exercise code that relies on env variable ... end end end 10
  8. Order dependency class Book < ApplicationRecord def self.published where.not(publication_date: nil)

    end end Rspec.describe Book do describe ".published" do it "returns only published books" do book1 = create(:book, :published) book2 = create(:book, :published) book3 = create(:book, :draft) expect(Book.published).to eq [book1, book2] end end end 11
  9. Order dependency - Refactor class Book < ApplicationRecord def self.published

    where.not(publication_date: nil) end end Rspec.describe Book do describe ".published" do it "returns only published books" do book1 = create(:book, :published) book2 = create(:book, :published) book3 = create(:book, :draft) expect(Book.published).to match_array [book1, book2] expect(Book.published).to contain_exactly(book1, book2) end end end 12
  10. Time Rspec.describe Venue do describe ".upcoming_concerts" do it "returns concerts

    today or in the future" do todays_concert = create(:concert, start_time: DateTime.current) yesterday_concert = create(:concert, start_time: DateTime.current - 1.day) tomorrows_concert = create(:concert, start_time: DateTime.current + 1.day) upcoming_concerts = Venue.upcoming_concerts expect(upcoming_concerts).to contain_exactly( todays_concert, tomorrows_concert ) end end end 13
  11. Time - Freeze Time with TimeCop Rspec.describe Venue do describe

    ".upcoming_concerts" do it "returns concerts today or in the future" do Timecop.freeze do todays_concert = create(:concert, start_time: DateTime.current) yesterday_concert = create(:concert, start_time: DateTime.current - 1.day) tomorrows_concert = create(:concert, start_time: DateTime.current + 1.day) upcoming_concerts = Venue.upcoming_concerts expect(upcoming_concerts).to contain_exactly( todays_concert, tomorrows_concert ) end end end end 14
  12. Time - Precision # Ruby Time objects maintain greater precision

    than the database. # use the `be_within` matcher to compare timestamps expect(database_record.updated_at).to be_within(1.second).of(ruby_object.timestamp) 15
  13. Resources Testing and Environment Variables - https://thoughtbot.com/blog/testing- and-environment-variables Climate Control

    - Easily manage your env - https://github.com/thoughtbot/ climate_control Timecop - A gem that provides time travel - https://github.com/travisjeffery/timecop It's About Time (Zones): - https://thoughtbot.com/blog/its-about- time-zones 16