The Doctor Is In: Using checkups to find bugs in production

The Doctor Is In: Using checkups to find bugs in production

Full speaker notes at: http://www.rofreg.com/talks
Presented at RailsConf 2018

A good test suite can help you catch errors in development, but how do you know if your code starts misbehaving in production?

In this talk, we’ll learn about checkups: a powerful and flexible way to ensure that your code continues to work as intended after you deploy. Through real-world examples, we’ll see how adding a simple suite of checkups to your app can help you detect unforeseen issues, including tricky problems like race conditions and data corruption. We’ll even look at how checkups can mitigate much larger disasters in real-time. Come give your app’s health a boost!

B6d3539d5ffda82fd2c26d0fc42a126f?s=128

Ryan Laughlin

April 17, 2018
Tweet

Transcript

  1. 3.
  2. 6.
  3. 7.
  4. 22.
  5. 27.
  6. 28.
  7. 35.

    $ bundle exec rspec ... Finished in 6 minutes 36

    seconds 1738 examples, 13 failures @rofreg
  8. 40.

    Every user should have a valid email address EXPECTATION Does

    every user have a valid email address? CHECKUP @rofreg
  9. 42.
  10. 43.
  11. 44.
  12. 45.

    When a checkup fails, it sends you an alert so

    that you can inves6gate STEP #3 @rofreg
  13. 46.

    Does every user have a valid email address? 2:00pm ✅

    3:00pm ✅ 4:00pm ⚠ ✉❗ @rofreg
  14. 52.
  15. 53.

    class User < ApplicationRecord has_many :email_addresses, autosave: true # Make

    sure all users have at least one email address validates :email_addresses, presence: true end @rofreg
  16. 54.

    class User < ApplicationRecord has_many :email_addresses, autosave: true # Make

    sure all users have at least one email address validates :email_addresses, presence: true end ? @rofreg
  17. 57.
  18. 58.

    # Check for recently updated users with no email address

    recently_updated_users = User.where(updated_at: 1.hour.ago...Time.now) recently_updated_users.each do |user| raise_an_alarm_about(user) if user.email_addresses.none? end @rofreg
  19. 59.

    Does every user have at least 1 email address? Day

    1 ✅ Day 2 ✅ Day 3 ⚠ @rofreg
  20. 60.

    Does every user have at least 1 email address? Day

    1 ✅ Day 2 ✅ Day 3 ⚠ @rofreg
  21. 61.

    Does every user have at least 1 email address? Day

    1 ✅ Day 2 ✅ Day 3 ⚠ @rofreg
  22. 62.
  23. 63.
  24. 67.

    REQUEST #2 REQUEST #1 Passes valida6on? ✅ Passes valida6on? ✅

    ada.lovelace@gmail.com lovelace@yahoo.com ada.lovelace@gmail.com lovelace@yahoo.com @rofreg
  25. 68.

    REQUEST #2 REQUEST #1 Passes valida6on? ✅ Passes valida6on? ✅

    COMMIT COMMIT ada.lovelace@gmail.com lovelace@yahoo.com ada.lovelace@gmail.com lovelace@yahoo.com @rofreg
  26. 70.
  27. 73.

    # Check for recently updated users with no email address

    recently_updated_users = User.where(updated_at: 1.hour.ago...Time.now) recently_updated_users.each do |user| raise_an_alarm_about(user) if user.email_addresses.none? end @rofreg
  28. 74.

    # lib/tasks/checkups/hourly.rake # called via `rake checkups:hourly`, at least once

    per hour task check_for_users_without_email_addresses: :environment do recently_updated_users = User.where(updated_at: 1.hour.ago...Time.now) recently_updated_users.each do |user| raise_an_alarm_about(user) if user.email_addresses.none? end end @rofreg
  29. 80.
  30. 96.

    users.any? { |user| ... } users.sample(100).any? { |user| ... }

    EXHAUSTIVE CHECKUPS SPOT-CHECK CHECKUPS @rofreg
  31. 98.
  32. 103.
  33. 115.

    # Check for recently updated users that now fail validation

    recently_updated_users = User.where(updated_at: 1.hour.ago...Time.now) recently_updated_users.each do |user| raise_an_alarm_about(user) unless user.valid? end @rofreg