Testing Isn't Enough: Fighting Bugs with Hacks

A6cbdd6473de1af38ef1d8e01588ae7e?s=47 Paul Gross
November 17, 2014

Testing Isn't Enough: Fighting Bugs with Hacks

This talk was given at RubyConf 2014.

A6cbdd6473de1af38ef1d8e01588ae7e?s=128

Paul Gross

November 17, 2014
Tweet

Transcript

  1. Testing Isn't Enough: Fighting Bugs with Hacks Paul Gross paul@braintreepayments.com

    @pgr0ss
  2. Bugs are a serious problem in software

  3. Mars Climate Orbiter September 23, 1999: Bug in calculation units

    caused the spacecraft to burn up in Mars atmosphere
  4. World of Warcraft Plague September 13, 2005: Bug in a

    spell caused a pandemic through the virtual world
  5. Knight Capital August 1, 2012: Bug in trading software lost

    $440 million
  6. What's our strategy for dealing with bugs today?

  7. Testing • Tests are great

  8. Testing • Tests are great • Regardless of philosophy or

    strategy
  9. Testing is not enough • Well tested software is not

    bug free
  10. If debugging is the process of removing bugs, then programming

    must be the process of putting them in. — Edsger W. Dijkstra Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence. — Edsger W. Dijkstra
  11. Mitigation • Fail fast when we know there is a

    problem
  12. Mitigation • Fail fast when we know there is a

    problem • Structure our systems to reduce the severity of bugs
  13. Real world example Operating rooms

  14. None
  15. Example Hospital management system

  16. class ClinicalNoteController < ApplicationController def show @note = ClinicalNote.find_by!(id: params[:id])

    end end
  17. class ClinicalNoteController < ApplicationController def show @note = ClinicalNote.find_by!(id: params[:id],

    patient_id: params[:patient_id]) end end
  18. class ClinicalNoteController < ApplicationController def show @note = ClinicalNoteService.find(params[:id], params[:patient_id])

    end end
  19. class ClinicalNoteController < ApplicationController def show @note = ClinicalNoteService.find(params[:patient_id], params[:id])

    raise "incorrect patient" unless @note.patient_id == params[:patient_id] end end
  20. Strategy • Figure out the invariants of a system

  21. Strategy • Figure out the invariants of a system •

    Add runtime checks for these invariants
  22. Strategy • Figure out the invariants of a system •

    Add runtime checks for these invariants • Raise and alert so the issue can be investigated and fixed
  23. Example Braintree payment gateway

  24. class CustomerController < ApplicationController def show @customer = Customer.find_by!(token: params[:token])

    end end
  25. @customer = @merchant.customers.find_by!(token: params[:token])

  26. How can we prevent simple mistakes?

  27. >> Customer.where(:token => 'Paul') RuntimeError: #finds must be scoped on

    Customer
  28. >> Customer.where(:token => 'Paul') RuntimeError: #finds must be scoped on

    Customer >> merchant = Merchant.find(1) >> merchant.customers.where(:token => 'Paul') [#<Customer id: 1 ...
  29. Allow bypassing >> Customer.allow_unscoped_find.where(:token => 'Paul')

  30. What about code like this? Customer.find_by_sql(["SELECT * FROM customers WHERE

    token = ?", "Paul"])
  31. URLs are scoped by merchant /merchants/<merchant_id>/customers/<token>

  32. class ApplicationController < ActionController::Base around_filter :ensure_merchant_consistency def ensure_merchant_consistency(&block) @merchant =

    Merchant.find_by!(:public_id => params["merchant_id"]) MerchantConsistencyCheck.with_merchant(@merchant, &block) end end
  33. class ApplicationController < ActionController::Base around_filter :ensure_merchant_consistency def ensure_merchant_consistency(&block) @merchant =

    Merchant.find_by!(:public_id => params["merchant_id"]) MerchantConsistencyCheck.with_merchant(@merchant, &block) end end ... Customer.find_by_sql(["SELECT * FROM customers WHERE token = ?", "Paul"]) ScopedFindHook::ScopeError: Customer cannot return objects scoped by the incorrect merchant. Got 6, expected 1.
  34. Strategy • Focus on the validity of the data

  35. Strategy • Focus on the validity of the data •

    Code can be fixed more easily than data
  36. Example Recurring billing

  37. Check data consistency if billing_period_start_date > billing_period_end_date raise "Subscription internal

    state is inconsistent -- " + "billing_period_start_date vs billing_period_end_date: #{inspect}" end
  38. Check data consistency if billing_period_start_date > billing_period_end_date raise "Subscription internal

    state is inconsistent -- " + "billing_period_start_date vs billing_period_end_date: #{inspect}" end if _correct_next_billing_date != next_billing_date raise "Subscription internal state is inconsistent -- " + "next_billing_date incorrect: #{inspect}" end
  39. Strategy • Write checks outside of the normal flow

  40. Strategy • Write checks outside of the normal flow •

    Keep checking code simpler than real code
  41. Example Settlement

  42. Nagios check SELECT COUNT(*) FROM transactions WHERE status = 'submitted_for_settlement'

    AND updated_at < NOW() - INTERVAL '1 DAY'
  43. Hacks for development

  44. Example Safe migrations

  45. def self.up add_index :transactions, :customer_last_name end

  46. def self.up add_index :transactions, :customer_last_name end ActiveRecord::SafeSchemaStatements::UnsafeMigrationError: :add_index is NOT

    SAFE! Use safe_add_concurrent_index instead
  47. def self.up safe_add_concurrent_index :transactions, :customer_last_name end

  48. Allow bypassing def self.up unsafe_remove_column :transactions, :old_column end

  49. Example Sanity specs

  50. require 'spec_helper' describe 'Sanity Specs' do it "has a unique

    index on every public_id column" do indexes = ActiveRecord::Base.connection.select_values(<<-SQL) SELECT relname FROM pg_class WHERE relkind = 'i'" SQL ActiveRecord::Base.connection.tables.each do |table| indexes.grep(/index_#{table}_on.*public_id/).size.should eql(1), "#{table} does not have an index on public_id" end end end
  51. Summary • Check for invariants

  52. Summary • Check for invariants • Focus on data validity

  53. Summary • Check for invariants • Focus on data validity

    • Write checks outside of the normal flow
  54. Summary • Check for invariants • Focus on data validity

    • Write checks outside of the normal flow • Find your own examples
  55. Questions? paul@braintreepayments.com @pgr0ss

  56. Photo Credits Mars Climate Orbiter: Courtesy NASA/JPL-Caltech World of Warcraft:

    ©2004 Blizzard Entertainment, Inc. Bugs: flickr.com/photos/editor/11341534765 (CC BY 2.0) NYSE: flickr.com/photos/ilamont/5538510845 (CC BY 2.0) Testing: flickr.com/photos/sidelong/246816211 (CC BY 2.0) Kids don't float: flickr.com/photos/iluvcocacola/535258842 (CC BY-NC-ND 2.0) Pillars: flickr.com/photos/tom-carden/80262 (CC BY-NC-SA 2.0)
  57. Photo Credits CDs: flickr.com/photos/swanksalot/2704017177 (CC BY-SA 2.0) Fence: flickr.com/photos/clonedmilkmen/2971765601 (CC

    BY- SA 2.0) Hacking: flickr.com/photos/adulau/9464930917 (CC BY-SA 2.0) Safety First: flickr.com/photos/krossbow/434064539 (CC BY 2.0)