Slide 1

Slide 1 text

Testing Isn't Enough: Fighting Bugs with Hacks Paul Gross [email protected] @pgr0ss

Slide 2

Slide 2 text

Bugs are a serious problem in software

Slide 3

Slide 3 text

Mars Climate Orbiter September 23, 1999: Bug in calculation units caused the spacecraft to burn up in Mars atmosphere

Slide 4

Slide 4 text

World of Warcraft Plague September 13, 2005: Bug in a spell caused a pandemic through the virtual world

Slide 5

Slide 5 text

Knight Capital August 1, 2012: Bug in trading software lost $440 million

Slide 6

Slide 6 text

What's our strategy for dealing with bugs today?

Slide 7

Slide 7 text

Testing • Tests are great

Slide 8

Slide 8 text

Testing • Tests are great • Regardless of philosophy or strategy

Slide 9

Slide 9 text

Testing is not enough • Well tested software is not bug free

Slide 10

Slide 10 text

If debugging is the process of removing bugs, then programming must be the process of putting them in. — Edsger W. Dijkstra Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence. — Edsger W. Dijkstra

Slide 11

Slide 11 text

Mitigation • Fail fast when we know there is a problem

Slide 12

Slide 12 text

Mitigation • Fail fast when we know there is a problem • Structure our systems to reduce the severity of bugs

Slide 13

Slide 13 text

Real world example Operating rooms

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

Example Hospital management system

Slide 16

Slide 16 text

class ClinicalNoteController < ApplicationController def show @note = ClinicalNote.find_by!(id: params[:id]) end end

Slide 17

Slide 17 text

class ClinicalNoteController < ApplicationController def show @note = ClinicalNote.find_by!(id: params[:id], patient_id: params[:patient_id]) end end

Slide 18

Slide 18 text

class ClinicalNoteController < ApplicationController def show @note = ClinicalNoteService.find(params[:id], params[:patient_id]) end end

Slide 19

Slide 19 text

class ClinicalNoteController < ApplicationController def show @note = ClinicalNoteService.find(params[:patient_id], params[:id]) raise "incorrect patient" unless @note.patient_id == params[:patient_id] end end

Slide 20

Slide 20 text

Strategy • Figure out the invariants of a system

Slide 21

Slide 21 text

Strategy • Figure out the invariants of a system • Add runtime checks for these invariants

Slide 22

Slide 22 text

Strategy • Figure out the invariants of a system • Add runtime checks for these invariants • Raise and alert so the issue can be investigated and fixed

Slide 23

Slide 23 text

Example Braintree payment gateway

Slide 24

Slide 24 text

class CustomerController < ApplicationController def show @customer = Customer.find_by!(token: params[:token]) end end

Slide 25

Slide 25 text

@customer = @merchant.customers.find_by!(token: params[:token])

Slide 26

Slide 26 text

How can we prevent simple mistakes?

Slide 27

Slide 27 text

>> Customer.where(:token => 'Paul') RuntimeError: #finds must be scoped on Customer

Slide 28

Slide 28 text

>> Customer.where(:token => 'Paul') RuntimeError: #finds must be scoped on Customer >> merchant = Merchant.find(1) >> merchant.customers.where(:token => 'Paul') [#

Slide 29

Slide 29 text

Allow bypassing >> Customer.allow_unscoped_find.where(:token => 'Paul')

Slide 30

Slide 30 text

What about code like this? Customer.find_by_sql(["SELECT * FROM customers WHERE token = ?", "Paul"])

Slide 31

Slide 31 text

URLs are scoped by merchant /merchants//customers/

Slide 32

Slide 32 text

class ApplicationController < ActionController::Base around_filter :ensure_merchant_consistency def ensure_merchant_consistency(&block) @merchant = Merchant.find_by!(:public_id => params["merchant_id"]) MerchantConsistencyCheck.with_merchant(@merchant, &block) end end

Slide 33

Slide 33 text

class ApplicationController < ActionController::Base around_filter :ensure_merchant_consistency def ensure_merchant_consistency(&block) @merchant = Merchant.find_by!(:public_id => params["merchant_id"]) MerchantConsistencyCheck.with_merchant(@merchant, &block) end end ... Customer.find_by_sql(["SELECT * FROM customers WHERE token = ?", "Paul"]) ScopedFindHook::ScopeError: Customer cannot return objects scoped by the incorrect merchant. Got 6, expected 1.

Slide 34

Slide 34 text

Strategy • Focus on the validity of the data

Slide 35

Slide 35 text

Strategy • Focus on the validity of the data • Code can be fixed more easily than data

Slide 36

Slide 36 text

Example Recurring billing

Slide 37

Slide 37 text

Check data consistency if billing_period_start_date > billing_period_end_date raise "Subscription internal state is inconsistent -- " + "billing_period_start_date vs billing_period_end_date: #{inspect}" end

Slide 38

Slide 38 text

Check data consistency if billing_period_start_date > billing_period_end_date raise "Subscription internal state is inconsistent -- " + "billing_period_start_date vs billing_period_end_date: #{inspect}" end if _correct_next_billing_date != next_billing_date raise "Subscription internal state is inconsistent -- " + "next_billing_date incorrect: #{inspect}" end

Slide 39

Slide 39 text

Strategy • Write checks outside of the normal flow

Slide 40

Slide 40 text

Strategy • Write checks outside of the normal flow • Keep checking code simpler than real code

Slide 41

Slide 41 text

Example Settlement

Slide 42

Slide 42 text

Nagios check SELECT COUNT(*) FROM transactions WHERE status = 'submitted_for_settlement' AND updated_at < NOW() - INTERVAL '1 DAY'

Slide 43

Slide 43 text

Hacks for development

Slide 44

Slide 44 text

Example Safe migrations

Slide 45

Slide 45 text

def self.up add_index :transactions, :customer_last_name end

Slide 46

Slide 46 text

def self.up add_index :transactions, :customer_last_name end ActiveRecord::SafeSchemaStatements::UnsafeMigrationError: :add_index is NOT SAFE! Use safe_add_concurrent_index instead

Slide 47

Slide 47 text

def self.up safe_add_concurrent_index :transactions, :customer_last_name end

Slide 48

Slide 48 text

Allow bypassing def self.up unsafe_remove_column :transactions, :old_column end

Slide 49

Slide 49 text

Example Sanity specs

Slide 50

Slide 50 text

require 'spec_helper' describe 'Sanity Specs' do it "has a unique index on every public_id column" do indexes = ActiveRecord::Base.connection.select_values(<<-SQL) SELECT relname FROM pg_class WHERE relkind = 'i'" SQL ActiveRecord::Base.connection.tables.each do |table| indexes.grep(/index_#{table}_on.*public_id/).size.should eql(1), "#{table} does not have an index on public_id" end end end

Slide 51

Slide 51 text

Summary • Check for invariants

Slide 52

Slide 52 text

Summary • Check for invariants • Focus on data validity

Slide 53

Slide 53 text

Summary • Check for invariants • Focus on data validity • Write checks outside of the normal flow

Slide 54

Slide 54 text

Summary • Check for invariants • Focus on data validity • Write checks outside of the normal flow • Find your own examples

Slide 55

Slide 55 text

Questions? [email protected] @pgr0ss

Slide 56

Slide 56 text

Photo Credits Mars Climate Orbiter: Courtesy NASA/JPL-Caltech World of Warcraft: ©2004 Blizzard Entertainment, Inc. Bugs: flickr.com/photos/editor/11341534765 (CC BY 2.0) NYSE: flickr.com/photos/ilamont/5538510845 (CC BY 2.0) Testing: flickr.com/photos/sidelong/246816211 (CC BY 2.0) Kids don't float: flickr.com/photos/iluvcocacola/535258842 (CC BY-NC-ND 2.0) Pillars: flickr.com/photos/tom-carden/80262 (CC BY-NC-SA 2.0)

Slide 57

Slide 57 text

Photo Credits CDs: flickr.com/photos/swanksalot/2704017177 (CC BY-SA 2.0) Fence: flickr.com/photos/clonedmilkmen/2971765601 (CC BY- SA 2.0) Hacking: flickr.com/photos/adulau/9464930917 (CC BY-SA 2.0) Safety First: flickr.com/photos/krossbow/434064539 (CC BY 2.0)