• Mobile payments (Android, iOS, WP7) company from Boston • Show QR code on phone to cashier to create an order • Order #create to Rails 4.1 app • Eventually hits credit/debit card via payment gateway.
Timeout & Accept • Wrap a charge in a timeout • If it times out, evaluate risk • If low risk, save it and return success • Cron task to retry timed-out orders
Timeout # app/models/customer_charger.rb def charge Timeout.timeout(TIMEOUT_IN_SECONDS) do charge_card_via_gateway end rescue Timeout::Error assess_risk_of_saving_order_without_charging_card end
def reconcile # search gateway for similar-‐looking charge if gateway_id = SimilarOrderFinder.new(self).find # found one! update this order and don't re-‐charge update_attribute :gateway_id, gateway_id else charge save end end Order.reconcilable.find_each do |order| order.reconcile end
Same risk as before • If an order is accepted that can’t be charged, we’re still on the hook. • Our support team follows up with customers to keep lost $$ as low as possible.
If order looks real... • Calculate risk: • If low, saves everything: params, headers, etc. to DB. • Returns a response that looks identical to a production response.
When we’re back up: • Order model on chocolate has a replay method. • Manual process run by support team to track results (and follow up if necessary).
De-duping • Akamai injects a unique request ID for every order we create. • Store this on each order in production and on replays in chocolate. • Chocolate sends this as part of a replay.
Triggering • Akamai has a rule that if a POST to our order #create endpoint takes > 15 seconds, retry the exact same request on chocolate. • Sometimes production will actually succeed, but not a problem: chocolate de-dupes.
sikachu
chalasr
shgishzk
mitchan
smiyawaki0820
miso
meihei3
speed1313
leichteckig
guvalif
hollycummins
kinakomoti321
eitanlees
keithpitt
rasmusluckow
orderedlist
davidbonilla
tanoku
jonyablonski
caitiem20
scottboms
bluesmoon
marcelosomers
mraible