Tuesday
Little Opportunity for Contention
=> BEGIN
=> UPDATE products ... WHERE sku = APL1
=> UPDATE products ... WHERE sku = ORG5
=> COMMIT
< 1 millisecond
Slide 43
Slide 43 text
Tuesday
Large Opportunity for Contention
=> BEGIN
=> UPDATE products ... WHERE sku = APL1
=> UPDATE products ... WHERE sku = ORG5
=> COMMIT
> 1 second
Slide 44
Slide 44 text
Tuesday
Large Opportunity for Contention
=> BEGIN
=> UPDATE products ... WHERE sku = APL1
=> UPDATE products ... WHERE sku = ORG5
=> COMMIT
Slow Queries
Slide 45
Slide 45 text
Tuesday
Large Opportunity for Contention
=> BEGIN
=> UPDATE products ... WHERE sku = APL1
=> UPDATE products ... WHERE sku = PLM3
=> UPDATE products ... WHERE sku = BRY4
=> UPDATE products ... WHERE sku = PER2
... (hundreds more)
=> UPDATE products ... WHERE sku = ORG5
=> COMMIT
Too Many Queries
Slide 46
Slide 46 text
Tuesday
Large Opportunity for Contention
=> BEGIN
=> UPDATE products ... WHERE sku = APL1
=> UPDATE products ... WHERE sku = PLM3
=> UPDATE products ... WHERE sku = BRY4
=> UPDATE products ... WHERE sku = PER2
... (hundreds more)
=> UPDATE products ... WHERE sku = ORG5
=> COMMIT
Waiting on Locks
Slide 47
Slide 47 text
Tuesday
def submit
Order.transaction do
@order.save!
@order.update_product_inventory
end
end
Slide 48
Slide 48 text
Tuesday
def submit
Order.transaction do
@order.save!
@order.update_product_inventory
end
end
Slide 49
Slide 49 text
Wednesday
Background Jobs
Slide 50
Slide 50 text
“
I don’t have time to
waste. I want my
orders to submit
without delay.
Slide 51
Slide 51 text
Wednesday
class Order < ApplicationRecord
after_create :finalize_order
end
Slide 52
Slide 52 text
Wednesday
class Order < ApplicationRecord
after_create :finalize_order
def finalize_order
# Sync with billing platform
# Send confirmation email
# etc.
end
end
Slide 53
Slide 53 text
Wednesday
class Order < ApplicationRecord
after_create :finalize_order
def finalize_order
OrderFinalizationJob.perform_later
end
end
Slide 54
Slide 54 text
!Bug Report
Confirmation emails delayed by minutes
Slide 55
Slide 55 text
Wednesday
=> BEGIN
=> INSERT INTO orders ...
=> Enqueue Job
Slide 56
Slide 56 text
Wednesday
=> BEGIN
=> INSERT INTO orders ...
=> Enqueue Job
=> Job Executes and Fails
Slide 57
Slide 57 text
Wednesday
=> BEGIN
=> INSERT INTO orders ...
=> Enqueue Job
=> Job Executes and Fails
=> UPDATE products ...
Slide 58
Slide 58 text
Wednesday
=> BEGIN
=> INSERT INTO orders ...
=> Enqueue Job
=> Job Executes and Fails
=> UPDATE products ...
=> Job Retries and Fails
Slide 59
Slide 59 text
Wednesday
=> BEGIN
=> INSERT INTO orders ...
=> Enqueue Job
=> Job Executes and Fails
=> UPDATE products ...
=> Job Retries and Fails
=> COMMIT
Slide 60
Slide 60 text
Wednesday
=> BEGIN
=> INSERT INTO orders ...
=> Enqueue Job
=> Job Executes and Fails
=> UPDATE products ...
=> Job Retries and Fails
=> COMMIT
=> Job Retries and Succeeds
Slide 61
Slide 61 text
Wednesday
=> BEGIN
=> INSERT INTO orders ...
=> Enqueue Job
=> Job Executes and Fails
=> UPDATE products ...
=> Job Retries and Fails
=> ROLLBACK
Slide 62
Slide 62 text
Wednesday
=> BEGIN
=> INSERT INTO orders ...
=> Enqueue Job
=> Job Executes and Fails
=> UPDATE products ...
=> Job Retries and Fails
=> ROLLBACK
=> Job Retries and Fails
Slide 63
Slide 63 text
Wednesday
=> BEGIN
=> INSERT INTO orders ...
=> Enqueue Job
=> Job Executes and Fails
=> UPDATE products ...
=> Job Retries and Fails
=> ROLLBACK
=> Job Retries and Fails
=> Job Retries and Fails
Slide 64
Slide 64 text
Wednesday
class Order < ApplicationRecord
after_create :finalize_order
def finalize_order
OrderFinalizationJob.perform_later
end
end
Slide 65
Slide 65 text
Wednesday
class Order < ApplicationRecord
after_create_commit :finalize_order
def finalize_order
OrderFinalizationJob.perform_later
end
end
Slide 66
Slide 66 text
Wednesday
=> BEGIN
=> INSERT INTO orders ...
=> UPDATE products ...
=> COMMIT
=> Enqueue Job
=> Job Executes and Succeeds
Slide 67
Slide 67 text
Thursday
External Services
Slide 68
Slide 68 text
“
I want my orders
fulfilled quickly and
accurately.
Slide 69
Slide 69 text
Thursday
def submit
Order.transaction do
@order.save!
@order.update_product_inventory
end
end
Slide 70
Slide 70 text
Thursday
def submit
Order.transaction do
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
end
end
Slide 71
Slide 71 text
Thursday
=> BEGIN
=> INSERT INTO orders ...
=> External Call
=> UPDATE products ...
=> COMMIT
Slide 72
Slide 72 text
!Bug Report
Site-wide failure
Slide 73
Slide 73 text
Thursday
def submit
Order.transaction do
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
end
end
Responding Slowly
Slide 74
Slide 74 text
Thursday
=> BEGIN
=> INSERT INTO orders ...
=> UPDATE products ...
=> COMMIT / ROLLBACK
Slow External Call
Slide 75
Slide 75 text
Thursday
=> BEGIN
=> INSERT INTO orders ...
=> UPDATE products ...
=> COMMIT / ROLLBACK
Idle
Slide 76
Slide 76 text
Thursday
=> BEGIN
=> INSERT INTO orders ...
=> UPDATE products ...
=> COMMIT / ROLLBACK
Idle Locks Held
Slide 77
Slide 77 text
Thursday
=> BEGIN
=> INSERT INTO orders ...
=> UPDATE products ...
=> COMMIT / ROLLBACK
Idle Connection in Use
Slide 78
Slide 78 text
Thursday
Database
Slide 79
Slide 79 text
Thursday
BEGIN
Database
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
BEGIN
Thursday
def submit
Order.transaction do
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
end
end
Configure Timeout
Slide 82
Slide 82 text
Thursday
def submit
FulfillmentClient.submit_order(@order)
Order.transaction do
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
end
end
Slide 83
Slide 83 text
Thursday
def submit
Order.transaction do
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
end
FulfillmentClient.submit_order(@order)
end
Slide 84
Slide 84 text
Thursday
def submit
Order.transaction do
@order.save!
@order.update_product_inventory
end
FulfillmentClient.submit_order(@order)
@order.update!(status: :submit)
end
Slide 85
Slide 85 text
Friday
Multiple Databases
Slide 86
Slide 86 text
“
I expect a reliable
website. BugHub
has been having
a lot of problems
lately.
Slide 87
Slide 87 text
Friday
Products
Orders
Slide 88
Slide 88 text
Friday
def submit
Order.transaction do
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
end
end
Slide 89
Slide 89 text
Friday
def submit
Order.transaction do
Product.transaction do
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
end
end
end
Slide 90
Slide 90 text
!Bug Report
All of the above, but worse
Slide 91
Slide 91 text
Friday
def submit
Order.transaction do
Product.transaction do
...
end
end
end
Problems Affect Both Databases
Friday
def submit
Order.transaction do
Product.transaction do
...
end
end
end
Not Atomic
Slide 94
Slide 94 text
Friday
Orders Database
=> BEGIN
=> INSERT INTO orders ...
=> COMMIT
Products Database
=> BEGIN
=> UPDATE products ...
=> COMMIT
Both Commit $
Slide 95
Slide 95 text
Friday
Orders Database
=> BEGIN
=> INSERT INTO orders ...
=> ROLLBACK
Products Database
=> BEGIN
=> UPDATE products ...
=> ROLLBACK
Both Rollback $
Slide 96
Slide 96 text
Friday
Orders Database
=> BEGIN
=> INSERT INTO orders ...
=> ROLLBACK
Products Database
=> BEGIN
=> UPDATE products ...
=> COMMIT
Not Atomic %
Slide 97
Slide 97 text
Friday
Orders Database
=> BEGIN
=> INSERT INTO orders ...
Products Database
=> BEGIN
=> UPDATE products ...
=> COMMIT
Idle
Slide 98
Slide 98 text
Friday
Orders Database
=> BEGIN
=> INSERT INTO orders ...
Products Database
=> BEGIN
=> UPDATE products ...
=> COMMIT
Idle
❌ Connection Failed
Slide 99
Slide 99 text
Friday
def submit
Order.transaction do
Product.transaction do
...
end
end
end
Slide 100
Slide 100 text
Friday
def submit
Product.transaction do
...
end
Order.transaction do
...
end
end
Slide 101
Slide 101 text
Friday
def submit
Order.transaction do
...
end
Product.transaction do
...
end
end
Slide 102
Slide 102 text
Weekend
Rest and Reflect
Slide 103
Slide 103 text
Weekend
External Calls within Transactions are a Risk
Slide 104
Slide 104 text
Weekend
External Calls within Transactions are a Risk
• Data Integrity Problems
Slide 105
Slide 105 text
Weekend
External Calls within Transactions are a Risk
• Data Integrity Problems
• Cascading Failures
Slide 106
Slide 106 text
Weekend
External Calls within Transactions are a Risk
• Data Integrity Problems
• Cascading Failures
• Includes: HTTP requests, emails, jobs, queries to other databases, etc.
Slide 107
Slide 107 text
Weekend
Slow Transactions are a Risk
Slide 108
Slide 108 text
Weekend
Slow Transactions are a Risk
• Slow Requests
Slide 109
Slide 109 text
Weekend
Slow Transactions are a Risk
• Slow Requests
• Contention
Slide 110
Slide 110 text
Weekend
Slow Transactions are a Risk
• Slow Requests
• Contention
• Resource Exhaustion
Slide 111
Slide 111 text
Monday
Metamorphosis
Slide 112
Slide 112 text
def submit
Order.transaction do
Product.transaction do
Chrysalis.transaction do
code.not_meant_for_reading!
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
@product_suggestions = SuggestionService.build_for(@bug)
seriously.stop_reading_this!
if @order.bug.caterpillar?
@bug.very_hungry!
basket = Chrysalis::Basket.create!(@bug)
PromotionMailer.new_basket(basket).deliver_later
@product_suggestions << Chrysalis.default_items_for(@bug)
else
@bug.book_suggestions.create!(
Book.find_by(author: "Eric Carle”, bug: @order.bug)
)
end
@bug.suggest(@product_suggestions)
@bug.friends.each do |friend|
@order.share_with(friend) if @bug.sharing_orders?
end
rescue => e
handle_transaction_error(e)
ensure
nobody.should_have_read_any_of_this
end
end
end
end
Monday
Slide 113
Slide 113 text
def submit
Order.transaction do
Product.transaction do
Chrysalis.transaction do
code.not_meant_for_reading!
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
@product_suggestions = SuggestionService.build_for(@bug)
seriously.stop_reading_this!
if @order.bug.caterpillar?
@bug.very_hungry!
basket = Chrysalis::Basket.create!(@bug)
PromotionMailer.new_basket(basket).deliver_later
@product_suggestions << Chrysalis.default_items_for(@bug)
else
@bug.book_suggestions.create!(
Book.find_by(author: "Eric Carle”, bug: @order.bug)
)
end
@bug.suggest(@product_suggestions)
@bug.friends.each do |friend|
@order.share_with(friend) if @bug.sharing_orders?
end
rescue => e
handle_transaction_error(e)
ensure
nobody.should_have_read_any_of_this
end
end
end
end
Monday
Small Incremental Changes
Slide 114
Slide 114 text
def submit
Order.transaction do
Product.transaction do
Chrysalis.transaction do
code.not_meant_for_reading!
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
@product_suggestions = SuggestionService.build_for(@bug)
seriously.stop_reading_this!
if @order.bug.caterpillar?
@bug.very_hungry!
basket = Chrysalis::Basket.create!(@bug)
PromotionMailer.new_basket(basket).deliver_later
@product_suggestions << Chrysalis.default_items_for(@bug)
else
@bug.book_suggestions.create!(
Book.find_by(author: "Eric Carle”, bug: @order.bug)
)
end
@bug.suggest(@product_suggestions)
@bug.friends.each do |friend|
@order.share_with(friend) if @bug.sharing_orders?
end
rescue => e
handle_transaction_error(e)
ensure
nobody.should_have_read_any_of_this
end
end
end
end
Monday
Defer Until After Commit
Slide 115
Slide 115 text
def submit
Order.transaction do
Product.transaction do
Chrysalis.transaction do
code.not_meant_for_reading!
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
@product_suggestions = SuggestionService.build_for(@bug)
seriously.stop_reading_this!
if @order.bug.caterpillar?
@bug.very_hungry!
basket = Chrysalis::Basket.create!(@bug)
PromotionMailer.new_basket(basket).deliver_later
@product_suggestions << Chrysalis.default_items_for(@bug)
else
@bug.book_suggestions.create!(
Book.find_by(author: "Eric Carle”, bug: @order.bug)
)
end
@bug.suggest(@product_suggestions)
@bug.friends.each do |friend|
@order.share_with(friend) if @bug.sharing_orders?
end
rescue => e
handle_transaction_error(e)
ensure
nobody.should_have_read_any_of_this
end
end
end
end
Monday
Defer Until After Commit
- Rails 7.2 - Automatically delay Active Job
enqueues to after commit #51426
Slide 116
Slide 116 text
def submit
Order.transaction do
Product.transaction do
Chrysalis.transaction do
code.not_meant_for_reading!
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
@product_suggestions = SuggestionService.build_for(@bug)
seriously.stop_reading_this!
if @order.bug.caterpillar?
@bug.very_hungry!
basket = Chrysalis::Basket.create!(@bug)
PromotionMailer.new_basket(basket).deliver_later
@product_suggestions << Chrysalis.default_items_for(@bug)
else
@bug.book_suggestions.create!(
Book.find_by(author: "Eric Carle”, bug: @order.bug)
)
end
@bug.suggest(@product_suggestions)
@bug.friends.each do |friend|
@order.share_with(friend) if @bug.sharing_orders?
end
rescue => e
handle_transaction_error(e)
ensure
nobody.should_have_read_any_of_this
end
end
end
end
Monday
Defer Until After Commit
- Rails 7.2 - Automatically delay Active Job
enqueues to after commit #51426
- Rails 7.2 - Allow to register transaction
callbacks outside of a record #51474
Slide 117
Slide 117 text
def submit
Order.transaction do
Product.transaction do
Chrysalis.transaction do
code.not_meant_for_reading!
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
@product_suggestions = SuggestionService.build_for(@bug)
seriously.stop_reading_this!
if @order.bug.caterpillar?
@bug.very_hungry!
basket = Chrysalis::Basket.create!(@bug)
PromotionMailer.new_basket(basket).deliver_later
@product_suggestions << Chrysalis.default_items_for(@bug)
else
@bug.book_suggestions.create!(
Book.find_by(author: "Eric Carle”, bug: @order.bug)
)
end
@bug.suggest(@product_suggestions)
@bug.friends.each do |friend|
@order.share_with(friend) if @bug.sharing_orders?
end
rescue => e
handle_transaction_error(e)
ensure
nobody.should_have_read_any_of_this
end
end
end
end
Monday
Identify Other Problematic Transactions
Slide 118
Slide 118 text
def submit
Order.transaction do
Product.transaction do
Chrysalis.transaction do
code.not_meant_for_reading!
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
@product_suggestions = SuggestionService.build_for(@bug)
seriously.stop_reading_this!
if @order.bug.caterpillar?
@bug.very_hungry!
basket = Chrysalis::Basket.create!(@bug)
PromotionMailer.new_basket(basket).deliver_later
@product_suggestions << Chrysalis.default_items_for(@bug)
else
@bug.book_suggestions.create!(
Book.find_by(author: "Eric Carle”, bug: @order.bug)
)
end
@bug.suggest(@product_suggestions)
@bug.friends.each do |friend|
@order.share_with(friend) if @bug.sharing_orders?
end
rescue => e
handle_transaction_error(e)
ensure
nobody.should_have_read_any_of_this
end
end
end
end
Monday
Identify Other Problematic Transactions
- Database Observability
Slide 119
Slide 119 text
def submit
Order.transaction do
Product.transaction do
Chrysalis.transaction do
code.not_meant_for_reading!
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
@product_suggestions = SuggestionService.build_for(@bug)
seriously.stop_reading_this!
if @order.bug.caterpillar?
@bug.very_hungry!
basket = Chrysalis::Basket.create!(@bug)
PromotionMailer.new_basket(basket).deliver_later
@product_suggestions << Chrysalis.default_items_for(@bug)
else
@bug.book_suggestions.create!(
Book.find_by(author: "Eric Carle”, bug: @order.bug)
)
end
@bug.suggest(@product_suggestions)
@bug.friends.each do |friend|
@order.share_with(friend) if @bug.sharing_orders?
end
rescue => e
handle_transaction_error(e)
ensure
nobody.should_have_read_any_of_this
end
end
end
end
Monday
Identify Other Problematic Transactions
- Database Observability
- Rails 7.1 - Instrument Active Record
transactions #49192
Slide 120
Slide 120 text
def submit
Order.transaction do
Product.transaction do
Chrysalis.transaction do
code.not_meant_for_reading!
@order.save!
FulfillmentClient.submit_order(@order)
@order.update_product_inventory
@product_suggestions = SuggestionService.build_for(@bug)
seriously.stop_reading_this!
if @order.bug.caterpillar?
@bug.very_hungry!
basket = Chrysalis::Basket.create!(@bug)
PromotionMailer.new_basket(basket).deliver_later
@product_suggestions << Chrysalis.default_items_for(@bug)
else
@bug.book_suggestions.create!(
Book.find_by(author: "Eric Carle”, bug: @order.bug)
)
end
@bug.suggest(@product_suggestions)
@bug.friends.each do |friend|
@order.share_with(friend) if @bug.sharing_orders?
end
rescue => e
handle_transaction_error(e)
ensure
nobody.should_have_read_any_of_this
end
end
end
end
Monday
Prevent Further Problematic Transactions
Slide 121
Slide 121 text
Safe Transactions
Slide 122
Slide 122 text
Safe Transactions
Keep Transactions Short
- Ideally < 1 Second
Slide 123
Slide 123 text
Safe Transactions
Fast Queries
Slide 124
Slide 124 text
Safe Transactions
Limit # of Queries
- Ideally < 100
Slide 125
Slide 125 text
Safe Transactions
No External Calls
Slide 126
Slide 126 text
Safe Transactions
As Little Code as Possible
Slide 127
Slide 127 text
Safe Transactions
As Little Code as Possible
- Default to after_commit callbacks
Slide 128
Slide 128 text
Safe Transactions
Do You Really Need a Transaction?
Slide 129
Slide 129 text
Safe Transactions
• Keep transactions short
• Fast queries
• Limit # of queries
• No external calls
• As little code as possible
• Do you really need a transaction?
Slide 130
Slide 130 text
Daniel Colson
@composerinteralia
Illustrated by ChangHo Kim and DongBeom Kim