Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scaling Rails for Black Friday / Cyber Monday at Shopify
Search
Christian Joudrey
February 18, 2015
Technology
6
5.6k
Scaling Rails for Black Friday / Cyber Monday at Shopify
Talk given at ConFoo 2015 on February 18th, 2015 and RailsConf 2015.
Christian Joudrey
February 18, 2015
Tweet
Share
More Decks by Christian Joudrey
See All by Christian Joudrey
Writing NES games! with assembly!!
cjoudrey
1
670
Developing at Scale
cjoudrey
3
460
Tips and Tricks from Shopify's codebase
cjoudrey
2
550
Scaling Shopify
cjoudrey
3
520
#pairwithme
cjoudrey
3
230
Two-factor authentication
cjoudrey
4
380
Automate your Infrastructure with Chef
cjoudrey
9
570
Other Decks in Technology
See All in Technology
AWSで”最小権限の原則”を実現するための考え方 /20240722-ssmjp-aws-least-privilege
opelab
10
4.4k
コンテナ・K8s研修 - 後半 Kubernetes 基礎&ハンズオン【MIXI 24新卒技術研修】
mixi_engineers
PRO
1
120
E2Eテスト自動化プラットフォームにおけるAIの活用
shift_evolve
0
190
シフトレフトで挑む セキュリティの生産性向上
sekido
PRO
0
270
ゆめみのアクセシビリティの現在地と今後
ryokatsuse
3
290
[I/O Extended Android 2024] What`s new in Android 2024
kyeongwan
0
220
dxd2024-生成AIに振り回された3か月間の成功と失敗/dxd2024-link-and-motivation
lmi
2
260
推薦システムを本番導入する上で一番優先すべきだったこと~NewsPicks記事推薦機能の改善事例を元に~
morinota
0
130
テスト・設計研修【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
170
「単なる OAuth 2.0 を認証に使うと、車が通れるほどのどでかいセキュリティー・ホールができる」のか検証してみた
terara
0
380
開発生産性をむしろ向上させる セキュリティパートナーの作り方 / Dev Productivity Con 2024
flatt_security
0
380
CTOから見た事業開発とプロダクト開発 / My Perspective on Business and Product Development as CTO
keisuke69
4
960
Featured
See All Featured
VelocityConf: Rendering Performance Case Studies
addyosmani
321
23k
Build The Right Thing And Hit Your Dates
maggiecrowley
28
2.2k
GraphQLとの向き合い方2022年版
quramy
36
13k
[RailsConf 2023] Rails as a piece of cake
palkan
35
4.4k
Gamification - CAS2011
davidbonilla
78
4.9k
The Mythical Team-Month
searls
217
43k
Practical Orchestrator
shlominoach
185
10k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
105
6.8k
The Invisible Side of Design
smashingmag
294
50k
How to Think Like a Performance Engineer
csswizardry
4
590
Git: the NoSQL Database
bkeepers
PRO
423
64k
How to train your dragon (web standard)
notwaldorf
79
5.5k
Transcript
scaling rails for Black Friday Cyber Monday
cjoudrey @
None
None
None
None
None
the stack
nginx unicorn • rails 4 • mysql 5.6 (percona) ruby
2.1 •
95 app servers 1,800 unicorn workers 18 job servers 1,400
job workers
scale?
None
400,000 reqs/min
3.7B$ annual GMV that’s 7,000$ per min
Black Friday Cyber Monday
Black Friday Cyber FUNday
None
~ 600,000 reqs/min
1 request 1 process =
scale++ ↓ resp. time ↑ workers
page caching
None
None
shopify/cacheable
generational caching
gzip • etag + 304 not modified
class PostsController < ApplicationController def index response_cache do @posts =
@shop.posts.paginate(params[:page]) respond_with(@posts) end end def cache_key_data { shop_id: @shop.id, path: request.path, format: request.format, params: params.slice(:page), shop_version: @shop.version } end end
md5( { shop_id: 1, path: '/posts', format: 'text/html', params: {
page: 2 }, shop_version: 123 }.to_s ) GET /posts?page=2
None
sale
query caching
shopify/identity_cache
full model caching
opt-in by design
after_commit expiry
class Product < ActiveRecord::Base include IdentityCache has_many :images cache_has_many :images,
:embed => true end product = Product.fetch(id) images = product.fetch_images
class Product < ActiveRecord::Base include IdentityCache cache_index :shop_id, :handle, :unique
=> true end Product.fetch_by_shop_id_and_handle(shop_id, handle)
None
sale
background jobs
webhooks emails • fraud detection • payment processing
None
priority queues payment • default • low realtime •
throttling
now what?
measure it! if it moves...
statsd
shopify/statsd-instrument
Liquid::Template.extend StatsD::Instrument Liquid::Template.statsd_measure :render, 'Liquid.Template.render'
PaymentProcessingJob.statsd_count :perform, 'PaymentProcessingJob.processed'
None
None
load testing
genghis khan
simulate Black Friday Cyber Monday before it happens
several times per week
slow queries
# User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify
Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1
determining root cause
https://github.com/newobj/nginx-x-rid-header nginx request_id header proxy_set_header X-Request-ID "$request_id"; log_format main '...
$request_id' step 1
https://gist.github.com/mnutt/566725 Complete 200 OK in 100ms (Views: 60ms | ActiveRecord:
40ms | request_id=bc12813bce...) log_process_action ActionController::Instrumentation step 2
https://github.com/basecamp/marginalia User Load (0.3ms) SELECT `users`.* FROM `users` WHERE `users`.`id`
= 1 LIMIT 1 /*application:Shopify, controller:users,action:show, request_id:bc12813bce...*/ basecamp/marginalia step 3
# User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify
Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay, request_id:bc12813bce...*/ profit!
access.log rails.log slow_query.log profit! (2)
bonus! background jobs
schema migration with zero downtime
soundcloud/lhm
class NewIndexOnOrders < ActiveRecord::Migration def self.up Lhm.change_table :orders do |m|
m.add_index [:shop_id, :customer_id] end end def self.down # end end
orders 20140520_orders
insert/delete/update triggers
INSERT INTO ... SELECT ... insert/delete/update triggers
async caveat
resiliency
A resilient system is one that functions with one or
more components being unavailable or unacceptably slow. -
don’t let a take you down minor dependencies
shopify/toxiproxy ☢ redis memcached sessions your app toxiproxy
def load_customer if customer_id = session[:customer_id] @customer = Customer.find_by_id(customer_id) end
end
def load_customer if customer_id = session[:customer_id] @customer = Customer.find_by_id(customer_id) end
rescue Sessions::DataStoreUnavailable @customer = nil end
def test_storefront_resilient_to_sessions_down Toxiproxy[:sessions_data_store].down do get '/' assert_response :success end end
rinse & repeat http://www.shopify.com/technology/16906928-building-and-testing-resilient-ruby-on-rails-applications
what about resources? slow
shard 2 shard 3 shard 1 shop 4, 5, 6
shop 7, 8, 9 shop 1, 2, 3
rails request shard 2 shard 3 shard 1 shop 4,
5, 6 shop 7, 8, 9 shop 1, 2, 3
rails request shard 2 shard 3 shard 1 shop 4,
5, 6 shop 7, 8, 9 shop 1, 2, 3
how can we fail fast?
shopify/semian smart circuit-breaker
shopify/semian Semian.register(:mysql_shard_1, tickets: 5, timeout: 0.5, error_threshold: 100, error_timeout: 10,
success_threshold: 2)
shopify/semian Semian[:mysql_shard_1].acquire do # Query the resource end
rails request shard 2 shard 3 shard 1 shop 4,
5, 6 shop 7, 8, 9 shop 1, 2, 3
what else can go wrong?
Shopify Shipping rate providers Payment gateways Fulfillment services (FedEX, UPS,
USPS, etc..) (Stripe, PayPal, etc..) (Shipwire, etc…) Internal services (MySQL, Memcached, etc..)
manual circuit breakers around external dependencies
thanks! :)