Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scaling Shopify
Search
Christian Joudrey
February 28, 2014
Technology
3
530
Scaling Shopify
Talk given at ConFoo 2014 on February 28th, 2014.
Christian Joudrey
February 28, 2014
Tweet
Share
More Decks by Christian Joudrey
See All by Christian Joudrey
Writing NES games! with assembly!!
cjoudrey
1
740
Developing at Scale
cjoudrey
3
480
Scaling Rails for Black Friday / Cyber Monday at Shopify
cjoudrey
6
5.9k
Tips and Tricks from Shopify's codebase
cjoudrey
2
560
#pairwithme
cjoudrey
3
250
Two-factor authentication
cjoudrey
4
390
Automate your Infrastructure with Chef
cjoudrey
9
610
Other Decks in Technology
See All in Technology
ZOZOマッチのアーキテクチャと技術構成
zozotech
PRO
2
880
Snowflakeの生成AI機能を活用したデータ分析アプリの作成 〜Cortex AnalystとCortex Searchの活用とStreamlitアプリでの利用〜
nayuts
0
120
生成AI時代に必要な価値ある意思決定を育てる「開発プロセス定義」を用いた中期戦略
kakehashi
PRO
1
240
個人CLAUDE.md紹介と設定から学んだこと/introduce-my-claude-md
shibayu36
0
140
人と組織に偏重したEMへのアンチテーゼ──なぜ、EMに設計力が必要なのか/An antithesis to the overemphasis of people and organizations in EM
dskst
7
840
プロダクトの成長に合わせたアーキテクチャの段階的進化と成長痛、そして、ユニットエコノミクスの最適化
kakehashi
PRO
1
110
実践データベース設計 ①データベース設計概論
recruitengineers
PRO
4
1.9k
AIエージェントの活用に重要な「MCP (Model Context Protocol)」とは何か
masayamoriofficial
0
240
サポートエンジニアから見たRancher運用の現場
masap
0
110
役割は変わっても、変わらないもの 〜スクラムマスターからEMへの転身で学んだ信頼構築の本質〜 / How to build trust
shinop
0
150
PRDの正しい使い方 ~AI時代にも効く思考・対話・成長ツールとして~
techtekt
PRO
0
230
トヨタ生産方式(TPS)入門
recruitengineers
PRO
5
1.4k
Featured
See All Featured
Raft: Consensus for Rubyists
vanstee
140
7.1k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
910
Rails Girls Zürich Keynote
gr2m
95
14k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.5k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.9k
Designing Experiences People Love
moore
142
24k
[RailsConf 2023] Rails as a piece of cake
palkan
56
5.8k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
Building Applications with DynamoDB
mza
96
6.6k
Measuring & Analyzing Core Web Vitals
bluesmoon
9
570
Transcript
SCALING SHOPIFY ...or ensuring happiness for online shoppers
cjoudrey @
None
None
None
None
the stack
nginx unicorn • rails 4 • mysql 5.6 (percona) ruby
2.1 •
95 app servers 3,884 unicorn workers 5 job servers 387
job workers
1 request 1 process =
scale?
over 90,000 shops
None
1.6B$ annual GMV that’s 3,600$ per min
cyber monday black friday
None
61 M$ in GMV in four days
flash sales
None
None
page caching
None
None
shopify/cacheable
generational caching
gzip • etag + 304 not modified
class PostsController < ApplicationController def show response_cache do @post =
@shop.posts.find(params[:id]) respond_with(@post) end end def cache_key_data { action: action_name, format: request.format, params: params.slice(:id), shop_version: @shop.version } end end
None
flash sale
query caching
shopify/identity_cache
full model caching
opt-in by design
after_commit expiry
class Product < ActiveRecord::Base include IdentityCache has_many :images cache_has_many :images,
:embed => true end product = Product.fetch(id) images = product.fetch_images
class Product < ActiveRecord::Base include IdentityCache cache_index :shop_id, :handle, :unique
=> true end Product.fetch_by_shop_id_and_handle(shop_id, handle)
None
flash sale
background jobs
webhooks emails • fraud detection • payment processing
None
priority queues payment • default • low realtime •
class ProductImportJob include BackgroundQueue::Realtime def perform(params) ... end end BackgroundQueue.push(ProductImportJob,
...)
throttling
the right data store for the job
ephemeral data sessions carts • inventory reservation •
now what?
catching regressions
measure it! if it moves...
statsd
None
Liquid::Template.extend StatsD::Instrument Liquid::Template.statsd_measure :parse, 'Liquid.Template.parse' Liquid::Template.statsd_measure :render, 'Liquid.Template.render'
PaymentProcessingJob.stats_count :perform, 'PaymentProcessingJob.processed'
load testing
None
simulates a flash sale
several times per week
slow queries
# User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify
Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1
determining root cause
https://github.com/snormore/nginx-x-rid-header nginx request_id header proxy_set_header X-Request-ID "$request_id"; log_format main '...
$request_id' step 1
https://gist.github.com/mnutt/566725 Complete 200 OK in 100ms (Views: 60ms | ActiveRecord:
40ms | request_id=bc12813bce...) log_process_action ActionController::Instrumentation step 2
https://github.com/basecamp/marginalia User Load (0.3ms) SELECT `users`.* FROM `users` WHERE `users`.`id`
= 1 LIMIT 1 /*application:Shopify, controller:users,action:show, request_id:bc12813bce...*/ basecamp/marginalia step 3
# User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify
Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay, request_id:bc12813bce...*/ profit!
access.log rails.log slow_query.log profit! (2)
schema migration with zero downtime
soundcloud/lhm
current schema new schema
insert/delete/update triggers
INSERT INTO ... SELECT ... insert/delete/update triggers
testing for external calls memcached • mysql • redis net/http
•
it’s not about preventing it’s about raising awareness
integration test with assert_externals(...) do .. end Unexpected external call
(mysql): !"" mysql_load("GiftCard") !"" "SELECT `gift_cards`.* FROM `gift_cards` WHERE `gift_cards`.`id` = 1063936318 LIMIT 1" #"" called from: app/services/gift_card_payment_processing.rb: 73:in `block in log_successful'
subscribe('sql.active_record') ActiveSupport::Notifications ["sql.active_record", 2014-02-26 02:38:43 +0000, 2014-02-26 02:38:43 +0000, "a119c5ac2aa6fb4a52fe",
{:sql=>"SELECT `users`.* FROM `users` LIMIT 1", :name=>"User Load", :connection_id=>69893685920420, :binds=>[]}]
monkey-patch other libs to add instrumentation
thanks! :)