Tweet
Tweet

Slide 1

Slide 1 text

SCALING SHOPIFY ...or ensuring happiness for online shoppers

Slide 2

Slide 2 text

cjoudrey   @

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

the stack

Slide 8

Slide 8 text

nginx unicorn • rails 4 • mysql 5.6 (percona) ruby 2.1 •

Slide 9

Slide 9 text

95 app servers 3,884 unicorn workers 5 job servers 387 job workers

Slide 10

Slide 10 text

1 request 1 process =

Slide 11

Slide 11 text

scale?

Slide 12

Slide 12 text

over 90,000 shops

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

1.6B$ annual GMV that’s 3,600$ per min

Slide 15

Slide 15 text

cyber monday black friday

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

61 M$ in GMV in four days

Slide 18

Slide 18 text

flash sales

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

page caching

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

shopify/cacheable

Slide 25

Slide 25 text

generational caching

Slide 26

Slide 26 text

gzip • etag + 304 not modified

Slide 27

Slide 27 text

class PostsController < ApplicationController def show response_cache do @post = @shop.posts.find(params[:id]) respond_with(@post) end end def cache_key_data { action: action_name, format: request.format, params: params.slice(:id), shop_version: @shop.version } end end

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

flash sale

Slide 30

Slide 30 text

query caching

Slide 31

Slide 31 text

shopify/identity_cache

Slide 32

Slide 32 text

full model caching

Slide 33

Slide 33 text

opt-in by design

Slide 34

Slide 34 text

after_commit expiry

Slide 35

Slide 35 text

class Product < ActiveRecord::Base include IdentityCache has_many :images cache_has_many :images, :embed => true end product = Product.fetch(id) images = product.fetch_images

Slide 36

Slide 36 text

class Product < ActiveRecord::Base include IdentityCache cache_index :shop_id, :handle, :unique => true end Product.fetch_by_shop_id_and_handle(shop_id, handle)

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

flash sale

Slide 39

Slide 39 text

background jobs

Slide 40

Slide 40 text

webhooks emails • fraud detection • payment processing

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

priority queues payment • default • low realtime •

Slide 43

Slide 43 text

class ProductImportJob include BackgroundQueue::Realtime def perform(params) ... end end BackgroundQueue.push(ProductImportJob, ...)

Slide 44

Slide 44 text

throttling

Slide 45

Slide 45 text

the right data store for the job

Slide 46

Slide 46 text

ephemeral data sessions carts • inventory reservation •

Slide 47

Slide 47 text

now what?

Slide 48

Slide 48 text

catching regressions

Slide 49

Slide 49 text

measure it! if it moves...

Slide 50

Slide 50 text

statsd

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

Liquid::Template.extend StatsD::Instrument Liquid::Template.statsd_measure :parse, 'Liquid.Template.parse' Liquid::Template.statsd_measure :render, 'Liquid.Template.render'

Slide 53

Slide 53 text

PaymentProcessingJob.stats_count :perform, 'PaymentProcessingJob.processed'

Slide 54

Slide 54 text

load testing

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

simulates a flash sale

Slide 57

Slide 57 text

several times per week

Slide 58

Slide 58 text

slow queries

Slide 59

Slide 59 text

# User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1

Slide 60

Slide 60 text

determining root cause

Slide 61

Slide 61 text

https://github.com/snormore/nginx-x-rid-header nginx request_id header proxy_set_header X-Request-ID "$request_id"; log_format main '... $request_id' step 1

Slide 62

Slide 62 text

https://gist.github.com/mnutt/566725 Complete 200 OK in 100ms (Views: 60ms | ActiveRecord: 40ms | request_id=bc12813bce...) log_process_action ActionController::Instrumentation step 2

Slide 63

Slide 63 text

https://github.com/basecamp/marginalia User Load (0.3ms) SELECT `users`.* FROM `users` WHERE `users`.`id` = 1 LIMIT 1 /*application:Shopify, controller:users,action:show, request_id:bc12813bce...*/ basecamp/marginalia step 3

Slide 64

Slide 64 text

# User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay, request_id:bc12813bce...*/ profit!

Slide 65

Slide 65 text

access.log rails.log slow_query.log profit! (2)

Slide 66

Slide 66 text

schema migration with zero downtime

Slide 67

Slide 67 text

soundcloud/lhm

Slide 68

Slide 68 text

current schema new schema

Slide 69

Slide 69 text

insert/delete/update triggers

Slide 70

Slide 70 text

INSERT INTO ... SELECT ... insert/delete/update triggers

Slide 71

Slide 71 text

testing for external calls memcached • mysql • redis net/http •

Slide 72

Slide 72 text

it’s not about preventing it’s about raising awareness

Slide 73

Slide 73 text

integration test with assert_externals(...) do .. end Unexpected external call (mysql): !"" mysql_load("GiftCard") !"" "SELECT `gift_cards`.* FROM `gift_cards` WHERE `gift_cards`.`id` = 1063936318 LIMIT 1" #"" called from: app/services/gift_card_payment_processing.rb: 73:in `block in log_successful'

Slide 74

Slide 74 text

subscribe('sql.active_record') ActiveSupport::Notifications ["sql.active_record", 2014-02-26 02:38:43 +0000, 2014-02-26 02:38:43 +0000, "a119c5ac2aa6fb4a52fe", {:sql=>"SELECT `users`.* FROM `users` LIMIT 1", :name=>"User Load", :connection_id=>69893685920420, :binds=>[]}]

Slide 75

Slide 75 text

monkey-patch other libs to add instrumentation

Slide 76

Slide 76 text

thanks! :)