Scaling Shopify

Scaling Shopify

Talk given at ConFoo 2014 on February 28th, 2014.

85b03650a2ec5235376b0b983a49511a?s=128

Christian Joudrey

February 28, 2014
Tweet

Transcript

  1. SCALING SHOPIFY ...or ensuring happiness for online shoppers

  2. cjoudrey   @

  3. None
  4. None
  5. None
  6. None
  7. the stack

  8. nginx unicorn • rails 4 • mysql 5.6 (percona) ruby

    2.1 •
  9. 95 app servers 3,884 unicorn workers 5 job servers 387

    job workers
  10. 1 request 1 process =

  11. scale?

  12. over 90,000 shops

  13. None
  14. 1.6B$ annual GMV that’s 3,600$ per min

  15. cyber monday black friday

  16. None
  17. 61 M$ in GMV in four days

  18. flash sales

  19. None
  20. None
  21. page caching

  22. None
  23. None
  24. shopify/cacheable

  25. generational caching

  26. gzip • etag + 304 not modified

  27. class PostsController < ApplicationController def show response_cache do @post =

    @shop.posts.find(params[:id]) respond_with(@post) end end def cache_key_data { action: action_name, format: request.format, params: params.slice(:id), shop_version: @shop.version } end end
  28. None
  29. flash sale

  30. query caching

  31. shopify/identity_cache

  32. full model caching

  33. opt-in by design

  34. after_commit expiry

  35. class Product < ActiveRecord::Base include IdentityCache has_many :images cache_has_many :images,

    :embed => true end product = Product.fetch(id) images = product.fetch_images
  36. class Product < ActiveRecord::Base include IdentityCache cache_index :shop_id, :handle, :unique

    => true end Product.fetch_by_shop_id_and_handle(shop_id, handle)
  37. None
  38. flash sale

  39. background jobs

  40. webhooks emails • fraud detection • payment processing

  41. None
  42. priority queues payment • default • low realtime •

  43. class ProductImportJob include BackgroundQueue::Realtime def perform(params) ... end end BackgroundQueue.push(ProductImportJob,

    ...)
  44. throttling

  45. the right data store for the job

  46. ephemeral data sessions carts • inventory reservation •

  47. now what?

  48. catching regressions

  49. measure it! if it moves...

  50. statsd

  51. None
  52. Liquid::Template.extend StatsD::Instrument Liquid::Template.statsd_measure :parse, 'Liquid.Template.parse' Liquid::Template.statsd_measure :render, 'Liquid.Template.render'

  53. PaymentProcessingJob.stats_count :perform, 'PaymentProcessingJob.processed'

  54. load testing

  55. None
  56. simulates a flash sale

  57. several times per week

  58. slow queries

  59. # User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify

    Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1
  60. determining root cause

  61. https://github.com/snormore/nginx-x-rid-header nginx request_id header proxy_set_header X-Request-ID "$request_id"; log_format main '...

    $request_id' step 1
  62. https://gist.github.com/mnutt/566725 Complete 200 OK in 100ms (Views: 60ms | ActiveRecord:

    40ms | request_id=bc12813bce...) log_process_action ActionController::Instrumentation step 2
  63. https://github.com/basecamp/marginalia User Load (0.3ms) SELECT `users`.* FROM `users` WHERE `users`.`id`

    = 1 LIMIT 1 /*application:Shopify, controller:users,action:show, request_id:bc12813bce...*/ basecamp/marginalia step 3
  64. # User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify

    Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay, request_id:bc12813bce...*/ profit!
  65. access.log rails.log slow_query.log profit! (2)

  66. schema migration with zero downtime

  67. soundcloud/lhm

  68. current schema new schema

  69. insert/delete/update triggers

  70. INSERT INTO ... SELECT ... insert/delete/update triggers

  71. testing for external calls memcached • mysql • redis net/http

  72. it’s not about preventing it’s about raising awareness

  73. integration test with assert_externals(...) do .. end Unexpected external call

    (mysql): !"" mysql_load("GiftCard") !"" "SELECT `gift_cards`.* FROM `gift_cards` WHERE `gift_cards`.`id` = 1063936318 LIMIT 1" #"" called from: app/services/gift_card_payment_processing.rb: 73:in `block in log_successful'
  74. subscribe('sql.active_record') ActiveSupport::Notifications ["sql.active_record", 2014-02-26 02:38:43 +0000, 2014-02-26 02:38:43 +0000, "a119c5ac2aa6fb4a52fe",

    {:sql=>"SELECT `users`.* FROM `users` LIMIT 1", :name=>"User Load", :connection_id=>69893685920420, :binds=>[]}]
  75. monkey-patch other libs to add instrumentation

  76. thanks! :)