$30 off During Our Annual Pro Sale. View Details »

Scaling Shopify

Scaling Shopify

Talk given at ConFoo 2014 on February 28th, 2014.

Christian Joudrey

February 28, 2014
Tweet

More Decks by Christian Joudrey

Other Decks in Technology

Transcript

  1. SCALING SHOPIFY ...or ensuring happiness for online shoppers

  2. cjoudrey   @

  3. None
  4. None
  5. None
  6. None
  7. the stack

  8. nginx unicorn • rails 4 • mysql 5.6 (percona) ruby

    2.1 •
  9. 95 app servers 3,884 unicorn workers 5 job servers 387

    job workers
  10. 1 request 1 process =

  11. scale?

  12. over 90,000 shops

  13. None
  14. 1.6B$ annual GMV that’s 3,600$ per min

  15. cyber monday black friday

  16. None
  17. 61 M$ in GMV in four days

  18. flash sales

  19. None
  20. None
  21. page caching

  22. None
  23. None
  24. shopify/cacheable

  25. generational caching

  26. gzip • etag + 304 not modified

  27. class PostsController < ApplicationController def show response_cache do @post =

    @shop.posts.find(params[:id]) respond_with(@post) end end def cache_key_data { action: action_name, format: request.format, params: params.slice(:id), shop_version: @shop.version } end end
  28. None
  29. flash sale

  30. query caching

  31. shopify/identity_cache

  32. full model caching

  33. opt-in by design

  34. after_commit expiry

  35. class Product < ActiveRecord::Base include IdentityCache has_many :images cache_has_many :images,

    :embed => true end product = Product.fetch(id) images = product.fetch_images
  36. class Product < ActiveRecord::Base include IdentityCache cache_index :shop_id, :handle, :unique

    => true end Product.fetch_by_shop_id_and_handle(shop_id, handle)
  37. None
  38. flash sale

  39. background jobs

  40. webhooks emails • fraud detection • payment processing

  41. None
  42. priority queues payment • default • low realtime •

  43. class ProductImportJob include BackgroundQueue::Realtime def perform(params) ... end end BackgroundQueue.push(ProductImportJob,

    ...)
  44. throttling

  45. the right data store for the job

  46. ephemeral data sessions carts • inventory reservation •

  47. now what?

  48. catching regressions

  49. measure it! if it moves...

  50. statsd

  51. None
  52. Liquid::Template.extend StatsD::Instrument Liquid::Template.statsd_measure :parse, 'Liquid.Template.parse' Liquid::Template.statsd_measure :render, 'Liquid.Template.render'

  53. PaymentProcessingJob.stats_count :perform, 'PaymentProcessingJob.processed'

  54. load testing

  55. None
  56. simulates a flash sale

  57. several times per week

  58. slow queries

  59. # User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify

    Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1
  60. determining root cause

  61. https://github.com/snormore/nginx-x-rid-header nginx request_id header proxy_set_header X-Request-ID "$request_id"; log_format main '...

    $request_id' step 1
  62. https://gist.github.com/mnutt/566725 Complete 200 OK in 100ms (Views: 60ms | ActiveRecord:

    40ms | request_id=bc12813bce...) log_process_action ActionController::Instrumentation step 2
  63. https://github.com/basecamp/marginalia User Load (0.3ms) SELECT `users`.* FROM `users` WHERE `users`.`id`

    = 1 LIMIT 1 /*application:Shopify, controller:users,action:show, request_id:bc12813bce...*/ basecamp/marginalia step 3
  64. # User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify

    Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay, request_id:bc12813bce...*/ profit!
  65. access.log rails.log slow_query.log profit! (2)

  66. schema migration with zero downtime

  67. soundcloud/lhm

  68. current schema new schema

  69. insert/delete/update triggers

  70. INSERT INTO ... SELECT ... insert/delete/update triggers

  71. testing for external calls memcached • mysql • redis net/http

  72. it’s not about preventing it’s about raising awareness

  73. integration test with assert_externals(...) do .. end Unexpected external call

    (mysql): !"" mysql_load("GiftCard") !"" "SELECT `gift_cards`.* FROM `gift_cards` WHERE `gift_cards`.`id` = 1063936318 LIMIT 1" #"" called from: app/services/gift_card_payment_processing.rb: 73:in `block in log_successful'
  74. subscribe('sql.active_record') ActiveSupport::Notifications ["sql.active_record", 2014-02-26 02:38:43 +0000, 2014-02-26 02:38:43 +0000, "a119c5ac2aa6fb4a52fe",

    {:sql=>"SELECT `users`.* FROM `users` LIMIT 1", :name=>"User Load", :connection_id=>69893685920420, :binds=>[]}]
  75. monkey-patch other libs to add instrumentation

  76. thanks! :)