Scaling Shopify

Transcript

1. SCALING SHOPIFY ...or ensuring happiness for online shoppers

2. cjoudrey   @

3. None
4. None
5. None
6. None

7. the stack

8. nginx unicorn • rails 4 • mysql 5.6 (percona) ruby

   2.1 •

9. 95 app servers 3,884 unicorn workers 5 job servers 387

   job workers

10. 1 request 1 process =

11. scale?

12. over 90,000 shops

13. None

14. 1.6B$ annual GMV that’s 3,600$ per min

15. cyber monday black friday

16. None

17. 61 M$ in GMV in four days

18. flash sales

19. None
20. None

21. page caching

22. None
23. None

24. shopify/cacheable

25. generational caching

26. gzip • etag + 304 not modified

27. class PostsController < ApplicationController def show response_cache do @post =

    @shop.posts.find(params[:id]) respond_with(@post) end end def cache_key_data { action: action_name, format: request.format, params: params.slice(:id), shop_version: @shop.version } end end
28. None

29. flash sale

30. query caching

31. shopify/identity_cache

32. full model caching

33. opt-in by design

34. after_commit expiry

35. class Product < ActiveRecord::Base include IdentityCache has_many :images cache_has_many :images,

    :embed => true end product = Product.fetch(id) images = product.fetch_images

36. class Product < ActiveRecord::Base include IdentityCache cache_index :shop_id, :handle, :unique

    => true end Product.fetch_by_shop_id_and_handle(shop_id, handle)
37. None

38. flash sale

39. background jobs

40. webhooks emails • fraud detection • payment processing

41. None

42. priority queues payment • default • low realtime •

43. class ProductImportJob include BackgroundQueue::Realtime def perform(params) ... end end BackgroundQueue.push(ProductImportJob,

    ...)

44. throttling

45. the right data store for the job

46. ephemeral data sessions carts • inventory reservation •

47. now what?

48. catching regressions

49. measure it! if it moves...

50. statsd

51. None

52. Liquid::Template.extend StatsD::Instrument Liquid::Template.statsd_measure :parse, 'Liquid.Template.parse' Liquid::Template.statsd_measure :render, 'Liquid.Template.render'

53. PaymentProcessingJob.stats_count :perform, 'PaymentProcessingJob.processed'

54. load testing

55. None

56. simulates a flash sale

57. several times per week

58. slow queries

59. # User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify

    Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1

60. determining root cause

61. https://github.com/snormore/nginx-x-rid-header nginx request_id header proxy_set_header X-Request-ID "$request_id"; log_format main '...

    $request_id' step 1

62. https://gist.github.com/mnutt/566725 Complete 200 OK in 100ms (Views: 60ms | ActiveRecord:

    40ms | request_id=bc12813bce...) log_process_action ActionController::Instrumentation step 2

63. https://github.com/basecamp/marginalia User Load (0.3ms) SELECT `users`.* FROM `users` WHERE `users`.`id`

    = 1 LIMIT 1 /*application:Shopify, controller:users,action:show, request_id:bc12813bce...*/ basecamp/marginalia step 3

64. # User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify

    Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay, request_id:bc12813bce...*/ profit!

65. access.log rails.log slow_query.log profit! (2)

66. schema migration with zero downtime

67. soundcloud/lhm

68. current schema new schema

69. insert/delete/update triggers

70. INSERT INTO ... SELECT ... insert/delete/update triggers

71. testing for external calls memcached • mysql • redis net/http

    •

72. it’s not about preventing it’s about raising awareness

73. integration test with assert_externals(...) do .. end Unexpected external call

    (mysql): !"" mysql_load("GiftCard") !"" "SELECT `gift_cards`.* FROM `gift_cards` WHERE `gift_cards`.`id` = 1063936318 LIMIT 1" #"" called from: app/services/gift_card_payment_processing.rb: 73:in `block in log_successful'

74. subscribe('sql.active_record') ActiveSupport::Notifications ["sql.active_record", 2014-02-26 02:38:43 +0000, 2014-02-26 02:38:43 +0000, "a119c5ac2aa6fb4a52fe",

    {:sql=>"SELECT `users`.* FROM `users` LIMIT 1", :name=>"User Load", :connection_id=>69893685920420, :binds=>[]}]

75. monkey-patch other libs to add instrumentation

76. thanks! :)