Scaling at Shopify

Présentation donné au Web à Québec en 2015 sur le scaling de Shopify

Transcript

1. SCALING AT SHOPIFY OU COMMENT J’AI APPRIS À NE PLUS M’EN FAIRE ET AIMER LA BOMBE
 Web à Québec 2015
2. @jpcaissy
10. Ventes totales en 2015 par les marchands sur Shopify ou 3,7 Instagrams ou 7000 $/min 3 700 000 000 $USD
11. ~10 000 requêtes/sec Black Friday & Cyber Monday
12. ~10 000 requêtes/sec Black Friday & Cyber Sunday
13. Pics de traffic? Flash sales!
14. Ventes éclairs
16. L’infrastructure
17. nginx ruby 2.1 rails 4 unicorn mysql 5.6 percona redis memcached
18. 95 app servers 1800 unicorn workers 18 job servers 1400 job workers
19. Scale ?
20. 1 web request = 1 running process
21. response time workers #webscale
22. Caching
23. Generational page caching Shopify/cacheable
24. gzip
 &
 ETag + 304 Not Modified
25. ETag or entity tag
26. $ curl -si http://crafted.shopify.com | grep -i ETag ETag: cacheable:b285a5ea6cbc5d9f0e95425c5ae1e6e6 $ curl -si -H 'If-None-Match: cacheable:b285a5ea6cbc5d9f0e95425c5ae1e6e6' \ http://crafted.shopify.com HTTP/1.1 304 Not Modified $ curl -si -H 'If-None-Match: cacheable:b285a5ea6cbc5d9f0e95425c5ae1e6e6' \ http://crafted.shopify.com HTTP/1.1 200 Ok
27. class CollectionController < ApplicationController def show response_cache do @collections = @shop.collections.paginate(params[:page]) respond_with(@collections) end end def cache_key_data { shop_id: @shop.id, path: request.path, format: request.format, params: params.slice(:page), shop_version: @shop.version } end end
28. class CollectionController < ApplicationController def show response_cache do @collections = @shop.collections.paginate(params[:page]) respond_with(@collections) end end def cache_key_data { shop_id: @shop_id, path: request.path, format: request.format, params: params.slice(:page), shop_version: @shop.version } end end
29. md5( { shop_id: 1, path: ‘/collection/unicorn-tshirts’, format: ‘html’, params: { page: 2 } shop_version: 321 } ) # ‘b285a5ea6cbc5d9f0e95425c5ae1e6e6’ GET /collection/unicorn-tshirts?page=2
31. Vente éclair
32. Query caching Shopify/identity_cache
33. full model caching opt-in by design plug and play™
34. class Product < ActiveRecord::Base include IdentityCache has_many :images cache_has_many :images, :embed => true end # Fetch the product by its id, the primary index. @product = Product.fetch(id) # Fetch the images for the Product. # Images are embedded so the fetch would have already loaded them. @images = @product.fetch_images
35. Indexes secondaires
36. class Product < ActiveRecord::Base include IdentityCache cache_index :handle, :unique => true cache_index :vendor, :product_type end # Fetch the product from the cache by the index. # If the object isn't in the cache it is pulled from the db and stored in the cache. product = Product.fetch_by_handle(handle) products = Product.fetch_by_vendor_and_product_type(vendor, type)
37. Attribute caching
38. class Redirect < ActiveRecord::Base cache_attribute :target, :by => [:shop_id, :path] end Redirect.fetch_target_by_shop_id_and_path(shop_id, path)
40. Background jobs
41. webhooks emails fraud detection payment processing
43. Système de priorité Défaut Basse Paiement Temps réel
44. Goulot d’étranglement intégré !
45. Métriques
46. StatsD Agrégateur de statistiques tel que des compteurs ou chronomètres
47. Shopify/statsd-instrument
48. # You can pass a key and a ms value StatsD.measure(‘Stripe.ExternalCall’, 2.5) # or more commonly pass a block that calls your code StatsD.measure(‘Stripe.ExternalCall') do Stripe.charge(order) end
49. # increment default to +1 StatsD.increment(‘FailedPayment') # you can also specify how much to increment the key by StatsD.increment('FailedPayment', 10) # you can also specify a sample rate, so only 1/10 of events # actually get to statsd. Useful for very high volume data StatsD.increment(‘FailedPayment', 1, sample_rate: 0.1)
50. PaymentProcessing.extend StatsD::Instrument PaymentProcessing.statsd_messure :perform, 'PaymentProcessing.Charge'
51. PaymentProcessing.extend StatsD::Instrument PaymentProcessing.statsd_count :perform, 'PaymentProcessing.Charge'
53. Logging
54. Slow queries
55. # User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1
56. access.log rails.log slow_query.log
57. nginx request_id header newobj/nginx-x-rid-header
58. log_format main ‘[…] $request_id’; server { listen 80; server_name example.com; location / { proxy_set_header X-Request-ID $request_id; proxy_pass http://localhost:8080; } }
59. log_process_action ActionController::Instrumentation Complete 200 OK in 100ms (Views: 60ms | ActiveRecord: 40ms | request_id=9e107d9d…)
60. Attach comments to SQL queries
61. Account Load (0.3ms) SELECT `products`.* FROM `products` WHERE `products`.`id` = 1234567890 LIMIT 1 Account Load (0.3ms) SELECT `products`.* FROM `products` WHERE `products`.`id` = 1234567890 LIMIT 1 /* application:Shopify,controller:products,action:show, request_id=9e107d9d372bb6826bd81d3542a419d6 */
62. basecamp/marginala
63. # User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1 /* application:Shopify,controller:discounts,action:show, request_id=9e107d9d372bb6826bd81d3542a419d6 */
64. access.log rails.log slow_query.log
65. Schema migration with ZERO downtime
66. Large Hadron Migrations Soundcloud/lhm
67. class AddOrderApiClientId < LhmMigration change_table :orders do |m| m.add_column :api_client_id, "BIGINT(20)" end revert_table do |m| m.remove_column :api_client_id end end
68. orders 20150316_orders insert/update/delete triggers INSERT INTO … SELECT Atomic rename of the tables
69. Load testing
70. Simuler un traffic du Black Friday Plusieurs fois par semaine
71. Yes, we DDoS ourselves!
72. Résilience
73. Resilience “The capacity to recover quickly from difficulties; toughness.”
74. Permettre à votre application de supporter quelques interruptions mineurs.
76. Section A Section B Section C Datastore A Down Down Degraded Datastore B Up Down Up Message queue A Degraded Up Down External HTTP API A Down Up Degraded
77. Section A Section B Section C Datastore A Down Up Degraded Datastore B Up Degraded Up Message queue A Up Up Up External HTTP API A Degraded Up Up
78. def load_customer if customer_id = session[:customer_id] @customer = Customer.find_by_id(customer_id) end end
79. def load_customer if customer_id = session[:customer_id] @customer = Customer.find_by_id(session[:customer_id]) end rescue Sessions::DataStoreUnavailable flash[:notice] = 'Customer sign in is currently unavailable' @customer = nil end
80. Toxiproxy Shopify/toxiproxy
81. App Toxiproxy Redis Memcached Sesssions
83. def test_storefront_resilient_to_session Toxiproxy[:session_datastore].down do get '/' assert_response :ok end end
84. Catching exceptions Doesn’t scale well. Use wrappers or decorators instead
85. Resources lentes
86. Disjoncteur intelligent
87. Semian The smart circuit-breaker Shopify/semian
88. Circuit breaker pattern
89. client = Redis.new(semian: { name: "sessions", tickets: 4, success_threshold: 2, error_threshold: 4, error_timeout: 20 }) client.acquire do #query Redis end
90. Resources externes lentes
91. Shipping rate providers (FedEx, UPS, USPS, etc) Payment gateways (Stripe, Paypal, etc) Fulfillment services (Shipwire, etc)
92. Disjoncteurs manuel
93. Comment j’en suis arrivé à aimer la bombe
94. Game day The doomsday machine https://stripe.com/blog/game-day-exercises-at-stripe
95. kill -9 `pgrep redis-server`
96. 1. Regrouper votre équipe pour une journée 2. Dessinez votre infrastructure 3. Élaborer des scénarios de défaillance 4. Pour chaque scénario, poser une ou plusieurs hypothèses 5. Effectuer une sauvegarde des données 6. Induire la défaillance dans le système
97. Questions ?