How Shopify Scales Rails

This talk goes into detail about how Shopify has been using and scaling rails up to now.

Transcript

1. shopi fy How Shopify Scales Rails John Duff
2. What is Shopify?
6. The Stack
7. • Ruby 1.9.3-p327 • Rails 3.2 • Percona MySQL 5.5 • Unicorn 4.5 • Memcached 1.4.14 • Redis 2.6 The Stack
8. The Stack • 33 App Servers • 1172 Unicorn Workers • 5 Job Servers • 370 Job Workers Nginx Unicorn Rails 3.2 Ruby 1.9.3- p327
9. The Stack Firewall Load Balancer App Servers Redis Job Servers Database Memcached Search
10. • 55,873 Lines of application code • 81,892 Lines of test code • 211 Controllers • 468 Models The Stack
11. Current Scale
12. 9.9 M Orders An order every 3.2 seconds
13. 2,008 Sales per Minute Cyber Monday
14. 50,000 RPM 45 ms response time
15. 13.3 billion requests
16. Looking Back, to Look Ahead
17. • First line of code written in 2004 • Shopify released July, 2005 • Same codebase • Over 9 years of Rails upgrades, improvements and changes Looking Back, to Look Ahead
18. Looking Back, to Look Ahead • 6,702 Lines of application code (55,873) • 4,386 Lines of test code (81,892) • 38 Controllers (211) • 77 Models (468)
19. Looking Back, to Look Ahead • Ruby 1.8.2 • Rails 0.13.1 • MySQL 4.1 • Lighttpd • Memcached
20. Know The System
21. One Request, One Process
22. RPM = W * 1/R
23. RPM = 1172 * 60 / 0.072
24. 976,666 = 1172 * 60 / 0.072
25. ↑ Workers ↓ Response Time
26. Know The System • Avoid network calls during requests • Speed up unavoidable network calls • The Storefront and Checkout • The Chive
27. RPM Chive Flash Sale
28. Measure ALL THE THINGS
29. Measure ALL THE THINGS • New Relic • Splunk • StatsD • Cacti • Conan
30. New Relic
31. Splunk
34. Caching
35. cacheable
36. cacheable • https://github.com/Shopify/cacheable • serve gzip’d content • generational caching • no explicit expiry
37. cacheable class PostsController < ApplicationController def show response_cache do @post = @shop.posts.find(params[:id]) respond_with(@post) end end def cache_key_data { :action => action_name, :format => request.format, :params => params.slice(:id), :shop_version => @shop.version } end end
38. requests Caching 404s
39. Identity Cache
40. Identity Cache • cache full model objects in memcached • can include associated objects in cache • must opt in to the cache • explicit, but automatic expiry
41. Identity Cache class Product < ActiveRecord::Base include IdentityCache has_many :images cache_index [:shop_id, :id] cache_has_many :images, :embed => true end @product = Product.fetch_by_shop_id_and_id(shop_id, id) @images = @product.fetch_images
42. Identity Cache
43. Get Out of My Process
44. Delayed Job • Jobs stored in the db • Workers run in their own process • Workers poll for jobs periodically • https://github.com/collectiveidea/delayed_job
45. Resque • Redis backed • No more db contention • Faster (300 jobs/sec vs 120 jobs/sec) • Extensible • https://github.com/defunkt/resque
46. Resque • Sending Email • Processing Payments • Geolocation • Import / Export • Indexing for Search • 86 Other things...
47. Background Payment Processing ms
48. Resque class AddressGeolocationJob max_retries 3 def self.perform(params) object = params[:model].constantize.find(params[:id]) object.latitude, object.longitude = Geocoder.geocode(object) object.save! end end Resque.enqueue(AddressGeolocationJob, :id => 1, :model => 'Address')
49. Redis • Inventory reservation system • Sessions • Theme uploads • Throttling • Sequenced Column
50. All Roads Lead To MySQL
51. MySQL Hardware • 4 x 8 Core Processor • SSD • 256 GB Ram • Full working set in memory
52. MySQL Query Optimization • pt-query-digest • Avoid queries that generate temp tables • Adding the right indexes • Forcing / Ignoring Indexes
53. MySQL Tuning • disable innodb_stats_on_metadata • increase table_open_cache • replace glibc memory allocator with tcmalloc • innodb_autoinc_lock_mode=‘interleaved’
54. after_commit db transactions best friend
55. after_commit • After transaction has been committed • Webhooks • Cache expiry • Update associated objects
56. after_commit class OrderObserver < ActiveRecord::Observer observe :order def after_save(order) if order.changes.keys.include?(:financial_status) order.flag_for_after_commit(:update_customer) end end def after_commit(order) if order.flagged_for_after_commit?(:update_customer) Resque.enqueue(UpdateCustomerJob, :id => order.id) end end end
57. Services
58. Services • Split out standalone services as needed • Independently scaled • Segmented metrics • Overall system is more complex • Limit to what is necessary
59. Imagery
60. Adapt and Evolve as Needed Using data and knowledge of the system to drive decisions
61. Thanks. @johnduff | john.duff@shopify.com