Abril Pro Ruby

Sharding presentation given at abrilproruby.com/en

Transcript

1. shopify Sharding Shopify Arthur Neves / @arthurnn
2. Shopifacts
3. Scale
4. ~100 app servers
5. ~100k customers
6. Stack Ruby 2.1.1, Rails 4.0, MySQL 5.6 (percona)
7. App servers (Unicorn) Job servers (Resque)
8. ~150k RPM with peaks up to ~400k RPM
9. ~20k QPS
10. Moneys
11. $1.6B Annual GMV Sales over all the platform
12. $3.7k / min
13. Outages
15. What is sharding?
16. Challenges of sharding Why you probably should not shard
17. Sharding Slicing your data over more than one database
18. Rails and libs assume one DB
19. Makes it hard to do aggregated queries Querying across DBs is a pain
20. AUTO_INCREMENT Does not quite work the same anymore
21. Likely custom code Be prepared to maintain
22. You might be able to buy your way out Normally just MOAR CPUs will do
23. Tune and cache Worked for us (tm) for a long time
24. Why did we shard? If it does sound like a pain
25. Traffic and customers double each year
26. Black Friday/ Cybermonday 2x normal load
27. You cannot cache writes
28. 81 2011 2012 •2xIntel E5640 2.67GHz •192GB of RAM •2 x 300GB OCZ Z- Drive R4 PCIe SSDs
29. 81 2011 2012 • 4 x Intel E5-4650 2.7 GHz •256GB of RAM •2 x 600GB OCZ Z-Drive •2xIntel E5640 2.67GHz •192GB of RAM •2 x 300GB OCZ Z- Drive R4 PCIe SSDs
30. Buy better CPUs? 8x CPUs get quite expensive
31. Horizontal scaling n-CPUs
32. Side benefits
33. Scale writes/reads horizontally
34. Smaller indexes Faster lookups
35. Failure isolation If a shard fails, it’s still bad, but not as bad as if the only DB fails
36. Sharding checklist Things that needed to happen
37. How to slice our data? Sharding key Rails code
38. Primary key generation Simple AUTO_INCREMENT will not cut it anymore MySQL config
39. Connection switching Teach rails app to know which db to talk to Rails code
40. What about JOINs and Transactions? Rails code
41. Rebalancing How do we move things across shards? Rails code + more
42. Sharding Shopify Or what did we do in 2013?
43. How did we slice our data? Sharding key
44. App servers (Unicorn) “The” Database
45. Most things are scoped to a shop Yay! easy shop_id is the sharding key
47. Denormalize shop_id Makes life so much easier
48. Add shard_id to shop We did this before we had any shards at all
49. Shard Master db Master
50. Primary keys
51. AUTO_INCR does not work anymore Ids will not be unique across DBs
52. Noeqd Like snowflake, k-sortable id generator in golang
53. Works well Ran in production for only one table that was safe- ish to screw up
54. One more service Do we really want that?
55. (ノಠ益ಠ)ノ⼺彡┻━┻
56. MySQL config
57. auto_increment_increment Controls the interval between successive column values.
58. auto_increment_offset Determines the starting point for the AUTO_INCREMENT column value.
59. next_id = auto_increment_offset + N × auto_increment_increment
    N is a member of 1,2,3,4...
60. Fix on an increment Different offset per shard
61. increment,offset ids shard_0 4,1 5,9,13,17 shard_1 4,2 6,10,14,18 shard_2 4,3 7,11,15,19
62. Make sure ids are big 62 Shopify/rails-bigint-pk
63. Connection switching Teach your app to know what db to talk to
64. Block based shard context Sharding.with_shard(id) { #stuff } Sharding.with_shop(shop) { #stuff }
65. def do_stuff_in_shard(shard_id) Sharding.with_shard(shard_id) do # db stuff happens here!!! end end
66. Within the block sharded models use the current shard connection Sharded models include Sharding::Concern, have shop_id column.
67. class Order < ActiveRecord::Base include Sharding::Concern end
68. Non-sharded models do not care Always talk to the master DB
69. Connection Swtching
70. HTTP requests
72. Domain to Shop
73. Shop to shard every shop has a shard_id.
74. Now run the request in a shard
75. around_action :with_shop ! def with_shop ! @shop = ShopManager.shop_for(request.host) ! Sharding.with_shop(@shop) yield # Actual request happens here. end end
76. Background jobs
77. if params[:shop_id]
78. easy
79. class CommentSpamCheckerJob include BackgroundQueue::Low include Sharding::BackgroundQueue::SelectShard include BackgroundQueue::Locking # perform will have the shard context of # params[:shop_id] end
80. What about JOINs and Transactions? :(
81. Can’t just iterate A shop can be locked, extraneous rows from moves
82. Take shared locks while iterating The user decides what to do with locked shops, ignore/raise
83. They also ignore data left orphaned It will get deleted, but might be hours
84. Primitives 84
85. query_on_each_shard Results from an ActiveRecord::Relation on each shard
86. things = [] rel = Model.preload(:checkouts, :order) .where(id: params[:id].split(+)) ! Sharding.query_on_each_shard(rel, :on_lock => :raise) do |recs| things.concat(recs) end
87. find_in_batches_on_each_shar d Like find_each except queries every shard
88. r = model.unscoped ! Sharding.find_in_batches_on_each_shard(r, on_lock: :ignore) do |recs| recs.each do |record| reserialize_record(record, params[:fields]) progress.tick end end
89. If someone absolutely must run custom SQL Take the global lock. Also, do not do this.
90. Rebalancing Do we move things across shards?
92. Lock-and-move
93. Global exclusive lock Maintenance tasks, some cron jobs and the shop mover
94. shop.is_locked No need to reach to a locking server, just HTTP 503 if set
95. Per shop shared lock Taken by the query across shard methods as they iterate, important for things that go across shards
96. Zookeeper or Redis
97. $ script/move_shop --shop_id=42 \--dest_id=2 --concurrency=16
98. T0 T1 (+100secs) T2 T3 T4 T5 Take global lock or bail, shop.is_locked = true
99. T0 T1 (+100secs) T2 T3 T4 T5 Unicorns dead Take global lock or bail, shop.is_locked = true
100. T0 T1 (+100secs) T2 T3 T4 T5 Unicorns dead Jobs have released shared locks Take global lock or bail, shop.is_locked = true
101. T0 T1 (+100secs) T2 T3 T4 T5 Unicorns dead Fork, copy, verify Jobs have released shared locks Take global lock or bail, shop.is_locked = true
102. T0 T1 (+100secs) T2 T3 T4 T5 Unicorns dead Fork, copy, verify Change shop.shard_id Jobs have released shared locks Take global lock or bail, shop.is_locked = true
103. T0 T1 (+100secs) T2 T3 T4 T5 Unicorns dead Fork, copy, verify Change shop.shard_id Jobs have released shared locks shop.is_locked = false Take global lock or bail, shop.is_locked = true
104. T0 T1 (+100secs) T2 T3 T4 T5 Unicorns dead Fork, copy, verify Change shop.shard_id Jobs have released shared locks shop.is_locked = false Take global lock or bail, shop.is_locked = true
105. What about the old data? Deleted offline
106. Why not online? Too hard
107. Sharding checklist • How to slice data • Primary key generation • Connection switching • Query across shards • Rebalancing 107
108. Results Trial by fire Black Friday/Cyber Monday 2013.
109. ~100k RPM higher than day-to-day
111. QPS 10-15k higher
113. All those spikes Flash sales 113
115. ~80ms avg response time zero downtime, all time record sales for one day
118. Thanks! Thanks @camilolopez for the slides