How shopify sharded rails

Talk presented in bigrubyconf 2014

Transcript

1. shopify How Shopify sharded Rails Camilo Lopez / @camilolopez Monday, February 24, 14
2. Shopifacts Monday, February 24, 14
3. Scale Monday, February 24, 14
4. ~100 app servers Monday, February 24, 14
5. ~88k customers Monday, February 24, 14
6. Stack Ruby 2.1, Rails 3.2, MySQL 5.5 (percona) Monday, February 24, 14
7. App servers (Unicorn) Job servers (Resque) Monday, February 24, 14
8. ~150k RPM with peaks up to ~400k RPM Monday, February 24, 14
9. ~20k QPS Over all shards Monday, February 24, 14
10. Moneys Monday, February 24, 14
11. $1.6B Annual GMV Sales over all the platform Monday, February 24, 14
12. $3.7k / min Monday, February 24, 14
13. Outages Monday, February 24, 14
14. Monday, February 24, 14
15. Cons of sharding Why you probably should not shard Monday, February 24, 14
16. Sharding Slicing your data over more than one database Monday, February 24, 14
17. Rails and libs assume one DB Monday, February 24, 14
18. Makes it hard to do aggregated queries Querying across DBs is a pain Monday, February 24, 14
19. AUTO_INCREMENT Does not quite work the same anymore Monday, February 24, 14
20. Likely custom code Be prepared to maintain Monday, February 24, 14
21. You might be able to buy your way out Normally just MOAR CPUs will do Monday, February 24, 14
22. Tune and cache Worked for us (tm) for a long time Monday, February 24, 14
23. Why did we shard? If it does sound like a pain Monday, February 24, 14
24. Traffic and customers double each year Monday, February 24, 14
25. Black Friday/ Cybermonday 2x normal load Monday, February 24, 14
26. You cannot cache writes Monday, February 24, 14
27. Monday, February 24, 14
28. “InnoDB tries to limit thread concurrency allowing only i n n o d b _ t h r e a d _ c o n c u r r e n c y threads to run inside InnoDB kernel at the same time.” Monday, February 24, 14
29. Monday, February 24, 14
30. 81 2011 2012 •2xIntel E5640 2.67GHz •192GB of RAM •2 x 300GB OCZ Z- Drive R4 PCIe SSDs Monday, February 24, 14
31. 81 2011 2012 • 4 x Intel E5-4650 2.7 GHz •256GB of RAM •2 x 600GB OCZ Z-Drive •2xIntel E5640 2.67GHz •192GB of RAM •2 x 300GB OCZ Z- Drive R4 PCIe SSDs Monday, February 24, 14
32. Buy better CPUs? 8x CPUs get quite expensive Monday, February 24, 14
33. Horizontal scaling n-CPUs Monday, February 24, 14
34. Also OMG Cybermonday! Monday, February 24, 14
35. Hokay... let’s shard Monday, February 24, 14
36. Side benefits Monday, February 24, 14
37. Better localized data make more effective buffer pools Monday, February 24, 14
38. Smaller indexes Faster lookups Monday, February 24, 14
39. Failure isolation If a shard fails, it’s still bad, but not as bad as if the only DB fails Monday, February 24, 14
40. Sharding Shopify Or what did we do in 2013? Monday, February 24, 14
41. Constraints Monday, February 24, 14
42. No mixing of shards in the same transaction We can get around it, and implementing distributed transactions is not trivial. Monday, February 24, 14
43. No JOINS between shards Monday, February 24, 14
44. No nested shard selection We use blocks to define shard contexts, but we also want developers to not have to think about it Monday, February 24, 14
45. Had to happen “live” Product developers were pushing features all the time there was no “stop and shard” moment 45 Monday, February 24, 14
46. How did we slice our data? Sharding key Monday, February 24, 14
47. App servers (Unicorn) “The” Database Monday, February 24, 14
48. Monday, February 24, 14
49. Most things are scoped to a shop Yay! easy shop_id is the sharding key Monday, February 24, 14
50. Denormalize shop_id Makes life so much easier Monday, February 24, 14
51. Add shard_id to shop We did this before we had any shards at all Monday, February 24, 14
52. Shard Master db Master Monday, February 24, 14
53. App servers (Unicorn) Master DB Shards Monday, February 24, 14
54. Master DB Monday, February 24, 14
55. App servers (Unicorn) Master DB Shards Monday, February 24, 14
56. Shard 0,1, ... ,n,n+1 Monday, February 24, 14
57. App servers (Unicorn) Master DB Shards Monday, February 24, 14
58. Primary keys Monday, February 24, 14
59. AUTO_INCR does not work anymore Ids will not be unique across DBs Monday, February 24, 14
60. Noeqd Like snowflake, meaningful, k-sortable id generator in golang Monday, February 24, 14
61. Works well Ran in production for only one table that was safe- ish to screw up Monday, February 24, 14
62. One more service Do we really want that? Monday, February 24, 14
63. Javascript does not like big numbers 53Bits max to be safe Monday, February 24, 14
64. (ϊಠӹಠ)ϊኯᵲᴸᵲ Monday, February 24, 14
65. MySQL config Monday, February 24, 14
66. auto_increment_increment Controls the interval between successive column values. Monday, February 24, 14
67. auto_increment_offset Determines the starting point for the AUTO_INCREMENT column value. Monday, February 24, 14
68. next_id = auto_increment_offset + N × auto_increment_increment N is a member of 1,2,3,4... Monday, February 24, 14
69. Fix on an increment Different offset per shard Monday, February 24, 14
70. increment,offset ids shard_0 4,1 5,9,13,17 shard_1 4,2 6,10,14,18 shard_2 4,3 7,11,15,19 Monday, February 24, 14
71. Connection switching Teach your app to know what db to talk to Monday, February 24, 14
72. Block based shard context Sharding.with_shard(id) { #stuff } Sharding.with_shop(shop) { #stuff } Monday, February 24, 14
73. def do_stuff_in_shard(shard_id) Sharding.with_shard(shard_id) do # stuff happens here!!! end end Monday, February 24, 14
74. Within the block sharded models use the current shard connection Sharded models include Sharding::Concern, have shop_id column. Monday, February 24, 14
75. class ImSharded < ActiveRecord::Base include Sharding::Concern end Monday, February 24, 14
76. Non-sharded models do not care Always talk to the master DB Monday, February 24, 14
77. Connection Swtching Monday, February 24, 14
78. HTTP requests Monday, February 24, 14
79. Monday, February 24, 14
80. Domain to Shop Monday, February 24, 14
81. Shop to shard every shop has a shard_id. Monday, February 24, 14
82. Now run the request in a shard context Monday, February 24, 14
83. def with_shop(request) @shop = ShopManager.shop_for(request.host) Sharding.with_shop(@shop) yield # Actual request happens here. end end Monday, February 24, 14
84. Background jobs Monday, February 24, 14
85. if params[:shop_id] Monday, February 24, 14
86. easy Monday, February 24, 14
87. class CommentSpamCheckerJob include BackgroundQueue::Low include Sharding::BackgroundQueue::SelectShard include BackgroundQueue::Locking # perform will have the shard context of # params[:shop_id] end Monday, February 24, 14
88. !params[:shop_id] Stay tuned Monday, February 24, 14
89. This is transparent to day-to-day tasks until it is not Monday, February 24, 14
90. Intermission Hard work here Monday, February 24, 14
91. Remember the constraints? Monday, February 24, 14
92. Existent code did not care about them Monday, February 24, 14
93. Write verifiers Monday, February 24, 14
94. Log Monday, February 24, 14
95. Started GET "/" for 127.0.0.1 at 2014-02-19 18:53:05 +0000 Processing by ShopController#index as */* [Sharding::MissingShard] SELECT `products`.* FROM `products`*application Completed 302 Found in 175.2ms Monday, February 24, 14
96. Fix one at a time Monday, February 24, 14
97. Gone? Make it raise Monday, February 24, 14
98. Started GET "/" for 127.0.0.1 at 2014-02-19 18:02:45 +0000 Processing by ShopController#index as */* ********************************* Exception Sharding::MissingShardError: MissingShardError: SELECT `products`.* FROM `products` Completed 500 Internal Server Error in 49.9ms Monday, February 24, 14
99. Rinse and repeat For each of the constraints Monday, February 24, 14
100. Took months Devs were writing all sorts of new features while we worked on sharding Monday, February 24, 14
101. Rebalancing Do we move things across shards? Monday, February 24, 14
102. Monday, February 24, 14
103. Lock-and-move Monday, February 24, 14
104. Global exclusive lock Maintenance tasks, some cron jobs and the shop mover Monday, February 24, 14
105. Per shop shared lock Taken by the query across shard methods as they iterate, important for things that go across shards Monday, February 24, 14
106. Zookeeper or Redis Monday, February 24, 14
107. shop.is_locked No need to reach to a locking server, just HTTP 503 if set Monday, February 24, 14
108. Ruby script Monday, February 24, 14
109. $ script/move_shop --shop_id=42 \--dest_id=2 --concurrency=16 Monday, February 24, 14
110. T0 T1 (+100secs) T2 T3 T4 T5 Take global lock or bail, shop.is_locked = true Monday, February 24, 14
111. T0 T1 (+100secs) T2 T3 T4 T5 Unicorns dead Take global lock or bail, shop.is_locked = true Monday, February 24, 14
112. T0 T1 (+100secs) T2 T3 T4 T5 Unicorns dead Jobs have released shared locks Take global lock or bail, shop.is_locked = true Monday, February 24, 14
113. T0 T1 (+100secs) T2 T3 T4 T5 Unicorns dead Fork, copy, verify Jobs have released shared locks Take global lock or bail, shop.is_locked = true Monday, February 24, 14
114. T0 T1 (+100secs) T2 T3 T4 T5 Unicorns dead Fork, copy, verify Change shop.shard_id Jobs have released shared locks Take global lock or bail, shop.is_locked = true Monday, February 24, 14
115. T0 T1 (+100secs) T2 T3 T4 T5 Unicorns dead Fork, copy, verify Change shop.shard_id Jobs have released shared locks shop.is_locked = false Take global lock or bail, shop.is_locked = true Monday, February 24, 14
116. T0 T1 (+100secs) T2 T3 T4 T5 Unicorns dead Fork, copy, verify Change shop.shard_id Jobs have released shared locks shop.is_locked = false Take global lock or bail, shop.is_locked = true Monday, February 24, 14
117. What about the old data? Deleted offline Monday, February 24, 14
118. Why not online? Too hard Monday, February 24, 14
119. What about JOINs :( Monday, February 24, 14
120. Can’t just iterate A shop can be locked, extraneous rows from moves Monday, February 24, 14
121. Take shared locks while iterating The user decides what to do with locked shops, ignore/raise Monday, February 24, 14
122. They also ignore data left orphaned It will get deleted, but might be hours Monday, February 24, 14
123. query_on_each_shard Results from an ActiveRecord::Relation on each shard Monday, February 24, 14
124. things = [] rel = Model.preload(:transactions, :order) .where(id: params[:id].split(+)) Sharding.query_on_each_shard(rel, :on_lock => :raise) do |recs| things.concat(recs) end Monday, February 24, 14
125. find_in_batches_on_each_shard Like find_each except queries every shard Monday, February 24, 14
126. r = model.unscoped Sharding.find_in_batches_on_each_shard(r, on_lock::ignore) do |recs| recs.each do |record| reserialize_record(record, params[:fields]) progress.tick end end Monday, February 24, 14
127. If someone absolutely must run custom SQL Take the global lock. Also, do not do this. Monday, February 24, 14
128. How long did it take? 2013 Monday, February 24, 14
129. Nov 2012 Dec-Jan March? May Jul-Oct Oct-Nov Start DB-charmer, constraints and verifiers Work, work, work Look for vendors? #dunnolol Move shops Cybermonday 2012 Query across DBs OMG Cybermonday Monday, February 24, 14
130. Sharding checklist • How to slice data • Primary key generation • Connection switching • Query across shards • Rebalancing 130 Monday, February 24, 14
131. OMG Cybermonday Trial by fire Black Friday/Cyber Monday 2013. Monday, February 24, 14
132. ~100k RPM higher than day-to-day Monday, February 24, 14
133. Monday, February 24, 14
134. QPS 10-15k higher Monday, February 24, 14
135. Monday, February 24, 14
136. All those spikes Flash sales 136 Monday, February 24, 14
137. Monday, February 24, 14
138. ~80ms avg response time zero downtime, all time record sales for one day Monday, February 24, 14
139. Monday, February 24, 14
140. Monday, February 24, 14
141. Thanks! Monday, February 24, 14
142. Questions? Monday, February 24, 14