Understanding Race Condition on Rails

Conference page: http://ruby.id/conf/2017/

Video: https://www.youtube.com/watch?v=QztcYr_8M6I

Source: https://github.com/mufid/race-condition-example

Transcript

1. CC0 Public Domain – Raffael Herrmann (Speedometer Lights)
2. Muhammad Mufid Afif https://mufid.github.io
3. @mufid
4. @mufid CATS
5. Rails’ing since 2012 Professionally 2015
7. SaaS Platform & IoT Devices
8. Devices
12. WHAT IS RACE CONDITION
13. http://www.publicdomainpictures.net/view-image.php?image=95019&picture=formula-sports-cars
14. http://www.publicdomainpictures.net/view-image.php?image=2404&picture=vintage-velocette-race-bike
15. https://www.slideshare.net/indicthreads/building-restful-applications-using-spring-mvc
17. REALWORLDEXAMPLE https://github.com/mufid/race-condition-example
19. $5202 PREV BALANCE
20. $5202 PREV BALANCE
21. $5202 PREV BALANCE
22. $5202 PREV BALANCE
25. $5207 PREV BALANCE
26. $5207 PREV BALANCE
27. $5207 PREV BALANCE
28. $5207 PREV BALANCE
29. -$25??
30. WHATWASTHATBUG
31. def top_up_balance(amount) validated_amount = PaymentVerifier.verify_top_up( Time.now, amount) self.balance += amount.to_i self.save! end
32. def top_up_balance(amount) validated_amount = PaymentVerifier.verify_top_up( Time.now, amount) self.balance += amount.to_i self.save! end SEEMS FINE??
34. A race condition or race hazard is the behavior of an electronic, software, or other system where the output is dependent on the sequence or timing of other uncontrollable events. It becomes a bug when events do not happen in the order the programmer intended
35. A race condition or race hazard is the behavior of an electronic, software, or other system where the output is dependent on the sequence or timing of other uncontrollable events. It becomes a bug when events do not happen in the order the programmer intended
36. A race condition or race hazard is the behavior of an electronic, software, or other system where the output is dependent on the sequence or timing of other uncontrollable events. It becomes a bug when events do not happen in the order the programmer intended
37. A race condition or race hazard is the behavior of an electronic, software, or other system where the output is dependent on the sequence or timing of other uncontrollable events. It becomes a bug when events do not happen in the order the programmer intended
38. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
39. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! Let initial balance $100 and amount is $5
40. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
41. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
42. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! @account.balance is now $105 But not yet persisted in database!
43. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! Now account is persisted in database with $105 balance
44. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
45. MULTIPLEPROCESSOR MULTITHREADING MULTIMACHINE
46. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
47. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
48. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
49. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
50. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
51. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! $105 or $110? IDK LOL
52. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
53. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
54. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
55. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
56. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
57. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
58. ATOMICOPERATION
59. Guaranteed done without interruption
60. verify_request! @account = Account.add_balance(params[:id],params[:balance])
61. verify_request! @account = Account.add_balance(params[:id],params[:balance])
62. class DepositController < ApplicationController def update verify_and_deposit(current_account, deposit_params) end end
63. class DepositController < ApplicationController def update verify_and_deposit(current_account, deposit_params) end end
64. ATOMIC != 1 LINE
65. READORWRITE
66. DESIRED ACTION
67. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
68. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
69. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! $100  $105 $105  $110
70. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
71. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
72. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
73. verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! $100  $105 $100  $105
74. You may read a value that is being written by other
75. .find .find_by .find_by!
76. .save .save!
77. In this case: read and write must be atomic
78. Lock
79. Gaining lock considered atomic
80. Built-in Lock
81. Pessimistic Locking Optimistic Locking
82. Pessimistic
83. Pessimistic REQUIRES TRANSACTION GUARANTEED OTHER’S READ WILL BE BLOCKED UNTIL TX DONE LESS LIKELY TO BE FAIL
84. Pessimistic verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end
85. Pessimistic verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end Guaranteed other can’t execute .find
86. Pessimistic verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end
87. Pessimistic verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end
88. Pessimistic verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end
89. Pessimistic verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end
90. Pessimistic verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end The first one who gain the lock will execute next line
91. Pessimistic verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end Blocked
92. Pessimistic verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end $100  $105 Persisted to DB
93. Pessimistic verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end $105  $110 Persisted to DB
94. Pessimistic verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end verify_request! Account.transaction do @account = Account.lock.find(params[:id]) @account.balance += params[:amount] @account.save! end
95. Pessimistic
96. Optimistic
97. Optimistic DOES NOT NEED TRANSACTION NO BLOCKING WRITE CAN FAIL IF RACE CONDITION DETECTED
98. Optimistic # add_column <table>, lock_version, :integer add_column :accounts, :lock_version, :integer
99. Optimistic UPDATE transactions SET lock_version = #{current_lock_version} + 1 your_column = key ... WHERE lock_version = #{current_lock_version} AND id = ...
100. Optimistic UPDATE transactions SET lock_version = #{current_lock_version} + 1 your_column = key ... WHERE lock_version = #{current_lock_version} AND id = ...
101. Optimistic verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
102. Optimistic verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
103. Optimistic verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! Update with id=1 and lock_version=1 lock_version  1
104. Optimistic verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! Update with id=1 and lock_version=1 lock_version  2
105. Optimistic verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! ActiveRecord::StaleObjectError lock_version  2
106. Optimistic verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save! verify_request! @account = Account.find(params[:id]) @account.balance += params[:amount] @account.save!
107. Optimistic
108. Optimistic RETRIES = 3 RETRIES.times do begin ... @account.save! break # exit from loop rescue ActiveRecord::StaleObjectError # retry end end
109. READORWRITE
110. WRITEWITHOUTREAD
111. LET POSTGRES LOCK THEMSELF
112. UPDATE transactions SET balance = balance + #{amount.to_i} ... WHERE id = ...
113. UPDATE transactions SET balance = balance + #{amount.to_i} ... WHERE id = ...
114. verify_request! @account = Account.find(params[:id]) @account.connection .execute( update_query_for(@account, params[:amount]))
115. BYPASSING CALLBACKS (after_* before_*) BYPASSING VALIDATIONS
116. RAW SQL! CAREFUL!!!!11
117. PEENTAR’SCASE
118. Thousand of data
119. Thousand of paginated data
120. Data Drift
121. PAGE 1 Data 1 Data 3 Data 9 PAGE 2 Data 15 Data 28 Data 4 PAGE 3 Data 35 Data 52 Data 91
122. PAGE 1 Data 1 Data 3 Data 9 PAGE 2 Data 15 Data 28 Data 4 PAGE 3 Data 35 Data 52 Data 91 order(:modified_at)
123. PAGE 1 Data 1 Data 3 Data 9 PAGE 2 Data 15 Data 28 Data 4 PAGE 3 Data 35 Data 52 Data 91 order(:modified_at)
124. PAGE 1 Data 1 Data 3 Data 9 PAGE 2 Data 15 Data 28 Data 4 PAGE 3 Data 35 Data 52 Data 91 order(:modified_at)
125. PAGE 1 Data 1 Data 3 Data 9 PAGE 2 Data 15 Data 28 Data 4 PAGE 3 Data 35 Data 52 Data 91 order(:modified_at)
126. PAGE 1 Data 28 Data 1 Data 3 PAGE 2 Data 9 Data 15 Data 4 PAGE 3 Data 35 Data 52 Data 91 order(:modified_at)
127. Persist ALL pages at first query
128. GET /orders/sync?page=1
129. GET /orders/sync?page=1 X-RequestNewTx: true
130. GET /orders/sync?page=1 X-RequestNewTx: true HTTP/1.1 OK X-TxId: 1f184051-7750-4033-b097
131. GET /orders/sync?page=2 X-TxId: 1f184051-7750-4033-b097
132. PAGE 1 Data 1 Data 3 Data 9 PAGE 2 Data 15 Data 28 Data 4 PAGE 3 Data 35 Data 52 Data 91
133. PAGE 1 Data 1 Data 3 Data 9 PAGE 2 Data 15 Data 28 Data 4 PAGE 3 Data 35 Data 52 Data 91
134. PAGE 1 Data 1 Data 3 Data 9 PAGE 2 Data 15 Data 28 Data 4 PAGE 3 Data 35 Data 52 Data 91
135. OTHER CASES AND ALTERNATIVES
136. Serializable Isolation Mode
137. rollback if >=1 query in tx may conflict other tx.
138. CREATEYOUROWNLOCK
139. REDIS MEMCACHED FILE RUBY QUEUE
140. Mitigate!
141. DRAW
142. DRAW State Diagram
143. DRAW Sequence Diagram
144. Understand how your program works
145. Understand how your program works on multiple machine
146. Simulate slowness
147. Try running it line by line
148. Run multi-thread test
149. Test with some thread being paused.
150. Understand how your program works
151. THANKYOU QUESTIONS? SOURCE CODE AND SLIDE: SPEAKERDECK.COM/MUFID i am mufid let’s make the software, better! mail me privately at mufidafif@icloud.com @mufid or you can tweet publicly to me via
152. thanks! see you later that is my vCard