Off to the races

Race conditions are a natural hazard whenever working with shared resources. If you have multiple processes such as using a database, or a microservice architecture, there are likely some race conditions that are hiding in your code. As usage scales up, they will become more and more common. They are difficult to detect, challenging to reproduce and test, and sometimes downright hard to prove that it is fixed. In this talk, we will dive into what a race condition is, how it can show up in Rails applications, and some strategies on how to test them to give confidence that they are fixed.

Transcript

3. Kyle d’Oliveira Principal software engineer https://www.linkedin.com/in/doliveirakn kyle@aha.io
4. Our Software: Tools to help companies build what matters
7. Unanticipated behaviour caused by multiple processes interacting with shared resources in a di ff erent order than expected
8. Dining Philosophers
9. Dining Philosophers
10. P1 P2 P3 P4 P5 Dining Philosophers
11. P1 P2 P3 P4 P5 Dining Philosophers
12. P1 P2 P3 P4 P5 Dining Philosophers
13. P1 P2 P3 P4 P5 There is one chopstick between each philosopher Dining Philosophers
14. P1 P2 P3 P4 P5 There is one chopstick between each philosopher A philosopher may eat if they can pick up both adjacent chopsticks Dining Philosophers
15. P1 P2 P3 P4 P5 There is one chopstick between each philosopher A philosopher may eat if they can pick up both adjacent chopsticks Each chopstick can only be picked up by one philosopher Dining Philosophers
16. P1 P2 P3 P4 P5 There is one chopstick between each philosopher A philosopher may eat if they can pick up both adjacent chopsticks Each chopstick can only be picked up by one philosopher Dining Philosophers What is a process all philosophers can agree to such that no one will starve?
17. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat
18. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat
19. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat
20. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat
21. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat
22. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat
23. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat
24. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat
25. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat
26. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat
27. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat
28. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat
29. P1 P2 P3 P4 P5 Dining Philosophers
30. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available.
31. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available.
32. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available.
33. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available.
34. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available.
35. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available.
36. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available.
37. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available.
38. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available.
39. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available.
40. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available.
41. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available. 1. Pick up the left chopstick when it is available.
42. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat
43. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat Multiple processes
44. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat Shared Resources Multiple processes
45. P1 P2 P3 P4 P5 Dining Philosophers 1. Pick up the left chopstick when it is available. 2. Pick up right chopstick when it is available 3. Eat for some amount of time 4. Put left chopstick down 5. Put right chopstick down 6. Repeat Shared Resources Critical section Multiple processes
48. Multiple processes
49. Multiple processes Requests
50. Multiple processes Requests Background jobs
51. Multiple processes Requests Background jobs Rake tasks
52. Multiple processes Shared Resources Requests Background jobs Rake tasks
53. Multiple processes Shared Resources Requests Background jobs Rake tasks
54. Multiple processes Shared Resources Requests Background jobs Rake tasks
55. Multiple processes Shared Resources Requests Background jobs Rake tasks
56. 0 Votes:
57. 0 Votes:
58. 0 Votes:
59. 0 Votes:
60. 0 Votes:
61. 1 Votes:
62. 1 Votes:
63. 1 Votes:
64. 1 Votes:
65. 1 Votes:
66. 2 Votes:
67. 0 Votes:
68. 0 Votes:
69. 0 Votes:
70. 0 Votes:
71. 0 Votes:
72. 0 Votes:
73. 1 Votes:
74. 1 Votes:
75. 1 Votes:
76. 1 Votes:
77. 1 Votes:
91. Read-modify-write
92. Manual Testing
109. Automated Testing
119. How to fi x
120. 1. Remove the critical section
127. 2. Detect and recover
136. 1. How many times will retry?
137. 1. How many times will retry? 2.What will you do if the retries aren’t successful?
138. Important note All changes need to be undone on failure
139. 3. Protect the critical section
155. 1. How long will you wait?
156. 1. How long will you wait? 2.What do you do when there is a timeout?
171. Important note Be really mindful of how long you will wait
172. Check-then-act
189. How to fi x
190. 1. Removing the critical section
191. 1. Removing the critical section
192. 1. Removing the critical section
193. 1. Removing the critical section
194. 1. Removing the critical section
195. 2. Detect and Recover
196. 2. Detect and Recover
197. 2. Detect and Recover
198. 2. Detect and Recover
199. 3. Protect critical section
200. 3. Protect critical section
201. 3. Protect critical section
202. 3. Protect critical section
203. Start
204. Start Working with shared resources?
205. Start Done Working with shared resources? No
206. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? No Yes
207. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? No No Yes
208. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? No No Yes Yes
209. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? No No Yes Yes • Read-modify-write • Check-then-act • Others
210. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? No No No Yes Yes • Read-modify-write • Check-then-act • Others
211. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? Can you remove the critical section? No No No Yes Yes Yes • Read-modify-write • Check-then-act • Others
212. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? Can you remove the critical section? No No No Yes Yes Yes • Can you remove/refactor the code? • Can you make the operation atomic? • Read-modify-write • Check-then-act • Others
213. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? Can you remove the critical section? No No No Yes Yes Yes Yes • Can you remove/refactor the code? • Can you make the operation atomic? • Read-modify-write • Check-then-act • Others
214. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? Can you remove the critical section? Can you detect the race condition and recover from it? No No No No Yes Yes Yes Yes • Can you remove/refactor the code? • Can you make the operation atomic? • Read-modify-write • Check-then-act • Others
215. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? Can you remove the critical section? Can you detect the race condition and recover from it? No No No No Yes Yes Yes Yes • Can you remove/refactor the code? • Can you make the operation atomic? • How many times can you retry? • What do you do when it exceeds that? • Read-modify-write • Check-then-act • Others
216. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? Can you remove the critical section? Can you detect the race condition and recover from it? No No No No Yes Yes Yes Yes Yes • Can you remove/refactor the code? • Can you make the operation atomic? • How many times can you retry? • What do you do when it exceeds that? • Read-modify-write • Check-then-act • Others
217. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? Can you remove the critical section? Can you detect the race condition and recover from it? Can you protect the critical section? No No No No No Yes Yes Yes Yes Yes • Can you remove/refactor the code? • Can you make the operation atomic? • How many times can you retry? • What do you do when it exceeds that? • Read-modify-write • Check-then-act • Others
218. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? Can you remove the critical section? Can you detect the race condition and recover from it? Can you protect the critical section? No No No No No Yes Yes Yes Yes Yes • Can you remove/refactor the code? • Can you make the operation atomic? • How many times can you retry? • What do you do when it exceeds that? • How long can a process wait? • What do you do when it exceeds that? • Read-modify-write • Check-then-act • Others
219. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? Can you remove the critical section? Can you detect the race condition and recover from it? Can you protect the critical section? No No No No No Yes Yes Yes Yes Yes • Can you remove/refactor the code? • Can you make the operation atomic? • How many times can you retry? • What do you do when it exceeds that? • How long can a process wait? • What do you do when it exceeds that? Yes • Read-modify-write • Check-then-act • Others
220. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? Can you remove the critical section? Can you detect the race condition and recover from it? Can you protect the critical section? No No No No No Yes Yes Yes Yes Yes • Can you remove/refactor the code? • Can you make the operation atomic? • How many times can you retry? • What do you do when it exceeds that? Application speci f ic solution • How long can a process wait? • What do you do when it exceeds that? Yes No • Read-modify-write • Check-then-act • Others
221. Start Done Working with shared resources? Can multiple processes work on those resources at the same time? Is there a race condition? Can you remove the critical section? Can you detect the race condition and recover from it? Can you protect the critical section? No No No No No Yes Yes Yes Yes Yes • Can you remove/refactor the code? • Can you make the operation atomic? • How many times can you retry? • What do you do when it exceeds that? Application speci f ic solution • How long can a process wait? • What do you do when it exceeds that? Yes No • Read-modify-write • Check-then-act • Others
222. Thank you