Data corrupting architectures we know and love

There’s an old distributed systems adage, “You can have a second computer when you learn how to use the first one.” When it comes to data-access patterns, most of our favorite patterns are unsafe on a single computer. Most of our applications assume a concurrency model where all access is serialized. What happens when that mental model meets distributed data?

Sean Allen reviews data race and corruption problems that exist on single-machine systems and shows how we’ve transferred many of those patterns over to distributed systems and distributed state. You’ll learn the basics of data races, deadlocks, and problems in even the simplest of our concurrent data access patterns and how our existing scaling patterns replicate the same issues across multiple machines, increasing the potential problems, as well as designs that can alleviate them.

Transcript

1. WE KNOW AND LOVE DATA CORRUPTING ARCHITECTURES
2. VP OF ENGINEERING @ WALLAROO LABS MEMBER OF THE PONY CORE TEAM AUTHOR OF “STORM APPLIED” CHICHARRONES LOVER @SEANTALLEN @WALLAROOLABS @PONYLANG SEAN T. ALLEN
3. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE TODAY’S TOPICS INCLUDE… ▸ Data corruption ▸ Concurrency ▸ Data races ▸ Shared values ▸ Atomic operations
4. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE GENERALIZATIONS AHEAD
5. OUR MOST POPULAR ARCHITECTURES CAN BE TRACED TO WEB APPPLICATIONS. Sean T. Allen DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
6. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE DATABASE WEB APP
7. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE A BIT MORE DETAILED… REQUEST to the APPLICATION on to the DATABASE back to the APPLICATION and out as a RESPONSE
8. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE WE COULD GET MORE DETAILED STILL… REQUEST to the APPLICATION on to the DATABASE back to the APPLICATION and out as a RESPONSE (WEB BROWSER) to the (WEB SERVER + DEVELOPER CODE) on to the DATABASE back to the (WEB SERVER + DEVELOPER CODE) and out as a (WEB BROWSER)
9. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE OUR SIMPLE WEB APP IS A DISTRIBUTED SYSTEM
10. DISTRIBUTED SYSTEMS AREN’T HARD. THEY’RE EASY*. Sean T. Allen DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE * sometimes
11. DISTRIBUTED SYSTEMS AREN’T HARD. THEY’RE EASY*. CONCURRENCY IS THE HARD PART. Sean T. Allen DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE * sometimes
12. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE SOME JUSTIFICATION AHEAD
13. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE DATABASE WEB APP
14. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE DATABASE WEB APP
15. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE DATABASE WEB APP
16. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE AN EXAMPLE APPLICATION
17. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return new value to the user
18. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return original value to the user SPOT THE DATA CORRUPTION?
19. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return new value to the user
20. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return original value to the user LET’S ADD SOME CONCURRENCY
21. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return new value to the user ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return new value to the user
22. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return original value to the user ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return original value to the user WHAT’S THE FINAL VALUE? 1? 2?
23. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE WE DON’T KNOW
24. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return new value to the user ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return new value to the user
25. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current value from the database 0 ‣ Read current value from the database 0 ‣ Add one to update it 1 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Write new value to the database 1 ‣ Return new value to the user 1 ‣ Return new value to the user 1 Request A Request B
26. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current value from the database 0 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Return new value to the user 1 ‣ Read current value from the database 1 ‣ Add one to update it 2 ‣ Write new value to the database 2 ‣ Return new value to the user 2 Request A Request B
27. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current value from the database 0 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Read current value from the database 1 ‣ Return new value to the user 1 ‣ Add one to update it 2 ‣ Write new value to the database 2 ‣ Return new value to the user 2 Request A Request B
28. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current value from the database 0 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Read current value from the database 1 ‣ Return new value to the user 1 ‣ Add one to update it 2 ‣ Write new value to the database 2 ‣ Return new value to the user 2 Request A Request B
29. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current value from the database 0 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Read current value from the database 1 ‣ Return new value to the user 1 ‣ Add one to update it 2 ‣ Write new value to the database 2 ‣ Return new value to the user 2 Request A Request B
30. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE WE’RE EXPERIENCING A DATA RACE
31. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE RECIPE FOR A DATA RACE ▸ A shared value ▸ Concurrency ▸ At least one writer changing the shared value ▸ Non-atomic operations
32. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE OUR DATABASE IS A SHARED VALUE
33. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE CONCURRENCY: WE HAVE TWO REQUESTS
34. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE WHOLE LOTTA WRITING GOING ON
35. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE OUR INCREMENTING ISN’T ATOMIC
36. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE AN OPERATION IS ATOMIC IF IT’S ISOLATED FROM OTHER OPERATIONS HAPPENING AT THE SAME TIME.
37. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current value from the database 0 ‣ Read current value from the database 0 ‣ Add one to update it 1 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Write new value to the database 1 ‣ Return new value to the user 1 ‣ Return new value to the user 1 Request A Request B
38. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current value from the database 0 ‣ Read current value from the database 0 ‣ Add one to update it 1 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Write new value to the database 1 ‣ Return new value to the user 1 ‣ Return new value to the user 1 Request A Request B
39. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Increment current value 1 ‣ Increment current value 2 ‣ Read new value 2 ‣ Read new value 2 ‣ Return new value to the user 2 ‣ Return new value to the user 2 Request A Request B
40. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Increment current value 1 ‣ Increment current value 2 ‣ Read new value 2 ‣ Read new value 2 ‣ Return new value to the user 2 ‣ Return new value to the user 2 Request A Request B
41. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE MORE CONCURRENCY MORE PROBLEMS
42. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1 1 1 2 Request A Request B Request C
43. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1 1 1 2 Request A Request B Request C
44. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1 1 1 2 Request A Request B Request C
45. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1 1 1 2 Request A Request B Request C
46. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE DATABASE VALUE IS 1
47. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1 1 1 2 Request A Request B Request C
48. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1 1 1 2 Request A Request B Request C
49. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1 1 1 2 Request A Request B Request C
50. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE DATABASE VALUE INCREASED TO 2
51. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1 1 1 2 Request A Request B Request C
52. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1 1 1 2 Request A Request B Request C
53. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE DATABASE VALUE DECREASED TO 1
54. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE WE’RE EXPERIENCING LAST WRITE WINS
55. LAST WRITE WINS MEANS “SOME WRITES LOSE”. Sean Cribbs & Russell Brown DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
56. LAST WRITE WINS IS A FANCY NAME FOR “LET’S CORRUPT SOME DATA”. Sean T. Allen DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
57. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE OUR MOST POPULAR ARCHITECTURES FOR SCALING
58. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THESE PROBLEMS ARE EVERYWHERE
59. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE “SERVERLESS”
60. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE “FUNCTIONS AS SERVICE”
61. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE “STATELESS”
62. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE “SHARED NOTHING”
63. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ALL FEATURE DATABASE AS A SHARED VALUE AND CONCURRENT WRITES
64. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ALL HAVE DATA CORRUPTION BAKED INTO THE CORE ARCHITECTURE
65. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE WHAT CAN WE DO?
66. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE RECIPE FOR A DATA RACE: CHANGE AT LEAST ONE ▸ A shared value ▸ Concurrency ▸ At least one writer changing the shared value ▸ Non-atomic operations
67. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE STORED PROCEDURES AND TRANSACTIONS
68. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE DISTRIBUTED TRANSACTIONS
69. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE SYNCHRONIZE DATA ACCESS
70. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE CONSENSUS ALGORITHMS
71. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE CONCURRENCY FRIENDLY DATA STRUCTURES
72. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE COMMUNICATE OPERATIONS NOT
 VALUES
73. GITHUB.COM/ SEANTALLEN/DATA- CORRUPTING- ARCHITECTURES LEARN MORE