Data corrupting architectures we know and love

There’s an old distributed systems adage, “You can have a second computer when you learn how to use the first one.” When it comes to data-access patterns, most of our favorite patterns are unsafe on a single computer. Most of our applications assume a concurrency model where all access is serialized. What happens when that mental model meets distributed data?

Sean Allen reviews data race and corruption problems that exist on single-machine systems and shows how we’ve transferred many of those patterns over to distributed systems and distributed state. You’ll learn the basics of data races, deadlocks, and problems in even the simplest of our concurrent data access patterns and how our existing scaling patterns replicate the same issues across multiple machines, increasing the potential problems, as well as designs that can alleviate them.

Transcript

1. WE KNOW AND LOVE
   DATA CORRUPTING ARCHITECTURES
   

   View Slide

2. VP OF ENGINEERING @ WALLAROO LABS
   MEMBER OF THE PONY CORE TEAM
   AUTHOR OF “STORM APPLIED”
   CHICHARRONES LOVER
   @SEANTALLEN
   @WALLAROOLABS
   @PONYLANG
   SEAN T. ALLEN
   

   View Slide

3. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
   TODAY’S TOPICS INCLUDE…
   ▸ Data corruption
   ▸ Concurrency
   ▸ Data races
   ▸ Shared values
   ▸ Atomic operations
   

   View Slide

4. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
   GENERALIZATIONS
   AHEAD
   

   View Slide

5. OUR MOST POPULAR ARCHITECTURES
   CAN BE TRACED TO WEB
   APPPLICATIONS.
   Sean T. Allen
   DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
   

   View Slide

6. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
   DATABASE
   WEB APP
   

   View Slide

7. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
   A BIT MORE DETAILED…
   REQUEST
   to the
   APPLICATION
   on to the
   DATABASE
   back to the
   APPLICATION
   and out as a
   RESPONSE
   

   View Slide

8. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
   WE COULD GET MORE DETAILED STILL…
   REQUEST
   to the
   APPLICATION
   on to the
   DATABASE
   back to the
   APPLICATION
   and out as a
   RESPONSE
   (WEB BROWSER)
   to the
   (WEB SERVER + DEVELOPER CODE)
   on to the
   DATABASE
   back to the
   (WEB SERVER + DEVELOPER CODE)
   and out as a
   (WEB BROWSER)
   

   View Slide

9. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
   OUR
   SIMPLE WEB APP IS
   A DISTRIBUTED SYSTEM
   

   View Slide

10. DISTRIBUTED SYSTEMS AREN’T HARD.
    THEY’RE EASY*.
    Sean T. Allen
    DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    * sometimes
    

    View Slide

11. DISTRIBUTED SYSTEMS AREN’T HARD.
    THEY’RE EASY*.
    CONCURRENCY IS THE HARD PART.
    Sean T. Allen
    DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    * sometimes
    

    View Slide

12. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    SOME JUSTIFICATION
    AHEAD
    

    View Slide

13. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DATABASE
    WEB APP
    

    View Slide

14. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DATABASE
    WEB APP
    

    View Slide

15. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DATABASE
    WEB APP
    

    View Slide

16. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    AN EXAMPLE
    APPLICATION
    

    View Slide

17. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return new value to the user
    

    View Slide

18. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return original value to the user
    SPOT THE
    DATA CORRUPTION?
    

    View Slide

19. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return new value to the user
    

    View Slide

20. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return original value to the user
    LET’S ADD SOME
    CONCURRENCY
    

    View Slide

21. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return new value to the user
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return new value to the user
    

    View Slide

22. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return original value to the user
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return original value to the user
    WHAT’S THE FINAL VALUE?
    1?
    2?
    

    View Slide

23. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    WE DON’T
    KNOW
    

    View Slide

24. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return new value to the user
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return new value to the user
    

    View Slide

25. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Write new value to the database 1
    ‣ Return new value to the user 1
    ‣ Return new value to the user 1
    Request A Request B
    

    View Slide

26. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Return new value to the user 1
    ‣ Read current value from the database 1
    ‣ Add one to update it 2
    ‣ Write new value to the database 2
    ‣ Return new value to the user 2
    Request A Request B
    

    View Slide

27. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Read current value from the database 1
    ‣ Return new value to the user 1
    ‣ Add one to update it 2
    ‣ Write new value to the database 2
    ‣ Return new value to the user 2
    Request A Request B
    

    View Slide

28. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Read current value from the database 1
    ‣ Return new value to the user 1
    ‣ Add one to update it 2
    ‣ Write new value to the database 2
    ‣ Return new value to the user 2
    Request A Request B
    

    View Slide

29. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Read current value from the database 1
    ‣ Return new value to the user 1
    ‣ Add one to update it 2
    ‣ Write new value to the database 2
    ‣ Return new value to the user 2
    Request A Request B
    

    View Slide

30. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    WE’RE EXPERIENCING
    A DATA RACE
    

    View Slide

31. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    RECIPE FOR A DATA RACE
    ▸ A shared value
    ▸ Concurrency
    ▸ At least one writer changing the shared value
    ▸ Non-atomic operations
    

    View Slide

32. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    OUR DATABASE IS
    A SHARED VALUE
    

    View Slide

33. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    CONCURRENCY:
    WE HAVE TWO REQUESTS
    

    View Slide

34. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    WHOLE LOTTA
    WRITING
    GOING ON
    

    View Slide

35. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    OUR INCREMENTING
    ISN’T ATOMIC
    

    View Slide

36. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    AN OPERATION IS ATOMIC IF IT’S
    ISOLATED FROM OTHER
    OPERATIONS HAPPENING AT THE
    SAME TIME.
    

    View Slide

37. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Write new value to the database 1
    ‣ Return new value to the user 1
    ‣ Return new value to the user 1
    Request A Request B
    

    View Slide

38. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Write new value to the database 1
    ‣ Return new value to the user 1
    ‣ Return new value to the user 1
    Request A Request B
    

    View Slide

39. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Increment current value 1
    ‣ Increment current value 2
    ‣ Read new value 2
    ‣ Read new value 2
    ‣ Return new value to the user 2
    ‣ Return new value to the user 2
    Request A Request B
    

    View Slide

40. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Increment current value 1
    ‣ Increment current value 2
    ‣ Read new value 2
    ‣ Read new value 2
    ‣ Return new value to the user 2
    ‣ Return new value to the user 2
    Request A Request B
    

    View Slide

41. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    MORE CONCURRENCY
    MORE PROBLEMS
    

    View Slide

42. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C
    

    View Slide

43. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C
    

    View Slide

44. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C
    

    View Slide

45. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C
    

    View Slide

46. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DATABASE VALUE
    IS 1
    

    View Slide

47. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C
    

    View Slide

48. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C
    

    View Slide

49. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C
    

    View Slide

50. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DATABASE VALUE
    INCREASED TO 2
    

    View Slide

51. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C
    

    View Slide

52. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C
    

    View Slide

53. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DATABASE VALUE
    DECREASED TO 1
    

    View Slide

54. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    WE’RE EXPERIENCING
    LAST WRITE WINS
    

    View Slide

55. LAST WRITE WINS MEANS “SOME
    WRITES LOSE”.
    Sean Cribbs & Russell Brown
    DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    

    View Slide

56. LAST WRITE WINS IS A FANCY NAME
    FOR “LET’S CORRUPT SOME DATA”.
    Sean T. Allen
    DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    

    View Slide

57. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    OUR MOST POPULAR
    ARCHITECTURES FOR SCALING
    

    View Slide

58. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THESE PROBLEMS
    ARE EVERYWHERE
    

    View Slide

59. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    “SERVERLESS”
    

    View Slide

60. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    “FUNCTIONS AS
    SERVICE”
    

    View Slide

61. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    “STATELESS”
    

    View Slide

62. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    “SHARED NOTHING”
    

    View Slide

63. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ALL FEATURE
    DATABASE AS A SHARED VALUE
    AND CONCURRENT WRITES
    

    View Slide

64. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ALL HAVE
    DATA CORRUPTION
    BAKED INTO THE CORE ARCHITECTURE
    

    View Slide

65. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    WHAT
    CAN WE DO?
    

    View Slide

66. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    RECIPE FOR A DATA RACE: CHANGE AT LEAST ONE
    ▸ A shared value
    ▸ Concurrency
    ▸ At least one writer changing the shared value
    ▸ Non-atomic operations
    

    View Slide

67. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    STORED PROCEDURES
    AND
    TRANSACTIONS
    

    View Slide

68. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DISTRIBUTED
    TRANSACTIONS
    

    View Slide

69. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    SYNCHRONIZE
    DATA ACCESS
    

    View Slide

70. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    CONSENSUS
    ALGORITHMS
    

    View Slide

71. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    CONCURRENCY
    FRIENDLY DATA STRUCTURES
    

    View Slide

72. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    COMMUNICATE
    OPERATIONS
    NOT

    VALUES
    

    View Slide

73. GITHUB.COM/
    SEANTALLEN/DATA-
    CORRUPTING-
    ARCHITECTURES
    LEARN MORE
    

    View Slide