Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Data corrupting architectures we know and love

Data corrupting architectures we know and love

There’s an old distributed systems adage, “You can have a second computer when you learn how to use the first one.” When it comes to data-access patterns, most of our favorite patterns are unsafe on a single computer. Most of our applications assume a concurrency model where all access is serialized. What happens when that mental model meets distributed data?

Sean Allen reviews data race and corruption problems that exist on single-machine systems and shows how we’ve transferred many of those patterns over to distributed systems and distributed state. You’ll learn the basics of data races, deadlocks, and problems in even the simplest of our concurrent data access patterns and how our existing scaling patterns replicate the same issues across multiple machines, increasing the potential problems, as well as designs that can alleviate them.

Sean T Allen

May 18, 2019
Tweet

More Decks by Sean T Allen

Other Decks in Technology

Transcript

  1. WE KNOW AND LOVE
    DATA CORRUPTING ARCHITECTURES

    View Slide

  2. VP OF ENGINEERING @ WALLAROO LABS
    MEMBER OF THE PONY CORE TEAM
    AUTHOR OF “STORM APPLIED”
    CHICHARRONES LOVER
    @SEANTALLEN
    @WALLAROOLABS
    @PONYLANG
    SEAN T. ALLEN

    View Slide

  3. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    TODAY’S TOPICS INCLUDE…
    ▸ Data corruption
    ▸ Concurrency
    ▸ Data races
    ▸ Shared values
    ▸ Atomic operations

    View Slide

  4. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    GENERALIZATIONS
    AHEAD

    View Slide

  5. OUR MOST POPULAR ARCHITECTURES
    CAN BE TRACED TO WEB
    APPPLICATIONS.
    Sean T. Allen
    DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE

    View Slide

  6. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DATABASE
    WEB APP

    View Slide

  7. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    A BIT MORE DETAILED…
    REQUEST
    to the
    APPLICATION
    on to the
    DATABASE
    back to the
    APPLICATION
    and out as a
    RESPONSE

    View Slide

  8. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    WE COULD GET MORE DETAILED STILL…
    REQUEST
    to the
    APPLICATION
    on to the
    DATABASE
    back to the
    APPLICATION
    and out as a
    RESPONSE
    (WEB BROWSER)
    to the
    (WEB SERVER + DEVELOPER CODE)
    on to the
    DATABASE
    back to the
    (WEB SERVER + DEVELOPER CODE)
    and out as a
    (WEB BROWSER)

    View Slide

  9. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    OUR
    SIMPLE WEB APP IS
    A DISTRIBUTED SYSTEM

    View Slide

  10. DISTRIBUTED SYSTEMS AREN’T HARD.
    THEY’RE EASY*.
    Sean T. Allen
    DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    * sometimes

    View Slide

  11. DISTRIBUTED SYSTEMS AREN’T HARD.
    THEY’RE EASY*.
    CONCURRENCY IS THE HARD PART.
    Sean T. Allen
    DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    * sometimes

    View Slide

  12. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    SOME JUSTIFICATION
    AHEAD

    View Slide

  13. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DATABASE
    WEB APP

    View Slide

  14. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DATABASE
    WEB APP

    View Slide

  15. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DATABASE
    WEB APP

    View Slide

  16. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    AN EXAMPLE
    APPLICATION

    View Slide

  17. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return new value to the user

    View Slide

  18. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return original value to the user
    SPOT THE
    DATA CORRUPTION?

    View Slide

  19. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return new value to the user

    View Slide

  20. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return original value to the user
    LET’S ADD SOME
    CONCURRENCY

    View Slide

  21. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return new value to the user
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return new value to the user

    View Slide

  22. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return original value to the user
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return original value to the user
    WHAT’S THE FINAL VALUE?
    1?
    2?

    View Slide

  23. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    WE DON’T
    KNOW

    View Slide

  24. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THE PAGES SERVED COUNTER
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return new value to the user
    ‣ Read current value from the database
    ‣ Add one to update it
    ‣ Write new value to the database
    ‣ Return new value to the user

    View Slide

  25. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Write new value to the database 1
    ‣ Return new value to the user 1
    ‣ Return new value to the user 1
    Request A Request B

    View Slide

  26. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Return new value to the user 1
    ‣ Read current value from the database 1
    ‣ Add one to update it 2
    ‣ Write new value to the database 2
    ‣ Return new value to the user 2
    Request A Request B

    View Slide

  27. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Read current value from the database 1
    ‣ Return new value to the user 1
    ‣ Add one to update it 2
    ‣ Write new value to the database 2
    ‣ Return new value to the user 2
    Request A Request B

    View Slide

  28. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Read current value from the database 1
    ‣ Return new value to the user 1
    ‣ Add one to update it 2
    ‣ Write new value to the database 2
    ‣ Return new value to the user 2
    Request A Request B

    View Slide

  29. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Read current value from the database 1
    ‣ Return new value to the user 1
    ‣ Add one to update it 2
    ‣ Write new value to the database 2
    ‣ Return new value to the user 2
    Request A Request B

    View Slide

  30. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    WE’RE EXPERIENCING
    A DATA RACE

    View Slide

  31. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    RECIPE FOR A DATA RACE
    ▸ A shared value
    ▸ Concurrency
    ▸ At least one writer changing the shared value
    ▸ Non-atomic operations

    View Slide

  32. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    OUR DATABASE IS
    A SHARED VALUE

    View Slide

  33. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    CONCURRENCY:
    WE HAVE TWO REQUESTS

    View Slide

  34. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    WHOLE LOTTA
    WRITING
    GOING ON

    View Slide

  35. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    OUR INCREMENTING
    ISN’T ATOMIC

    View Slide

  36. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    AN OPERATION IS ATOMIC IF IT’S
    ISOLATED FROM OTHER
    OPERATIONS HAPPENING AT THE
    SAME TIME.

    View Slide

  37. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Write new value to the database 1
    ‣ Return new value to the user 1
    ‣ Return new value to the user 1
    Request A Request B

    View Slide

  38. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Read current value from the database 0
    ‣ Read current value from the database 0
    ‣ Add one to update it 1
    ‣ Add one to update it 1
    ‣ Write new value to the database 1
    ‣ Write new value to the database 1
    ‣ Return new value to the user 1
    ‣ Return new value to the user 1
    Request A Request B

    View Slide

  39. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Increment current value 1
    ‣ Increment current value 2
    ‣ Read new value 2
    ‣ Read new value 2
    ‣ Return new value to the user 2
    ‣ Return new value to the user 2
    Request A Request B

    View Slide

  40. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ‣ Increment current value 1
    ‣ Increment current value 2
    ‣ Read new value 2
    ‣ Read new value 2
    ‣ Return new value to the user 2
    ‣ Return new value to the user 2
    Request A Request B

    View Slide

  41. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    MORE CONCURRENCY
    MORE PROBLEMS

    View Slide

  42. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C

    View Slide

  43. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C

    View Slide

  44. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C

    View Slide

  45. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C

    View Slide

  46. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DATABASE VALUE
    IS 1

    View Slide

  47. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C

    View Slide

  48. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C

    View Slide

  49. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C

    View Slide

  50. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DATABASE VALUE
    INCREASED TO 2

    View Slide

  51. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C

    View Slide

  52. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    0
    0
    1
    1
    1 2
    Request A
    Request B
    Request C

    View Slide

  53. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DATABASE VALUE
    DECREASED TO 1

    View Slide

  54. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    WE’RE EXPERIENCING
    LAST WRITE WINS

    View Slide

  55. LAST WRITE WINS MEANS “SOME
    WRITES LOSE”.
    Sean Cribbs & Russell Brown
    DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE

    View Slide

  56. LAST WRITE WINS IS A FANCY NAME
    FOR “LET’S CORRUPT SOME DATA”.
    Sean T. Allen
    DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE

    View Slide

  57. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    OUR MOST POPULAR
    ARCHITECTURES FOR SCALING

    View Slide

  58. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    THESE PROBLEMS
    ARE EVERYWHERE

    View Slide

  59. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    “SERVERLESS”

    View Slide

  60. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    “FUNCTIONS AS
    SERVICE”

    View Slide

  61. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    “STATELESS”

    View Slide

  62. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    “SHARED NOTHING”

    View Slide

  63. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ALL FEATURE
    DATABASE AS A SHARED VALUE
    AND CONCURRENT WRITES

    View Slide

  64. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    ALL HAVE
    DATA CORRUPTION
    BAKED INTO THE CORE ARCHITECTURE

    View Slide

  65. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    WHAT
    CAN WE DO?

    View Slide

  66. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    RECIPE FOR A DATA RACE: CHANGE AT LEAST ONE
    ▸ A shared value
    ▸ Concurrency
    ▸ At least one writer changing the shared value
    ▸ Non-atomic operations

    View Slide

  67. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    STORED PROCEDURES
    AND
    TRANSACTIONS

    View Slide

  68. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    DISTRIBUTED
    TRANSACTIONS

    View Slide

  69. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    SYNCHRONIZE
    DATA ACCESS

    View Slide

  70. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    CONSENSUS
    ALGORITHMS

    View Slide

  71. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    CONCURRENCY
    FRIENDLY DATA STRUCTURES

    View Slide

  72. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
    COMMUNICATE
    OPERATIONS
    NOT

    VALUES

    View Slide

  73. GITHUB.COM/
    SEANTALLEN/DATA-
    CORRUPTING-
    ARCHITECTURES
    LEARN MORE

    View Slide