Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

designing for concurrency with riak

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Mathias Meyer Mathias Meyer
May 29, 2012
Programming
1.9k
11

designing for concurrency with riak

http://riakhandbook.com

Avatar for Mathias Meyer

Mathias Meyer

May 29, 2012

More Decks by Mathias Meyer

See All by Mathias Meyer
Building and Scaling an Distributed and Inclusive Team
Avatar for Mathias Meyer roidrage
0
1.5k
cooking infrastructure with chef
Avatar for Mathias Meyer roidrage
4
270
The Message Queue is Dead, Long Live the Message Queue
Avatar for Mathias Meyer roidrage
4
740
riak-js
Avatar for Mathias Meyer roidrage
1
330
metrics, monitoring, logging
Avatar for Mathias Meyer roidrage
82
15k
design for cloud - jax 2012
Avatar for Mathias Meyer roidrage
2
340
A Riak Query Tale
Avatar for Mathias Meyer roidrage
5
1.1k
Don't Use NoSQL
Avatar for Mathias Meyer roidrage
10
1.1k
Designing Applications for Amazon Web Services (GOTO Aarhus)
Avatar for Mathias Meyer roidrage
6
390

Other Decks in Programming

See All in Programming
SODA - FACT BOOK（EN）
Avatar for SODA inc. sodainc
1
200
私なりのハーネスエンジニアリング ～独自の言語処理系の開発を通して～
Avatar for 瀬尾佳隆 seosoft
0
180
小さなアプリを作って公開するまで
Avatar for onozaty onozaty
0
110
Herb in Rails 8.2: Your ERB views, now HTML-aware @ Kaigi on Rails 2026, Tokyo, Japan
Avatar for Marco Roth marcoroth
2
400
動画プレイヤーはなぜカクつくのか 〜Flutterのレンダリングとネイティブの境界を解剖する〜
Avatar for Naruo Yoshiki naruogram
0
320
エージェントを使い続けるためのメタワーク / Meta-work for keeping using the agent
Avatar for r-kagaya rkaga
6
2.2k
Coding with AI: How to trust the storyteller (AWS Community Day Adria, Ljubljana 2026)
Avatar for Slobodan Stojanović slobodan
0
140
느린 네트워크와 저사양 디바이스 사용자를 위한 웹뷰 로딩 시간 개선기
Avatar for kmkim kmkim
1
180
作れる、でも消せるか？──安全に「枯らして消す」機能削除術 / Drain, Then Delete
Avatar for chobishiba chobishiba
3
1.4k
Event is what happens
Avatar for MOROHASHI Kyosuke moro
2
3.4k
AI時代のTerminal環境を UNIX哲学で設計・運用する
Avatar for yasunori yasunori0418
1
340
flutter_test のその先へ — Flutter アプリの LLM 機能に「評価」という品質保証を
Avatar for Kosuke Saigusa kosukesaigusa
0
180

Featured

See All Featured
B2B Lead Gen: Tactics, Traps & Triumph
Avatar for Sophie Logan marketingsoph
1
300
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
66
8.7k
Skip the Path - Find Your Career Trail
Avatar for Mark Kilby mkilby
1
280
Conquering PDFs: document understanding beyond plain text
Avatar for Ines Montani inesmontani
PRO
4
3.3k
Designing for Performance
Avatar for Lara Hogan lara
612
70k
Impact Scores and Hybrid Strategies: The future of link building
Avatar for Tamara Novitovic tamaranovitovic
0
480
How to Align SEO within the Product Triangle To Get 
Buy-In & Support - #RIMC
Avatar for Aleyda Solis aleyda
2
1.9k
Dealing with People You Can't Stand - Big Design 2015
Avatar for Cassini Nazir cassininazir
367
27k
The Art of Delivering Value - GDevCon NA Keynote
Avatar for David Neal reverentgeek
16
2.2k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
76
5.4k
Product Roadmaps are Hard
Avatar for C. Todd Lombardo iamctodd
55
13k
Getting science done with accelerated Python computing platforms
Avatar for Jacob Tomlinson jacobtomlinson
2
560

Transcript

  1. designing for concurrency with riak nosql matters mathias meyer, @roidrage

  2. None

  3. http://riakhandbook.com

  4. design for concurrency?

  5. design data for concurrency

  6. data starts out simple

  7. ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3

    karen [email protected]

  8. single source of truth

  9. always consistent

  10. mostly consistent

  11. monotonic

  12. increase number of sources

  13. replication

  14. ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3

    karen [email protected] ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3 karen [email protected]

  15. ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3

    karen [email protected] ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3 karen [email protected] ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3 karen [email protected]

  16. ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3

    karen [email protected] ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3 karen [email protected] ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3 karen [email protected] ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3 karen [email protected] ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3 karen [email protected] ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3 karen [email protected]

  17. eventual consistency* * if no new updates are made to

    the object, eventually all accesses will return the last updated value. werner vogels, 2008, http://queue.acm.org/detail.cfm?id=1466448

  18. multiple clients

  19. ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3

    karen [email protected] ID Username Email 1 roidrage [email protected] 2 thomas [email protected] 3 karen [email protected] Client 1 Client 2 PUT PUT

  20. conflicting writes

  21. siblings

  22. data diverges

  23. the challenge

  24. determine the winner

  25. determine order

  26. designing data for concurrency

  27. designing data for non-monotonic writes

  28. no atomicity in riak

  29. no coordination

  30. all state is in the data

  31. (eventual) consistency and logical monoticity * hellerstein: the declarative imperative:

    experiences and conjectures in distributed logic (2010)

  32. designing data with conflicts in mind

  33. write now, converge later

  34. rethink the data structures

  35. ID Username Email 1 roidrage [email protected] {    "id":  1,

       "username":  "roidrage",    "email":  "[email protected]" }

  36. track updates

  37. {    "id":  1,    "username":  "roidrage",    "email":  "[email protected]"

       "changes":  [        {            "client":  "client-­‐1",            "timestamp":  1337001337,            "updates":  [                "firstname":  "Mathias",                "lastname":  "Meyer"            ]        }    ] }

  38. {    "id":  1,    "username":  "roidrage",    "email":  "[email protected]"

       "changes":  [  {            "client":  "client-­‐2",            "timestamp":  1337001337,            "updates":  [                "email":  "[email protected]"            ]        }    ] }

  39. apply all updates ordered by time

  40. what about removing data?

  41. {    "id":  1,    "username":  "roidrage",    "email":  "[email protected]"

       "changes":  [{        "client":  "client-­‐1",        "timestamp":  1337001337,        "updates":  [  {                "_op":  "delete",                "attribute":  "email"            }        ]    }] }

  42. {    "id":  1,    "username":  "roidrage",    "email":  "[email protected]"

       "changes":  [{        "client":  "client-­‐2",        "timestamp":  1337001337,        "updates":  [            {                "_op":  "add",                "attribute":  "email",                "value":  "[email protected]"            }        ]    }] }

  43. keep a changelog

  44. client converges data

  45. time as a means of ordering* * leslie lamport, et.

    al.: time, clocks and the ordering of events in a distributed system (1977)

  46. time is not a guarantee for uniqueness

  47. vector clocks?

  48. {    "id":  1,    "username":  "roidrage",    "email":  "[email protected]"

       "changes":  [{        "id":  "ca0cb932-­‐a74e-­‐11e1-­‐9ce4-­‐1093e90b5d80",        "timestamp":  1337001337,        "updates":  [            {                "_op":  "delete",                "attribute":  "email"            }        ]    ] }

  49. timelines* * riak at yammer: http://basho.com/blog/technical/2011/03/28/Riak-and-Scala-at-Yammer/

  50. time-ordered series of events

  51. kept per user

  52. {    "events":  [        {    

           "id":  "ca0cb932-­‐a74e-­‐11e1-­‐9ce4-­‐1093e90b5d80",            "timestamp":  1337001337,            "event":  {                "type":  "push",                "repository":  "rails/rails",                "sha1":  "0ea43bf"            }        },  {            "id":  "e018f024-­‐a74e-­‐11e1-­‐9feb-­‐1093e90b5d80", "timestamp":  1337001337, "event":  {    "type":  "pull_request",    "repository":  "rails/rails",    "sha1":  "84efda0" }          }    ] }

  53. clients dedup, sort and truncate

  54. observation: clients manage the data

  55. sets, counters, graphs

  56. monotonic data structures

  57. sets

  58. an unordered bag of unique items

  59. simplest thing that could possibly work...in riak

  60. secondary indexes

  61. X-­‐Riak-­‐Index-­‐tags_bin:  nosql,  cloud,  infrastructure {    "id":  1,    "username":

     "roidrage",    "email":  "[email protected]" }

  62. always unique

  63. useful for simple things

  64. useful for object associations

  65. add-only

  66. set: time-ordered list of operations

  67. {    "set":  [        {    

           "id":  "e018f024-­‐a74e-­‐11e1-­‐9feb-­‐1093e90b5d80",            "timestamp":  1337001337,            "op":  "add",            "value":  "roidrage"        }    ] }

  68. {    "set":  [        {    

           "id":  "e018f024-­‐a74e-­‐11e1-­‐9feb-­‐1093e90b5d80",            "timestamp":  1337001337,            "op":  "add",            "value":  "roidrage"        },  {            "id":  "56707cee-­‐a757-­‐11e1-­‐8e1b-­‐1093e90b5d80",            "timestamp":  1337001339,            "op":  "add",            "value":  "josh"        }    ] }

  69. {    "set":  [        {    

           "id":  "e018f024-­‐a74e-­‐11e1-­‐9feb-­‐1093e90b5d80",            "timestamp":  1337001337,            "op":  "add",            "value":  "roidrage"        },  {            "id":  "56707cee-­‐a757-­‐11e1-­‐8e1b-­‐1093e90b5d80",            "timestamp":  1337001339,            "op":  "add",            "value":  "josh"        },  {            "id":  "a525f16c-­‐a968-­‐11e1-­‐8b07-­‐1093e90b5d80",            "timestamp":  1337001343,            "op":  "remove",            "value":  "josh"        }    ] }

  70. slightly inefficient

  71. 2-phase set* * https://github.com/aphyr/meangirls

  72. {    "set":  {        "adds":  ["roidrage",  "josh"],

           "removes":  ["josh"]    } }

  73. counters

  74. increment, decrement

  75. {    "counter":  [        {    

           "id":  "e018f024-­‐a74e-­‐11e1-­‐9feb-­‐1093e90b5d80",            "timestamp":  1337001337,            "op":  "incr",            "value":  4        }    ], }

  76. g-counters* *a comprehensive study of convergent and commutative replicated data

    types http://hal.inria.fr/docs/00/55/55/88/PDF/techreport.pdf

  77. {    "elements":  {        "client-­‐1":  1,  

         "client-­‐2":  3,        "client-­‐3":  5    } } value = 1 + 3 + 5 = 9

  78. counters are easy when you increment only

  79. convergent replicated data types *shapiro et. al.: a comprehensive study

    of convergent and commutative replicated data types http://hal.inria.fr/docs/00/55/55/88/PDF/techreport.pdf

  80. statebox for erlang* * https://github.com/mochi/statebox

  81. knockbox for clojure* * https://github.com/reiddraper/knockbox

  82. data represents state

  83. state-based means growth

  84. data increases with lots of updates

  85. dealing with growth

  86. truncate

  87. roll up, discard

  88. {    "counter":  [{        "id":  "458f5936-­‐a752-­‐11e1-­‐a876-­‐1093e90b5d80",  

         "timestamp":  1337001347,        "op":  "inc",        "value":  1    }],    "value":  2 }

  89. garbage collection

  90. not easy with riak

  91. not easy with stateful data

  92. garbage collection requires coordination

  93. network partitions cause stale data

  94. the solution?

  95. trade off data size vs. consistency

  96. commutative replicated data types* *shapiro et. al.: a comprehensive study

    of convergent and commutative replicated data types http://hal.inria.fr/docs/00/55/55/88/PDF/techreport.pdf

  97. operations instead of state

  98. not yet possible with riak

  99. eventual consistency is hard

  100. thanks