Consistency and Riak

Consistency and Riak

Christopher Meiklejohn
Riak Meetup, Paris, February 2015

3e09fee7b359be847ed5fa48f524a3d3?s=128

Christopher Meiklejohn

February 25, 2015
Tweet

Transcript

  1. 2.
  2. 10.
  3. 13.

    node 0 node 1 node 2 Replicas are stored to

    the N - 1 contiguous partitions
  4. 20.
  5. 21.
  6. 26.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    hash(users/clay-davis) == 10, 11, 12 Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
  7. 27.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    get(users/clay-davis) Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
  8. 28.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring R=2
  9. 29.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring R=2 obj
  10. 39.

    merkle tree to track changes coordinated at the vnode level

    runs as a background process exchange with neighbor vnodes for inconsistencies resolution semantics: trigger read-repair
  11. 41.
  12. 42.
  13. 43.
  14. 44.
  15. 50.
  16. 51.
  17. 52.
  18. 57.

    [{a, v1, t1}] [{b, v1, t2}] [{a, v1, t1}] [{b,

    v1, t2}] [{b, v1, t2}] [{b, v1, t2}] Last Writer Wins
  19. 58.

    [{a, v1, t1}] [{b, v1, t2}] [{a, v1, t1}] [[{a,

    v1, t1}, {b, v1, t2}] [[{a, v1, t1}, {b, v1, t2}] [[{a, v1, t1}, {b, v1, t2}] Allow Mult
  20. 65.
  21. 66.
  22. 67.
  23. 68.
  24. 77.

    Associativity: (X · Y) · Z = X · (Y

    · Z) Bounded Join-Semilattice
  25. 81.

    b a c a, b a, c a, b, c

    Set; merge function: union. b, c
  26. 87.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

    [ [{1, a}], [{1, a}] ] [ [{1, a}], [{1, a}] ]
  27. 88.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

    [ [{1, a}], [{1, a}] ] [ [{1, a}], [{1, a}] ] [ [{1, a}, {2, a}], [{1, a}] ]
  28. 89.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

    [ [{1, a}], [{1, a}] ] [ [{1, a}], [{1, a}] ] [ [{1, a}, {2, a}], [{1, a}] ] [ [{1, a}, {2, a}], [{1, a}] ]
  29. 91.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

    [ [{1, a}, {2, b}], [] ]
  30. 92.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

    [ [{1, a}, {2, b}], [] ] [ [{1, a}], [{1, a}] ]
  31. 93.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

    [ [{1, a}, {2, b}], [] ] [ [{1, a}], [{1, a}] ] [ [{1, a}, {2, b}], [{1, a}] ]
  32. 97.
  33. 100.
  34. 106.
  35. 107.

    Distributed Consensus The problem of reaching agreement among remote processes

    is one of the most fundamental problems in distributed computing and is at the core of many algorithms for distributed data processing, distributed file management, and fault-tolerant distributed applications. Fischer, Lynch, Paterson
  36. 115.

    Node 1 Node 2 Node 3 N++ prepare(N) promise(N, Vb)

    promise(N, Vc) Vn = f(Va, Vb, Vc) commit(N, Vn) accept(N)
  37. 117.

    Node 1 Node 2 Node 3 N++; I = 0

    prepare(N, I) promise(N, I, Vb) promise(N, I, Vc) Vn = f(Va, Vb, Vc) commit(N, I, Vn) accept(N, I)
  38. 121.
  39. 127.
  40. 128.
  41. 129.
  42. 130.
  43. 131.
  44. 133.
  45. 136.

    Node 1 Node 2 Node 3 obj.epoch < epoch get(key)

    reply(Epochb, Seqb, Valb) Val = latest(Vala, Valb, Valc) Val.epoch = epoch write(Epoch, ++Seq, Val) ack(Epoch, Seq) reply(Epochc, Seqc, Valc)
  46. 140.

    Node 1 Node 2 Node 3 obj.epoch < epoch get(key)

    reply(Epochb, Seqb, Valb) Latest = latest(Vala, Valb, Valc) Val = modify(Latest) write(Epoch, ++Seq, Val) ack(Epoch, Seq) reply(Epochc, Seqc, Valc)
  47. 141.

    Node 1 Node 2 Node 3 obj.epoch == epoch Latest

    = local_get(Key) Val = modify(Latest) write(Epoch, ++Seq, Val) ack(Epoch, Seq)
  48. 146.

    Existing Ensemble Joining Ensemble riak_01 riak_02 riak_03 riak_07 riak_08 riak_09

    [{riak_01}, {riak_02}, {riak_03}] [{riak_07}, {riak_08}, {riak_09}]
  49. 151.

    $ riak-admin bucket-type create strongly_consistent \ ‘{"props":{"consistent":true}}' $ riak-admin bucket-type

    status strongly_consistent $ riak-admin bucket-type activate strongly_consistent Enable strong consistency; http://docs.basho.com/riak/latest/dev/advanced/strong-consistency/
  50. 153.

    $ riak-admin bucket-type create maps \ '{"props":{"datatype":"map"}}' $ riak-admin bucket-type

    create sets \ '{"props":{"datatype":"set"}}' $ riak-admin bucket-type create counters \ ‘{“props":{"datatype":"counter"}}' $ riak-admin bucket-type status maps $ riak-admin bucket-type activate maps Create bucket type for data types; http://docs.basho.com/riak/latest/dev/using/data-types/
  51. 154.

    $ curl -XPOST http://localhost:10018/types/counters/buckets/counters/ datatypes/traffic_tickets \ -H "Content-Type: application/json" \

    -d '{"increment": 1}’ $ curl http://localhost:10018/types/counters/buckets/counters/ datatypes/traffic_tickets Operate on counters; http://docs.basho.com/riak/latest/dev/using/data-types/
  52. 155.

    $ curl -XPOST http://localhost:10018/types/sets/buckets/travel/ datatypes/cities \ -H "Content-Type: application/json" \

    -d '{"add_all":["Toronto", “Montreal"]}' $ curl -XPOST http://localhost:10018/types/sets/buckets/travel/ datatypes/cities \ -H "Content-Type: application/json" \ -d '{"remove": “Montreal"}' $ curl http://localhost:10018/types/sets/buckets/travel/datatypes/ cities Operate on sets; http://docs.basho.com/riak/latest/dev/using/data-types/
  53. 156.

    $ curl -XPOST http://localhost:10018/types/maps/buckets/customers/ datatypes/ahmed_info \ -H "Content-Type: application/json" \

    -d ' { "update": { "first_name_register": "Ahmed", "phone_number_register": "5551234567" } }' $ curl -XPOST http://localhost:8098/types/maps/buckets/customers/ datatypes/ahmed_info \ -H "Content-Type: application/json" \ -d ' { "update": { "annika_info_map": { "update": { "interests_set": { "add": "tango dancing" } } } } } ' Operate on maps; http://docs.basho.com/riak/latest/dev/using/data-types/
  54. 157.