Introduction to Riak

Introduction to Riak

BOBkonf, 2015
Berlin, Germany
Tutorial

3e09fee7b359be847ed5fa48f524a3d3?s=128

Christopher Meiklejohn

January 23, 2015
Tweet

Transcript

  1. 2.
  2. 9.

    $ curl -O http://s3.amazonaws.com/downloads.basho.com/erlang/ otp_src_R16B02-basho5.tar.gz $ tar -xvf otp_src_R16B02-basho5.tar.gz $

    cd otp_src_R16B02-basho5 $ ./configure && make && sudo make install Installing Erlang
  3. 14.

    $ dev2/bin/riak-admin cluster join dev1@127.0.0.1 $ dev3/bin/riak-admin cluster join dev1@127.0.0.1

    $ dev4/bin/riak-admin cluster join dev1@127.0.0.1 $ dev5/bin/riak-admin cluster join dev1@127.0.0.1 Stage a join
  4. 16.

    =============================== Staged Changes ================================ Action Nodes(s) ------------------------------------------------------------------------------- join 'dev2@127.0.0.1' join

    'dev3@127.0.0.1' join 'dev4@127.0.0.1' join 'dev5@127.0.0.1' ------------------------------------------------------------------------------- NOTE: Applying these changes will result in 1 cluster transition ############################################################################### After cluster transition 1/1 ############################################################################### ================================= Membership ================================== Status Ring Pending Node ------------------------------------------------------------------------------- valid 100.0% 20.3% 'dev1@127.0.0.1' valid 0.0% 20.3% 'dev2@127.0.0.1' valid 0.0% 20.3% 'dev3@127.0.0.1' valid 0.0% 20.3% 'dev4@127.0.0.1' valid 0.0% 18.8% 'dev5@127.0.0.1' ------------------------------------------------------------------------------- Valid:5 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 Transfers resulting from cluster changes: 51 12 transfers from 'dev1@127.0.0.1' to 'dev5@127.0.0.1' 13 transfers from 'dev1@127.0.0.1' to 'dev4@127.0.0.1' 13 transfers from 'dev1@127.0.0.1' to 'dev3@127.0.0.1' 13 transfers from 'dev1@127.0.0.1' to 'dev2@127.0.0.1' View a staged plan
  5. 19.

    ================================= Membership ================================== Status Ring Pending Node ------------------------------------------------------------------------------- valid 20.3%

    -- 'dev1@127.0.0.1' valid 20.3% -- 'dev2@127.0.0.1' valid 20.3% -- 'dev3@127.0.0.1' valid 20.3% -- 'dev4@127.0.0.1' valid 18.8% -- 'dev5@127.0.0.1' ------------------------------------------------------------------------------- Valid:5 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 View members of cluster
  6. 26.
  7. 29.

    node 0 node 1 node 2 Replicas are stored to

    the N - 1 contiguous partitions
  8. 36.
  9. 37.
  10. 42.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    hash(users/clay-davis) == 10, 11, 12 Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
  11. 43.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    get(users/clay-davis) Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
  12. 44.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring R=2
  13. 45.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring R=2 obj
  14. 55.

    merkle tree to track changes coordinated at the vnode level

    runs as a background process exchange with neighbor vnodes for inconsistencies resolution semantics: trigger read-repair
  15. 57.
  16. 58.
  17. 59.
  18. 60.
  19. 66.
  20. 67.
  21. 68.
  22. 72.

    Eventual Consistency Eventual consistency is a consistency model used in

    distributed computing that informally guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value. Wikipedia
  23. 75.

    [{a, v1, t1}] [{b, v1, t2}] [{a, v1, t1}] [{b,

    v1, t2}] [{b, v1, t2}] [{b, v1, t2}] Last Writer Wins
  24. 76.

    [{a, v1, t1}] [{b, v1, t2}] [{a, v1, t1}] [[{a,

    v1, t1}, {b, v1, t2}] [[{a, v1, t1}, {b, v1, t2}] [[{a, v1, t1}, {b, v1, t2}] Allow Mult
  25. 84.
  26. 87.
  27. 93.
  28. 94.

    Distributed Consensus The problem of reaching agreement among remote processes

    is one of the most fundamental problems in distributed computing and is at the core of many algorithms for distributed data processing, distributed file management, and fault-tolerant distributed applications. Fischer, Lynch, Paterson
  29. 102.

    Node 1 Node 2 Node 3 N++ prepare(N) promise(N, Vb)

    promise(N, Vc) Vn = f(Va, Vb, Vc) commit(N, Vn) accept(N)
  30. 104.

    Node 1 Node 2 Node 3 N++; I = 0

    prepare(N, I) promise(N, I, Vb) promise(N, I, Vc) Vn = f(Va, Vb, Vc) commit(N, I, Vn) accept(N, I)
  31. 108.
  32. 114.
  33. 115.
  34. 116.
  35. 117.
  36. 118.
  37. 120.
  38. 123.

    Node 1 Node 2 Node 3 obj.epoch < epoch get(key)

    reply(Epochb, Seqb, Valb) Val = latest(Vala, Valb, Valc) Val.epoch = epoch write(Epoch, ++Seq, Val) ack(Epoch, Seq) reply(Epochc, Seqc, Valc)
  39. 127.

    Node 1 Node 2 Node 3 obj.epoch < epoch get(key)

    reply(Epochb, Seqb, Valb) Latest = latest(Vala, Valb, Valc) Val = modify(Latest) write(Epoch, ++Seq, Val) ack(Epoch, Seq) reply(Epochc, Seqc, Valc)
  40. 128.

    Node 1 Node 2 Node 3 obj.epoch == epoch Latest

    = local_get(Key) Val = modify(Latest) write(Epoch, ++Seq, Val) ack(Epoch, Seq)
  41. 133.

    Existing Ensemble Joining Ensemble riak_01 riak_02 riak_03 riak_07 riak_08 riak_09

    [{riak_01}, {riak_02}, {riak_03}] [{riak_07}, {riak_08}, {riak_09}]
  42. 139.
  43. 140.

    $ curl -XPUT http://localhost:10018/buckets/training/keys/foo \ -H 'Content-Type: text/plain' \ -d

    'caremad data goes here' $ curl -XPUT http://localhost:10018/buckets/training/keys/bar \ -H 'Content-Type: text/plain' \ -d 'caremad caremad caremad caremad' $ curl -XPUT http://localhost:10018/buckets/training/keys/baz \ -H 'Content-Type: text/plain' \ -d 'nothing to see here' $ curl -XPUT http://localhost:10018/buckets/training/keys/bam \ -H 'Content-Type: text/plain' \ -d 'caremad caremad caremad' Create some objects; http://docs.basho.com/riak/latest/dev/using/mapreduce/
  44. 141.

    > ReFun = fun(O, _, Re) -> case re:run(riak_object:get_value(O), Re,

    [global]) of {match, Matches} -> [{riak_object:key(O), length(Matches)}]; nomatch -> [{riak_object:key(O), 0}] end end. > {ok, Re} = re:compile("caremad"). > {ok, Riak} = riakc_pb_socket:start_link("127.0.0.1", 8087). > riakc_pb_socket:mapred_bucket(Riak, <<"training">>, [{map, {qfun, ReFun}, Re, true}]). Run Erlang MapReduce; http://docs.basho.com/riak/latest/dev/using/mapreduce/
  45. 145.

    $ curl -XPOST localhost:8098/types/mytype/buckets/users/keys/ john_smith \ -H 'x-riak-index-twitter_bin: jsmith123' \

    -H 'x-riak-index-email_bin: jsmith@basho.com' \ -H 'Content-Type: application/json' \ -d '{"userData":"data"}' Create values with secondary index tags; http://docs.basho.com/riak/latest/dev/using/2i/
  46. 148.
  47. 154.

    $ curl -XPUT http://localhost:10018/search/index/famous \ -H 'Content-Type: application/json' \ -d

    '{"schema":"_yz_default"}' Create default index using default schema; http://docs.basho.com/riak/latest/dev/using/search/
  48. 155.

    $ riak-admin bucket-type create animals '{"props":{}}' $ riak-admin bucket-type activate

    animals Create bucket type for search; http://docs.basho.com/riak/latest/dev/using/search/
  49. 156.

    $ curl -XPUT http://localhost:10018/types/animals/buckets/cats/props \ -H 'Content-Type: application/json' \ -d

    '{"props":{"search_index":"famous"}}' Associate bucket, bucket type, and index; http://docs.basho.com/riak/latest/dev/using/search/
  50. 157.

    $ curl -XPUT http://localhost:10018/types/animals/buckets/cats/keys/liono \ -H 'Content-Type: application/json' \ -d

    '{"name_s":"Lion-o", "age_i":30, "leader_b":true}' $ curl -XPUT http://localhost:10018/types/animals/buckets/cats/keys/cheetara \ -H 'Content-Type: application/json' \ -d '{"name_s":"Cheetara", "age_i":28, "leader_b":false}' $ curl -XPUT http://localhost:10018/types/animals/buckets/cats/keys/snarf \ -H 'Content-Type: application/json' \ -d '{"name_s":"Snarf", "age_i":43}' $ curl -XPUT http://localhost:10018/types/animals/buckets/cats/keys/panthro \ -H 'Content-Type: application/json' \ -d '{"name_s":"Panthro", "age_i":36}' Store some values; http://docs.basho.com/riak/latest/dev/using/search/
  51. 158.

    $ curl “http://localhost:10018/search/query/famous? wt=json&q=name_s:Lion*” | jsonpp $ curl “http://localhost:10018/search/query/famous?wt=json&q=age_i: %5B30%20TO%20*%5D”

    | jsonpp $ curl “http://localhost:10018/search/query/famous? wt=json&q=leader_b:true%20AND%20age_i:%5B25%20TO%20*%5D” | jsonpp Perform search queries; http://docs.basho.com/riak/latest/dev/using/search/
  52. 160.

    $ riak-admin bucket-type create strongly_consistent \ ‘{"props":{"consistent":true}}' $ riak-admin bucket-type

    status strongly_consistent $ riak-admin bucket-type activate strongly_consistent Enable strong consistency; http://docs.basho.com/riak/latest/dev/advanced/strong-consistency/
  53. 163.
  54. 164.

    $ riak-admin bucket-type create maps \ '{"props":{"datatype":"map"}}' $ riak-admin bucket-type

    create sets \ '{"props":{"datatype":"set"}}' $ riak-admin bucket-type create counters \ ‘{“props":{"datatype":"counter"}}' $ riak-admin bucket-type status maps $ riak-admin bucket-type activate maps Create bucket type for data types; http://docs.basho.com/riak/latest/dev/using/data-types/
  55. 165.

    $ curl -XPOST http://localhost:10018/types/counters/buckets/counters/ datatypes/traffic_tickets \ -H "Content-Type: application/json" \

    -d '{"increment": 1}’ $ curl http://localhost:10018/types/counters/buckets/counters/ datatypes/traffic_tickets Operate on counters; http://docs.basho.com/riak/latest/dev/using/data-types/
  56. 166.

    $ curl -XPOST http://localhost:10018/types/sets/buckets/travel/ datatypes/cities \ -H "Content-Type: application/json" \

    -d '{"add_all":["Toronto", “Montreal"]}' $ curl -XPOST http://localhost:10018/types/sets/buckets/travel/ datatypes/cities \ -H "Content-Type: application/json" \ -d '{"remove": “Montreal"}' $ curl http://localhost:10018/types/sets/buckets/travel/datatypes/ cities Operate on sets; http://docs.basho.com/riak/latest/dev/using/data-types/
  57. 167.

    $ curl -XPOST http://localhost:10018/types/maps/buckets/customers/ datatypes/ahmed_info \ -H "Content-Type: application/json" \

    -d ' { "update": { "first_name_register": "Ahmed", "phone_number_register": "5551234567" } }' $ curl -XPOST http://localhost:8098/types/maps/buckets/customers/ datatypes/ahmed_info \ -H "Content-Type: application/json" \ -d ' { "update": { "annika_info_map": { "update": { "interests_set": { "add": "tango dancing" } } } } } ' Operate on maps; http://docs.basho.com/riak/latest/dev/using/data-types/
  58. 169.