Riak Techniques for Advanced Web and Mobile Application Development

Slide 1

Slide 1 text

Riak for Advanced Web and Mobile Applications Sunny Gleason May 13, 2013 1

Slide 2

Slide 2 text

whoami •distributed systems engineer sunnycloud - boston, ma •previous work @ •github: sunnygleason twitter: @sunnygleason speakerdeck: sunnygleason [email protected] 2

Slide 3

Slide 3 text

what’s in store •get distributed •sample app •modeling in riak 3

Slide 4

Slide 4 text

ack’s •Sean Cribbs : ripple, riak-{ruby-client, dt} Russell Brown: riak-dt •Kyle Kingsbury : risky library •Coda Hale : modeling in riak •Bret Taylor : friendfeed arch 2009 •Amazon Dynamo team 2007 •Basho 4

Slide 5

Slide 5 text

(most important slide) •symmetric distributed systems •2-query model 5

Slide 6

Slide 6 text

web & mobile apps •Strong durability •High Availability •Low-latency •Dynamic Scaling •Evolvable (pivot pivot pivot) 6

Slide 7

Slide 7 text

relational database •Strong durability •High Availability •Low-latency •Dynamically Scalable •Evolvable (pivot pivot pivot) 7

Slide 8

Slide 8 text

relational database Sources: http://www.breeze-cg.com/uploads/rac.png http://www.cloudadminzone.com/wp-content/images/master-slave.jpg 8

Slide 9

Slide 9 text

mongodb •Strong durability •High Availability •Low-latency •Dynamically Scalable •Evolvable (pivot pivot pivot) 9

Slide 10

Slide 10 text

mongodb Source: http://sett.ociweb.com/sett/settAug2011.html 10

Slide 11

Slide 11 text

hbase •Strong durability •High Availability •Low-latency •Dynamically Scalable •Evolvable (pivot pivot pivot) 11

Slide 12

Slide 12 text

hbase Source: http://blog.sematext.com/2012/07/09/intro-to-hbase-internals-and-schema-desig/ 12

Slide 13

Slide 13 text

riak •Strong durability •High Availability •Low-latency •Dynamically Scalable •Evolvable (pivot pivot pivot) 13

Slide 14

Slide 14 text

riak Source: http://docs.basho.com/riak/latest/tutorials/fast-track/What-is-Riak/ 14

Slide 15

Slide 15 text

restpinte “pinterest clone” 15

Slide 16

Slide 16 text

Slide 17

Slide 17 text

user model class User attr_accessor :user_id attr_accessor :email def lc_email email.downcase.strip end end 17

Slide 18

Slide 18 text

credential model class Credential attr_accessor :credential_id attr_accessor :token has_one :user end 18

Slide 19

Slide 19 text

board 19

Slide 20

Slide 20 text

board model class Board attr_accessor :board_id attr_accessor :title has_one :creator, :class => User has_many :pins end 20

Slide 21

Slide 21 text

pin 21

Slide 22

Slide 22 text

pin model class Pin attr_accessor :pin_id attr_accessor :caption has_one :board has_one :original, :class => Pin has_many :comments end 22

Slide 23

Slide 23 text

discovery 23

Slide 24

Slide 24 text

modeling with riak 24

Slide 25

Slide 25 text

riak kv •bucket : key namespace •key : key identifier •value : blob or structured GET /riak/users/LijPmIDZ {"email":"[email protected]"} 25

Slide 26

Slide 26 text

links •link : metadata pointer •tag : link type •target : key reference GET /riak/boards/B5SMouGg > Link: ; riaktag="creator" 26

Slide 27

Slide 27 text

traversing links •bucket : bucket to include •tag : link type to collect •keep : (0 or 1) return linked values •“_” is wildcard GET /buckets/boards/keys/J9MlC2EN/ users,creator,1 27

Slide 28

Slide 28 text

2i : secondary indexes •index : index name + type •type : string or int •key : reference GET /riak/users/LijPmIDZ > x-riak-index-email_bin: [email protected] 28

Slide 29

Slide 29 text

2i querying •bucket : implicit in result •range queries as well •full list of keys returned unsorted GET /buckets/users/index/ email_bin/foo%40bar.com {"keys":["LijPmIDZ"]} 29

Slide 30

Slide 30 text

search •bucket : values to search •full-text tokenized •“+” and “-” allowed GET /solr/users/select?q=email%3A+foo %40bar.com&wt=json {... "docs":[{"id":"LijPmIDZ","index":"users", "ﬁelds":{"_type":"User", "email":"[email protected]"}}]}} 30

Slide 31

Slide 31 text

map-reduce •POST to /mapred endpoint •input : optional input values •map : function to generate intermediate values •reduce : function to process intermediate values 31

Slide 32

Slide 32 text

map-reduce querying POST /mapred > {"inputs":[["pins","MDJcfTpf"], > ["pins","HI8wHAja"]], > "query":[{"map":{"language":"erlang", > "module":"riak_kv_mapreduce", > "function":"map_object_value", > "keep":true},}]} 32

Slide 33

Slide 33 text

modeling with ripple 33

Slide 34

Slide 34 text

key-value class User include Ripple::Document property :user_id, String property :email, String end 34

Slide 35

Slide 35 text

embedded types class Board include Ripple::Document ... many :pins end class Pin include Ripple::EmbeddedDocument ... embedded_in Board end 35

Slide 36

Slide 36 text

links class Board include Ripple::Document ... many :pins, :using => :linked end class Pin include Ripple::Document ... end 36

Slide 37

Slide 37 text

stored_key class Board include Ripple::Document ... many :pins, :using => :stored_key end class Pin include Ripple::Document ... end 37

Slide 38

Slide 38 text

2i class User include Ripple::Document property :user_id, String property :email, String, :index => true end 38

Slide 39

Slide 39 text

search •enable at bucket level •available via Riak client Ripple.client.search User.bucket.name, "email:[email protected]" 39

Slide 40

Slide 40 text

adding features •activity feeds •likes / popularity counts •content quarantine •social graph •real-time 40

Slide 41

Slide 41 text

activity feeds •secondary index + sort •search (sort by key) •see @coda’s http://basho.com/riak- and-scala-at-yammer/ 41

Slide 42

Slide 42 text

likes / popularity •embedded id sets •secondary index + mapred •or, see @seancribbs riak_dt http://vimeo.com/ 52414903 42

Slide 43

Slide 43 text

content quarantining •secondary index & mapred •search & boolean query {"status_id":["Q_LijPmIDZ"]} {"status_id":["A_J9MlC2EN"]} 43

Slide 44

Slide 44 text

social graph •link walking •secondary index •stored_keys 44

Slide 45

Slide 45 text

real-time •live activity & edits •chat & presence •using PubNub (akamai for real-time) 45

Slide 46

Slide 46 text

live activity (ruby) pubnub.publish( :channel =>"boards:J9MlC2EN", :message => { :type => :pin_added, :target_id => "pins:LijPmIDZ" } ) 46

Slide 47

Slide 47 text

live activity (js) PUBNUB.subscribe({ "channel":"boards:J9MlC2EN", "message": function(m) { alert(m); } }) 47

Slide 48

Slide 48 text

presence PUBNUB.subscribe({ "channel":"boards:J9MlC2EN", "presence": function(m) { alert(m); } }) PUBNUB.here_now({ "channel":"boards:J9MlC2EN", "callback": function(m) { alert(m); } }) 48

Slide 49

Slide 49 text

redis acceleration •primitives for sets & sorted sets •fast set operations (membership, union, intersection) •HA via replication, Zookeeper & master-slave 49

Slide 50

Slide 50 text

gotchas •non-transactional updates •sibling resolution •no unique indexes •metadata vs. data •aggregation: counters & sort by •multi-column indexes •map-reduce code distribution •index migrations 50

Slide 51

Slide 51 text

upshot •modeling is usually tricky •primitives are there, and evolving •hybrid approaches can help 51

Slide 52

Slide 52 text

coming soon •github : restpinte •pubnub angular.js service •pubnub riak integration 52

Slide 53

Slide 53 text

going further •tutorial: querying riak •github.com/basho/{ripple, riak-client} •github.com/aphyr/risky •github.com/basho/riak_dt •redis.io •github.com/pubnub/javascript 53

Slide 54

Slide 54 text

thank you! questions? 54