Riak Techniques for Advanced Web and Mobile Application Development

[RICON East 2013] In recent years, there have been tremendous advances in high-performance, high-availability data storage for scalable web and mobile application development. Oftentimes, these NoSQL solutions are portrayed as sacrificing the crispness and rapid application development features of relational database alternatives. In this presentation, we show the amazing things that are possible using a variety of techniques to apply Riak's advanced features such as map-reduce, search, and secondary indexes. We review each feature in the context of a demanding real-world Ruby & Javascript "Pinterest clone" application with advanced features such as real-time updates via Websocket, comment feeds, content quarantining, permissions, search and social graph modeling. We pay specific attention to explaining the 'why' of these Riak techniques for high-performance, high availability applications, not just the 'how'.

Transcript

1. Riak for Advanced Web and Mobile Applications Sunny Gleason May 13, 2013 1
2. whoami •distributed systems engineer sunnycloud - boston, ma •previous work @ •github: sunnygleason twitter: @sunnygleason speakerdeck: sunnygleason hello@thesunnycloud.com 2
3. what’s in store •get distributed •sample app •modeling in riak 3
4. ack’s •Sean Cribbs : ripple, riak-{ruby-client, dt} Russell Brown: riak-dt •Kyle Kingsbury : risky library •Coda Hale : modeling in riak •Bret Taylor : friendfeed arch 2009 •Amazon Dynamo team 2007 •Basho 4
5. (most important slide) •symmetric distributed systems •2-query model 5
6. web & mobile apps •Strong durability •High Availability •Low-latency •Dynamic Scaling •Evolvable (pivot pivot pivot) 6
7. relational database •Strong durability •High Availability •Low-latency •Dynamically Scalable •Evolvable (pivot pivot pivot) 7
8. relational database Sources: http://www.breeze-cg.com/uploads/rac.png http://www.cloudadminzone.com/wp-content/images/master-slave.jpg 8
9. mongodb •Strong durability •High Availability •Low-latency •Dynamically Scalable •Evolvable (pivot pivot pivot) 9
10. mongodb Source: http://sett.ociweb.com/sett/settAug2011.html 10
11. hbase •Strong durability •High Availability •Low-latency •Dynamically Scalable •Evolvable (pivot pivot pivot) 11
12. hbase Source: http://blog.sematext.com/2012/07/09/intro-to-hbase-internals-and-schema-desig/ 12
13. riak •Strong durability •High Availability •Low-latency •Dynamically Scalable •Evolvable (pivot pivot pivot) 13
14. riak Source: http://docs.basho.com/riak/latest/tutorials/fast-track/What-is-Riak/ 14
15. restpinte “pinterest clone” 15
16. login 16
17. user model class User attr_accessor :user_id attr_accessor :email def lc_email email.downcase.strip end end 17
18. credential model class Credential attr_accessor :credential_id attr_accessor :token has_one :user end 18
19. board 19
20. board model class Board attr_accessor :board_id attr_accessor :title has_one :creator, :class => User has_many :pins end 20
21. pin 21
22. pin model class Pin attr_accessor :pin_id attr_accessor :caption has_one :board has_one :original, :class => Pin has_many :comments end 22
23. discovery 23
24. modeling with riak 24
25. riak kv •bucket : key namespace •key : key identifier •value : blob or structured GET /riak/users/LijPmIDZ {"email":"foo@bar.com"} 25
26. links •link : metadata pointer •tag : link type •target : key reference GET /riak/boards/B5SMouGg > Link: </riak/users/LijPmIDZ>; riaktag="creator" 26
27. traversing links •bucket : bucket to include •tag : link type to collect •keep : (0 or 1) return linked values •“_” is wildcard GET /buckets/boards/keys/J9MlC2EN/ users,creator,1 27
28. 2i : secondary indexes •index : index name + type •type : string or int •key : reference GET /riak/users/LijPmIDZ > x-riak-index-email_bin: foo@bar.com 28
29. 2i querying •bucket : implicit in result •range queries as well •full list of keys returned unsorted GET /buckets/users/index/ email_bin/foo%40bar.com {"keys":["LijPmIDZ"]} 29
30. search •bucket : values to search •full-text tokenized •“+” and “-” allowed GET /solr/users/select?q=email%3A+foo %40bar.com&wt=json {... "docs":[{"id":"LijPmIDZ","index":"users", "fields":{"_type":"User", "email":"foo@bar.com"}}]}} 30
31. map-reduce •POST to /mapred endpoint •input : optional input values •map : function to generate intermediate values •reduce : function to process intermediate values 31
32. map-reduce querying POST /mapred > {"inputs":[["pins","MDJcfTpf"], > ["pins","HI8wHAja"]], > "query":[{"map":{"language":"erlang", > "module":"riak_kv_mapreduce", > "function":"map_object_value", > "keep":true},}]} 32
33. modeling with ripple 33
34. key-value class User include Ripple::Document property :user_id, String property :email, String end 34
35. embedded types class Board include Ripple::Document ... many :pins end class Pin include Ripple::EmbeddedDocument ... embedded_in Board end 35
36. links class Board include Ripple::Document ... many :pins, :using => :linked end class Pin include Ripple::Document ... end 36
37. stored_key class Board include Ripple::Document ... many :pins, :using => :stored_key end class Pin include Ripple::Document ... end 37
38. 2i class User include Ripple::Document property :user_id, String property :email, String, :index => true end 38
39. search •enable at bucket level •available via Riak client Ripple.client.search User.bucket.name, "email:foo@bar.com" 39
40. adding features •activity feeds •likes / popularity counts •content quarantine •social graph •real-time 40
41. activity feeds •secondary index + sort •search (sort by key) •see @coda’s http://basho.com/riak- and-scala-at-yammer/ 41
42. likes / popularity •embedded id sets •secondary index + mapred •or, see @seancribbs riak_dt http://vimeo.com/ 52414903 42
43. content quarantining •secondary index & mapred •search & boolean query {"status_id":["Q_LijPmIDZ"]} {"status_id":["A_J9MlC2EN"]} 43
44. social graph •link walking •secondary index •stored_keys 44
45. real-time •live activity & edits •chat & presence •using PubNub (akamai for real-time) 45
46. live activity (ruby) pubnub.publish( :channel =>"boards:J9MlC2EN", :message => { :type => :pin_added, :target_id => "pins:LijPmIDZ" } ) 46
47. live activity (js) PUBNUB.subscribe({ "channel":"boards:J9MlC2EN", "message": function(m) { alert(m); } }) 47
48. presence PUBNUB.subscribe({ "channel":"boards:J9MlC2EN", "presence": function(m) { alert(m); } }) PUBNUB.here_now({ "channel":"boards:J9MlC2EN", "callback": function(m) { alert(m); } }) 48
49. redis acceleration •primitives for sets & sorted sets •fast set operations (membership, union, intersection) •HA via replication, Zookeeper & master-slave 49
50. gotchas •non-transactional updates •sibling resolution •no unique indexes •metadata vs. data •aggregation: counters & sort by •multi-column indexes •map-reduce code distribution •index migrations 50
51. upshot •modeling is usually tricky •primitives are there, and evolving •hybrid approaches can help 51
52. coming soon •github : restpinte •pubnub angular.js service •pubnub riak integration 52
53. going further •tutorial: querying riak •github.com/basho/{ripple, riak-client} •github.com/aphyr/risky •github.com/basho/riak_dt •redis.io •github.com/pubnub/javascript 53
54. thank you! questions? 54