Polyglot Persistence with Ruby

Modern web architectures now need more than a single data store to perform complex tasks. Polyglot Persistence, a term coined by Martin Fowler, revolves around the application of multiple types of data stores employed simultaneously. This talk is about good practices w.r.t. Polyglot Persistence in Ruby. Contents of this talk can be referred to as guidelines and are completely based on experiences by the Engineering team at Jombay.com.

Transcript

1. Polyglot Persistence with Ruby engineering@jombay.com
2. Consumer Internet in 2014 http://thenextweb.com/facebook/2014/03/08/a-deep-dive-into-facebooks-news-feed-redesign-shows-nothing-but-net/
3. Consumer Internet in 2014 Mashup of multiple use-cases
4. Consumer Internet in 2014 Signup / Login
5. Consumer Internet in 2014 Signup / Login Messages
6. Consumer Internet in 2014 Signup / Login Messages Email
7. Consumer Internet in 2014 Signup / Login Messages Email Chat
8. Consumer Internet in 2014 Signup / Login Messages Email Chat Photo Sharing
9. Consumer Internet in 2014 Signup / Login Messages Email Chat Photo Sharing Discussions
10. Consumer Internet in 2014 Signup / Login Messages Email Chat Photo Sharing Discussions Ad-platform
11. Consumer Internet in 2014 Signup / Login Messages Email Chat Photo Sharing Discussions Ad-platform Recommendations
12. Consumer Internet in 2014 Signup / Login Messages Email Chat Photo Sharing Discussions Ad-platform Recommendations Geo-tagging
13. Consumer Internet in 2014 Signup / Login Messages Email Chat Photo Sharing Discussions Ad-platform Recommendations Geo-tagging Developer API
14. Consumer Internet in 2014 Each use-case demands special attention
15. Persistence in 2014 Data stores have come a long way
16. Persistence in 2014 1. Relational 2. Key-value 3. Document 4. Column Family 5. Graph
17. Persistence in 2014 1. Relational 2. Key-value 3. Document 4. Column Family 5. Graph Still counting
18. Persistence in 2014 Each data-store specializes in specific areas
19. Polyglot Persistence http://martinfowler.com/bliki/PolyglotPersistence.html
20. USE A HYBRID DATA ARCHITECTURE, YOU SAY? TELL ME MORE ABOUT DATA CONSISTENCY http://onlyhdwallpapers.com/food-and-drink/meme-charlie-and-the-chocolate-factory-willy-wonka-desktop- hd-wallpaper-1114758/
21. Ruby Bindings for Data Access Mongoid Neography Neo4j.rb couchdb-ruby rexster-ruby cassandra-ruby hbase-ruby tire sunspot
22. Ask your best friend for more Ruby bindings! http://ihumanable.com/blog/wp-content/uploads/2010/03/lmgtfy.png
23. Persistence with Ruby 1. ActiveRecord 2. Active Relation 3. Active Model
24. Polyglot Persistence with Ruby 1. ActiveRecord 2. Active Relation 3. Active Model
25. Polyglot Persistence with Ruby 1. ActiveRecord 2. Active Relation 3. Active Model On Steroids!
26. Identify your primary data store Which piece of data can save my ass? During a catastrophe, what data can help me reconcile everything else? (E.g.: Amazon US East went down due to a lightning storm!) Am I finicky about transactions and ACID shit? Do I care if data isn’t persisted once in a while?
27. Use an ActiveRecord adapter 1. Well supported 2. Can be seamlessly used in popular Ruby frameworks like Rails, Sinatra, Padrino, etc 3. Gives high-level validation checks, database migrations, callbacks and readability for free
28. Or roll your own 1. AREL is awesome! 2. Custom data persistence logic can be hooked at multiple locations in ActiveRecord a. Schema migration b. CRUD operations c. Complex querying d. Anything else
29. Replication and Consistency KEEP CALM AND USE A CALLBACK
30. Use callbacks for all writes 1. Create 2. Update 3. Destroy
31. Replication and Consistency KEEP CALM AND before_save :ensure_mongo_compatible after_save :save_document after_create :update_graph_relationships after_destroy :delete_couch_copy
32. Callbacks can make writes slow 1. Callbacks run in the same request cycle 2. Transaction rollbacks can lead to delayed writes 3. Defer callbacks when possible. delayed_job, resque, sidekiq are awesome!
33. When should I defer replication writes? 1. Consistency compromise on NoSQL data stores is allowed. E.g. It’s OK if a News Feed item doesn’t show up once in a while. 2. Transaction rollback system across deferred writes is in place. E.g. User was created on MySQL but graph relations were not created successfully on Neo4j. Use a message queue to flag bad transactions. Clean/report/fix them periodically. 3. Replication is cheap and quick. Do not forget to quantify ‘quick’.
34. Cleaner approach with acts_as_ 1. Abstract common callback logic within modules 2. Organize modules in a Ruby Gem 3. Name it acts_as_blahable. E.g. acts_as_searchable 4. Include or extend it on ActiveRecord::Base 5. Inject dependency via configuration
35. Cleaner approach with acts_as_ class Job < ActiveRecord::Base [...] acts_as_searchable({ :indices => { :name => {}, :bucket_type => { :filter => true }, :description => {}, :locations => { :properties => { :term => [:id, :node_id], :text => [:name] }, :master => { :class => Location, :conditions => { :state_name => 'Maharashtra' } } }) [...] end #class Job
36. What lies within def acts_as_searchable(options={}) # Tire module should be mixed in after InstanceMethods to prevent # tire from replacing the update_index instance method include Searchable::Model::InstanceMethods include Tire::Model::Search options.reverse_merge!({ :indices => {}, :filters => {}, :text_fields => {} }) cattr_accessor(:searchable_options) { options } indices = options[:indices] # Validate if the fields can be used to run search filters ensure_filterable_fields(indices) # Add callbacks on the class define_master_callbacks(indices) define_destroy_handler end
37. ActiveRecord adapters are cool too! However... 1. Active Relation was meant for Relational Data Stores (of course!) 2. Tightly coupled with SQL 3. Anything non-SQL should be added and served à la carte. 4. Maintaining tree structured querying might get tricky. (Chaining of queries such as User.where.select is done by maintaining a tree data structure in AREL) 5. Need to enforce a schema, even on schemaless data stores
38. But you get to use config/database. yml :) development: adapter: mysql2 database: myfunkyapp_development username: myfunkyusername password: myfunkypassword port: 3306 host: localhost graph_development: adapter: neo4j_rest host: localhost port: 7474 root: 0
39. Summary 1. Polyglot Persistence involves using more than one types of data stores suited for specific tasks 2. Staying closer to ActiveRecord helps 3. Use ActiveRecord callbacks to maintain replication and consistency 4. Defer callbacks to optimize write performance 5. acts_as_blahable 6. ActiveRecord Adapters for NoSQL data stores