Riak In Production - Baltimore Riak Meetup

Recently we were tasked with building the core API and user service for a high-traffic mobile product. As part of the project, we were given two main priorities to focus on; that we could never have downtime, and that we couldn't lose data. With those constraints, we
turned toward Riak, which provided us with a highly available, fault tolerant, easily scalable solution.

Through node failures, upgrades, and migrations, Riak held its own. When we needed features from a traditional relational database, we were still able to use Riak for
up-front write guarantees, and synched that data over to Postgres, never losing data. All in all, we're very happy with our choice to use Riak, and expect it to become a regular choice in our tool box.

Transcript

1. Riak in Production Brian Bickerton Web Engineer bbickerton@omniti.com @tau_zero
2. Who is OmniTI? “We provide a full service, from sustainable architectures that can support hundreds of millions of users, to web application development … and user interface design.” ~ http://omniti.com/is
3. What the Client Wants, the Client Gets • The product is a mobile second screen TV application. Users check in to shows, participate in live events, interact socially. • Focusing here on the API. Entry point for all external communication and user authentication. • Failure of the API is not an option. The data store needs to keep up.
4. All I need is… • Persistent data storage • Fault Tolerant • Easy to Manage & Scale • Multiple Index Lookup • Solid support (docs, irc, etc.) • Real-time relational queries http://thereelist.com/media/4ff4beb86ce85e4c5d00052e/
5. Chosen Tech (The Erlang Sandwich) API and sync script ( orig: 0.6.x, curr: 0.8.21 ) Primary write, authoritative ( orig: 1.0.0, curr: 1.3.1 ) Queue for syncing ( orig: 2.7.1, curr: 2.7.1) Secondary write ( orig: 9.1, curr: 9.2)
6. Cluster Maintenance • Setup • Rolling Upgrades • Backups • Monitoring • Migration http://theinfosphere.org/File:Scruffy_promo.jpg
7. Monitored and Graphed with Circonus
8. Migraine-free Migration 1. Set up network connection between new and old cluster locations 2. Spin up new nodes 3. Join new nodes to existing cluster 4. Monitor transfers and ownership handoffs. 5. Update load balancer (remove/add nodes) 6. Remove old nodes from cluster. 7. Monitor transfers. Have a drink
9. Problems? • File descriptors and ulimit • Waiting for riak_kv • Frozen stats • Stalled ownership handoff https://encyclopediadramatica.se/File:Trollface_More_HD.png
10. An Object By Any Other Name • Bitcask backend has only one key per value, meaning multiple indices need to be simulated • Pre-commit hooks create objects in “index buckets” based on content, linking to original • Linkwalk from index to original • Alternative: LevelDB and 2i
11. Sharing is Caring: Postgres • No extra client work • Never lose track of data to be synced • Never sync out-dated data • Keep sync lag low • Flexible http://taurusmarketing.com.au/sharing-is-caring/unknown-5/
12. Baby’s First pgsync 1. Save Riak object 2. Post-commit hook creates object in “pgsync” bucket with link to original 3. Node.js sweeper periodically keylists pgsync bucket (every few seconds) 4. Sweeper linkwalks to original object 5. Sweeper saves data to PostgreSQL 6. Sweeper cleans up on success
13. pgsync v2: You gotta keep 'em separated • Growing data + keylisting in main backend = bad times • You’ll pry keylisting from my cold, dead hands • Use multi-backend to explicitly separate pgsync bucket and add fail buckets. • Ordered keys, prefixed with timestamps were a good fit for LevelDB to be this second backend. Bitcask still primary.
14. pgsync v3: A Proper Queue • Finally admit defeat on keylisting • Time for a real queue: RabbitMQ • Create pgsync and fail queues • Still use pgsync bucket for safety, though no longer need fail buckets
15. pgsync v3 in Detail 1. Save Riak object 2. Post-commit hook creates object in “pgsync” bucket with link to original 3. Post-commit hook publishes to RabbitMQ 4. Node.js sweeper subscribes to RabbitMQ 5. Sweeper linkwalks to original object 6. Sweeper saves data to PostgreSQL 7. Sweeper cleans up on success or… 8. Sweeper publishes message on fail
16. API pgsync http://www.nyan.cat/cats/original.gif
17. API pgsync http://www.nyan.cat/cats/original.gif
18. API pgsync http://www.nyan.cat/cats/original.gif
19. API pgsync http://www.nyan.cat/cats/original.gif
20. API pgsync http://www.nyan.cat/cats/original.gif
21. API pgsync http://www.nyan.cat/cats/original.gif
22. Oh God, everything’s on fire
23. Oh God, everything’s on fire • Riak • Single nodes: replace in cluster • Full cluster: restore from backup • pgsync • Riak fails: reprocess Rabbit fail queues • Rabbit fails, Still an open question
24. Additional Resources • Riak Documentation • http://docs.basho.com/riak/latest/ • Riak-Rabbit Hook • https://github.com/jbrisbin/riak-rabbitmq- commit-hooks • Ownership Handoff Script • https://github.com/basho/riak/blob/1.0.2-release/ RELEASE-NOTES.org#ownership-handoff-stall
25. Any Questions? Brian Bickerton bbickerton@omniti.com @tau_zero OmniTI omniti.com @omniti