Getting started with Riak in the Cloud

RICON 2012 talk

Riak’s highly available nature makes it ideal for cloud environments where any of your resources may disappear without notice but your database must still be up. Unfortunately databases need IO and the cloud, where everything is virtualized, is not the most performant place to run one. Getting started with Riak in the cloud brings a different set of assumptions than hardware based installations and we’ll go through them together.

Come hear the tale of 2 engineers who have no hardware but decided to try Riak for a production-critical application. We’ll tell you how we ported our application from MySQL, rolled it to production, and the lessons we learned while running Riak in the Cloud.

Transcript

1. Getting started with Riak in the Cloud
2. I work with Code! I work with Data! Michael Brodhead Ines Sombra @RandomMood @mojinations
4. Agenda Application layer Database layer
5. A tale of 2 apps • One brand new (no code written) • One in production running MySQL
9. Multiple Regions
10. Multiple Clouds
12. The Libraries riak-client • simple Ruby access to Riak's HTTP & protocol buffer APIs
13. The Libraries Ripple • Higher level, ORM interface • ActiveModel based buffer APIs
14. Simplicity
16. The Libraries dm-riak • Already using DataMapper • Hoping for trivial switchover
17. The Libraries Riak Shim • database.yml • Bucket naming • Hides JSON • 2i
18. Challenges • Unit tests • Data migration • Deployment
19. Testing
20. Testing
21. Deployment
22. Use your app logic
23. How do we get here?
24. ... but we have no hardware “We need a Riak cluster” (@mojinations circa 3 months ago)
25. Let’s do it!
26. Why Riak Product? • Linearly Scalable • Fault-tolerant • Ops friendly • Product lacks HA options
27. Riak @ Engine Yard • We’ll use it first • No custom Chef • Provisioning, configuration, monitoring, & backups
28. Your Choices • Number of nodes • Instance type • Data location • Backend type You are trying to create a Riak cluster. Do you need any help?
30. AWS Ops The Riak on AWS slide
31. Instances & I/O • Riak’s I/O patterns • Small blobs from many places on disk • EBS best at bulk reads & writes
32. Instance types • m1.large & m1.xlarge • EBS-optimized • Consider PIOPS • High I/O Quadruple XL (hi1.4xlarge) - SSDs! NEW! NEW! NEW!
33. More instances to compensate for the performance variability of virtualized resources Prepare Yourself
34. Cluster Sizes • 5 nodes or more Best performance & growth • Scales linearly • Don’t be stingy
35. Ring Size • Change default value! • Cannot be reset • Ring size: 256 8/64 vnodes per node • Don’t go above 512
36. Provisioning App haproxy MyRiakCluster: node-0 MyRiakCluster: node-1 MyRiakCluster: node-2 MyRiakCluster: node-3 MyRiakCluster: node-4 Riak Cluster HTTP or Protobuff App haproxy App haproxy
37. Disable Swap • Riak process pages should not hit swap • Swap usage can result in unresponsive server • Let the kernel kill it
38. Mount & Scheduler • mount with noatime flag • Use deadline scheduler on EBS cat /sys/block/xvdj1/queue/scheduler noop [deadline] cfq echo deadline > /sys/block/xvdj1/ queue/scheduler
39. Linux Filesystems • xfs, ext3 • ext4 • barrier = 0 • data = writeback • ZFS not recommended
40. Backend choices • Bitcask* • help mitigate EBS • LevelDB • Memory Can be changed and optimized!
41. Kernel Optimizations • /etc/sysctl.conf vm.swappiness = 0 net.core.somaxconn= 4000 net.ipv4.tcp_timestamps = 0 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_max_syn_backlog = 40000 net.ipv4.tcp_sack = 1 net.ipv4.tcp_fin_timeout = 15 net.ipv4.tcp_keepalive_intvl = 30 net.ipv4.tcp_tw_reuse = 1 http://wiki.basho.com/Linux-Performance-Tuning.html
42. Monitoring • stats or riak-admin status • Collectd & Nagios • Riaknostic • Riak Control (maybe soon)
43. Backups • Backup both the ring and data directories • Integrate with EY tools • scp or rsync data directory • EBS snapshots
44. Scaling Riak • Vertical • Horizontal • reduces node load • batch is better riak-admin cluster replace <old> <new>
45. Benchmarks
46. Tests • 5-node cluster / 5 hours • 50 concurrent connections • gets 10 /puts 5 /deletes 1 • Bitcask backend • Protobuffs protocol
47. Medium - PIOPS 500
48. Medium - PIOPS 1K
49. Medium - Ephemeral FAIL!
50. XL - EBS
51. XL -PIOPS 500
52. 4XL - EBS
53. 4XL - PIOPS 1k
54. What’s next?