Getting started with Riak in the Cloud

Transcript

1. Getting started with Riak in the Cloud

2. I work with Code! I work with Data! Michael Brodhead

   Ines Sombra @RandomMood @mojinations
3. None

4. Agenda Application layer Database layer

5. A tale of 2 apps • One brand new (no

   code written) • One in production running MySQL
6. None
7. None
8. None

9. Multiple Regions

10. Multiple Clouds

11. 11

12. The Libraries riak-client • simple Ruby access to Riak's HTTP

    & protocol buffer APIs

13. The Libraries Ripple • Higher level, ORM interface • ActiveModel

    based buffer APIs

14. Simplicity

15. None

16. The Libraries dm-riak • Already using DataMapper • Hoping for

    trivial switchover

17. The Libraries Riak Shim • database.yml • Bucket naming •

    Hides JSON • 2i

18. Challenges • Unit tests • Data migration • Deployment

19. Testing

20. Testing

21. Deployment

22. Use your app logic

23. How do we get here?

24. ... but we have no hardware “We need a Riak

    cluster” (@mojinations circa 3 months ago)

25. Let’s do it!

26. Why Riak Product? • Linearly Scalable • Fault-tolerant • Ops

    friendly • Product lacks HA options

27. Riak @ Engine Yard • We’ll use it first •

    No custom Chef • Provisioning, configuration, monitoring, & backups

28. Your Choices • Number of nodes • Instance type •

    Data location • Backend type You are trying to create a Riak cluster. Do you need any help?
29. None

30. AWS Ops The Riak on AWS slide

31. Instances & I/O • Riak’s I/O patterns • Small blobs

    from many places on disk • EBS best at bulk reads & writes

32. Instance types • m1.large & m1.xlarge • EBS-optimized • Consider

    PIOPS • High I/O Quadruple XL (hi1.4xlarge) - SSDs! NEW! NEW! NEW!

33. More instances to compensate for the performance variability of virtualized

    resources Prepare Yourself

34. Cluster Sizes • 5 nodes or more Best performance &

    growth • Scales linearly • Don’t be stingy

35. Ring Size • Change default value! • Cannot be reset

    • Ring size: 256 8/64 vnodes per node • Don’t go above 512

36. Provisioning App haproxy MyRiakCluster: node-0 MyRiakCluster: node-1 MyRiakCluster: node-2 MyRiakCluster:

    node-3 MyRiakCluster: node-4 Riak Cluster HTTP or Protobuff App haproxy App haproxy

37. Disable Swap • Riak process pages should not hit swap

    • Swap usage can result in unresponsive server • Let the kernel kill it

38. Mount & Scheduler • mount with noatime flag • Use

    deadline scheduler on EBS cat /sys/block/xvdj1/queue/scheduler noop [deadline] cfq echo deadline > /sys/block/xvdj1/ queue/scheduler

39. Linux Filesystems • xfs, ext3 • ext4 • barrier =

    0 • data = writeback • ZFS not recommended

40. Backend choices • Bitcask* • help mitigate EBS • LevelDB

    • Memory Can be changed and optimized!

41. Kernel Optimizations • /etc/sysctl.conf vm.swappiness = 0 net.core.somaxconn= 4000 net.ipv4.tcp_timestamps

    = 0 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_max_syn_backlog = 40000 net.ipv4.tcp_sack = 1 net.ipv4.tcp_fin_timeout = 15 net.ipv4.tcp_keepalive_intvl = 30 net.ipv4.tcp_tw_reuse = 1 http://wiki.basho.com/Linux-Performance-Tuning.html

42. Monitoring • stats or riak-admin status • Collectd & Nagios

    • Riaknostic • Riak Control (maybe soon)

43. Backups • Backup both the ring and data directories •

    Integrate with EY tools • scp or rsync data directory • EBS snapshots

44. Scaling Riak • Vertical • Horizontal • reduces node load

    • batch is better riak-admin cluster replace <old> <new>

45. Benchmarks

46. Tests • 5-node cluster / 5 hours • 50 concurrent

    connections • gets 10 /puts 5 /deletes 1 • Bitcask backend • Protobuffs protocol

47. Medium - PIOPS 500

48. Medium - PIOPS 1K

49. Medium - Ephemeral FAIL!

50. XL - EBS

51. XL -PIOPS 500

52. 4XL - EBS

53. 4XL - PIOPS 1k

54. What’s next?

55. None