Operating & Tuning Riak

Tom Santero @tsantero

Hector Castro @hectcastro

Let’s build a Database!

Desired Properties High Availability Low Latency Scalable Fault Tolerance Ops-Friendly Predictable

No content

DYNAMO

1. Data Model 2. Persistence Mechanism

{“key”: “value”} 1. Data Model 2. Persistence Mechanism

{“key”: “value”} 1. Data Model 2. Persistence Mechanism

1. Data Model 2. Persistence Mechanism key value key value key value key value key value key value key value key value key value

1. Data Model 2. Persistence Mechanism key value key value key value key value key value key value key value key value key value Bitcask LevelDB

3. Distribution Logic 4. Replication A ----> B ----> C ----> D ---->

3. Distribution Logic 4. Replication A ----> B ----> C ----> D ----> index = hash(key) % count(servers)

3. Distribution Logic 4. Replication A ----> B ----> C ----> D ----> index = hash(key) % count(servers) =>3

3. Distribution Logic 4. Replication A ----> B ----> C ----> D ----> index = hash(key) % count(servers) E ---->

3. Distribution Logic 4. Replication A ----> B ----> C ----> D ----> index = hash(key) % count(servers) E ----> =>4

3. Distribution Logic 4. Replication A ----> B ----> C ----> D ----> index = hash(key) % count(servers) E ----> remap keys with every topology change???

AIN’T NOBODY GOT TIME FOR THAT!

3. Distribution Logic 4. Replication A ----> B ----> C ----> D ----> E ----> SOLUTION: consistent hashing!

hash ring

tokenize it

node 0 node 1 node 2

node 0 node 1 node 2 hash(key)

node 0 node 1 node 2 node 3 + hash(key)

3. Distribution Logic 4. Replication node 0 node 1 node 2 node 3 Replicas are stored N - 1 contiguous partitions

3. Distribution Logic 4. Replication node 0 node 1 node 2 node 3 hash(“cities/atlanta”) Replicas are stored N - 1 contiguous partitions

5. CRUD 6. Global State http://basho.com/understanding-riaks-con!gurable-behaviors-part-1/ Quorum requests N R W PR/PW DW (just open a socket)

5. CRUD 6. Global State gossip protocol /var/lib/riak/data/ring

7. Fault Tolerance 8. Con!ict Resolution node 0 node 1 node 2 node 3 Replicas are stored N - 1 contiguous partitions node 2

7. Fault Tolerance 8. Con!ict Resolution node 0 node 1 node 2 node 3 Replicas are stored N - 1 contiguous partitions node 2 o"ine

7. Fault Tolerance 8. Con!ict Resolution node 0 node 1 node 2 node 3 Replicas are stored N - 1 contiguous partitions node 2 o"ine put(“cities/atlanta”)

7. Fault Tolerance 8. Con!ict Resolution node 0 node 1 node 2 node 3 Replicas are stored N - 1 contiguous partitions node 2 o"ine put(“cities/atlanta”)

7. Fault Tolerance 8. Con!ict Resolution node 0 node 1 node 2 node 3 Replicas are stored N - 1 contiguous partitions node 2 o"ine put(“cities/atlanta”) FALLBACK “SECONDARY”

7. Fault Tolerance 8. Con!ict Resolution node 0 node 1 node 2 node 3 Replicas are stored N - 1 contiguous partitions node 2 o"ine FALLBACK “SECONDARY”

7. Fault Tolerance 8. Con!ict Resolution node 0 node 1 node 2 node 3 Replicas are stored N - 1 contiguous partitions node 2 o"ine FALLBACK “SECONDARY” node 2

7. Fault Tolerance 8. Con!ict Resolution node 0 node 1 node 2 node 3 Replicas are stored N - 1 contiguous partitions node 2 o"ine node 2 HINTED HANDOFF

7. Fault Tolerance 8. Con!ict Resolution Vector Clocks establish temporality - gives us “happened before” - easy to reason about - provide a way for resolving con!icting writes

7. Fault Tolerance 8. Con!ict Resolution Vector Clocks establish temporality - gives us “happened before” - easy to reason about - provide a way for resolving con!icting writes

9. Anti-Entropy 10. Map/Reduce merkle tree to track changes coordinated at the vnode level run as a background process exchange with neighbor vnodes for inconsistencies resolution semantics: trigger read-repair

9. Anti-Entropy 10. Map/Reduce = hashes marked “dirty”

9. Anti-Entropy 10. Map/Reduce

9. Anti-Entropy 10. Map/Reduce

9. Anti-Entropy 10. Map/Reduce

9. Anti-Entropy 10. Map/Reduce

9. Anti-Entropy 10. Rich Queries = keys to read-repair

9. Anti-Entropy 10. Rich Queries 2i Riak Search Map/Reduce

9. Anti-Entropy 10. Rich Queries 2i Yokozuna Riak Search Map/Reduce

What did we build?

A masterless, distributed key/value store with peer-to-peer replication, object versioning logic, rich query capabilities that’s resilient to failures and self-healing.

Problem?

is excessive friendliness unfriendly?

What we talk about when we talk about love Availability

Latency Node Liveness Network Partitions

Riak should have a !at latency curve. 95th & 99th are irregular, your developers are ****ing you.

packets

FAILURE

How many hosts do you need to survive “F” failures?

F + 1 2F + 1 3F + 1 fundamental minimum a majority are alive Byzantine Fault Tolerance

“...widespread underestimation of the speci!c di"culties of size seems one of the major underlying causes of the current software failure.” --EW Dijkstra Notes on Structured Programming 1969

Common Mistakes

1. No Monitoring 2. List All The Keys! 3. hadoooooops! 4. reallllly BIG objects 5. 1 cluster, several AZs 6. No Firewall 7. No Capacity Planning 8. Wait too long to scale

1. No Monitoring 2. List All The Keys! 3. hadoooooops! 4. reallllly BIG objects 5. 1 cluster, several AZs 6. No Firewall 7. No Capacity Planning 8. Wait too long to scale Metric CPU Memory Disk Space Disk IO Network File Descriptors Swap Threshold 75% * num_cores 70% - bu"ers 75% 80% sustained 70% sustained 75% of ulimit > 0KB

1. No Monitoring 2. List All The Keys! 3. hadoooooops! 4. reallllly BIG objects 5. 1 cluster, several AZs 6. No Firewall 7. No Capacity Planning 8. Wait too long to scale http://:/stats Riak Counters graph them with:

1. No Monitoring 2. List All The Keys! 3. hadoooooops! 4. reallllly BIG objects 5. 1 cluster, several AZs 6. No Firewall 7. No Capacity Planning 8. Wait too long to scale http://:/stats Riak Counters graph them with: or if you dont want to run your own monitoring service, there’s a aaS for that...

1. No Monitoring 2. List All The Keys! 3. hadoooooops! 4. reallllly BIG objects 5. 1 cluster, several AZs 6. No Firewall 7. No Capacity Planning 8. Wait too long to scale http://:/stats Riak Counters graph them with: or if you dont want to run your own monitoring service, there’s a aaS for that... 1        import  json 2        from  urllib2  import  urlopen 3        import  socket 4        from  time  import  sleep 5   6        UDP_ADDRESS  =  "carbon.hostedgraphite.com" 7        UDP_PORT  =  2003 8        RIAK_STATS_URL='http://localhost:11098/stats' 9   10      HG_API_KEY='Your  Api  Key  from  HostedGraphite.com' 11   12      stats=json.load(urlopen(RIAK_STATS_URL)) 13   14      nn  =  stats['nodename'].replace('.',  '-­‐') 15      sock  =  socket.socket(socket.AF_INET,  socket.SOCK_DGRAM)  #  UDP#  Internet 16 17      for  k  in  stats: 18          if  type(1)  ==  type(stats[k]): 19              message='%s.%s.%s  %s'  %  (HG_API_KEY,nn,k,stats[k]) 20              sock.sendto(message,  (UDP_ADDRESS,  UDP_PORT)) 21              #sleep(0.1) 22              print  message 23      print  'Sent  %s'  %  len(stats) 24

1. No Monitoring 2. List All The Keys! 3. hadoooooops! 4. reallllly BIG objects 5. 1 cluster, several AZs 6. No Firewall 7. No Capacity Planning 8. Wait too long to scale

1. No Monitoring 2. List All The Keys! 3. hadoooooops! 4. reallllly BIG objects 5. 1 cluster, several AZs 6. No Firewall 7. No Capacity Planning 8. Wait too long to scale Run Queues Process Process Process Process Process Process OS + kernel CPU Core 1 . . . . . . CPU Core N Erlang VM N 1 SMP Schedulers (one per core)

1. No Monitoring 2. List All The Keys! 3. hadoooooops! 4. reallllly BIG objects 5. 1 cluster, several AZs 6. No Firewall 7. No Capacity Planning 8. Wait too long to scale Riak != Hadoop

1. No Monitoring 2. List All The Keys! 3. hadoooooops! 4. reallllly BIG objects 5. 1 cluster, several AZs 6. No Firewall 7. No Capacity Planning 8. Wait too long to scale busy_dist_port got you down?

1. No Monitoring 2. List All The Keys! 3. hadoooooops! 4. reallllly BIG objects 5. 1 cluster, several AZs 6. No Firewall 7. No Capacity Planning 8. Wait too long to scale

1. No Monitoring 2. List All The Keys! 3. hadoooooops! 4. reallllly BIG objects 5. 1 cluster, several AZs 6. No Firewall 7. No Capacity Planning 8. Wait too long to scale http://aphyr.com/posts/224-do-not-expose-riak-to-the-internet

1. No Monitoring 2. List All The Keys! 3. hadoooooops! 4. reallllly BIG objects 5. 1 cluster, several AZs 6. No Firewall 7. No Capacity Planning 8. Wait too long to scale https://github.com/basho/basho_bench Basho Bench

1. No Monitoring 2. List All The Keys! 3. hadoooooops! 4. reallllly BIG objects 5. 1 cluster, several AZs 6. No Firewall 7. No Capacity Planning 8. Wait too long to scale (actual footage of a cluster under load attempting hando")

No content

Tuning Demo