HyperDex: Next Generation NoSQL

Emin Gün Sirer, Robert Escriva & Bernard Wong @ CloudPhysics Inc, June 28 , 2013

Transcript

1. HyperDex: Next Generation NoSQL Emin Gün Sirer Robert Escriva Bernard Wong CloudPhysics June 28, 2013 Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 1 / 40
2. From RDBMS to NoSQL RDBMS have difficulty with scalability and performance NoSQL systems emerged to fill the gap Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 2 / 40
3. Problems Typical of NoSQL Lack of ... Search Consistency Fault-Tolerance Specifics vary between systems Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 3 / 40
4. Typical NoSQL Architecture K Consistent hashing maps each key to a server Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 4 / 40
5. The Search Problem Searching for objects without the key involves many servers Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 4 / 40
6. The Consistency Problem Clients may read inconsistent data and writes may be lost Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 5 / 40
7. The Fault-Tolerance Problem Many systems’ default settings consider a write complete after writing to just one node Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 6 / 40
8. HyperDex: An Overview Hyperspace hashing Value-dependent chaining ACID Transactions ⇓ High-Performance: High throughput with low variance Strong Consistency: Strong safety guarantees Fault Tolerance: Tolerates a threshold of failures Scalable: Adding resources increases performance Rich API: Support for complex datastructures and search Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 7 / 40
9. Introduction Design and Implementation Hyperspace Hashing Value-Dependent Chaining Evaluation Perspective Conclusion Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 8 / 40
10. Attributes map to dimensions in a multidimensional hyperspace First Name Phone Number Last Name Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
11. Attribute values are hashed independently Any hash function may be used First Name Phone Number Last Name H(“Neil”) H(“607-555-1024”) H(“Armstrong”) Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
12. Objects reside at the coordinate specified by the hashes First Name Phone Number Last Name H(“Neil”) H(“607-555-1024”) H(“Armstrong”) Neil Armstrong Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
13. First Name Phone Number Last Name Neil Armstrong Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
14. Different objects reside at different coordinates First Name Phone Number Last Name Neil Armstrong Lance Armstrong Neil Diamond Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
15. The hyperspace is divided into regions where each object resides in exactly one region First Name Phone Number Last Name Neil Armstrong Lance Armstrong Neil Diamond Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
16. Each server is responsible for a region of the hyperspace First Name Phone Number Last Name Neil Armstrong Lance Armstrong Neil Diamond Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
17. Each search intersects a subset of regions of the hyperspace First Name Phone Number Last Name Neil Armstrong Lance Armstrong Neil Diamond Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
18. All people named Neil are mapped to the yellow plane First Name Phone Number Last Name Neil Armstrong Lance Armstrong Neil Diamond Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
19. All people named Neil are mapped to the yellow plane First Name Phone Number Last Name Neil Armstrong Lance Armstrong Neil Diamond Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
20. All people named Armstrong are mapped to the gray plane First Name Phone Number Last Name Neil Armstrong Lance Armstrong Neil Diamond Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
21. All people named Armstrong are mapped to the gray plane First Name Phone Number Last Name Neil Armstrong Lance Armstrong Neil Diamond Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
22. A more restrictive search for Neil Armstrong contacts fewer servers First Name Phone Number Last Name Neil Armstrong Lance Armstrong Neil Diamond Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
23. Range searches are natively supported First Name Phone Number Last Name Neil Armstrong Lance Armstrong Neil Diamond Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 9 / 40
24. Space Partitioning In a naive implementation, the hyperspace would grow exponentially in the number of dimensions Space partitioning prevents exponential growth in the number of searchable attributes k a1 a2 a3 a4 a5 . . . aD-2 aD-1 aD Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 10 / 40
25. Space Partitioning In a naive implementation, the hyperspace would grow exponentially in the number of dimensions Space partitioning prevents exponential growth in the number of searchable attributes k a1 a2 a3 a4 a5 . . . aD-2 aD-1 aD k a1 a2 a3 a4 a5 . . . aD-2 aD-1 aD key subspace subspace 0 subspace 1 subspace S A search is performed in the most restrictive subspace Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 10 / 40
26. Space Partitioning In a naive implementation, a 9-dimensional space could require 512 machines HyperDex can store this space on just 24 machines using three subspaces Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 11 / 40
27. Hyperspace Hashing Implications searches are efficient Hyperspace hashing is a mapping, not an index No per-object updates to a shared datastructure No overhead for building and maintaining B-trees Functionality gained solely through careful placement Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 12 / 40
28. Introduction Design and Implementation Hyperspace Hashing Value-Dependent Chaining Evaluation Perspective Conclusion Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 13 / 40
29. Replication As an object changes, so too must the set of servers holding it Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 14 / 40
30. Value-Dependent Chaining k key subspace A B subspace 1 C D subspace 2 Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 15 / 40
31. Value-Dependent Chaining k key subspace A B subspace 1 C D subspace 2 put(k, A=1,B=1, C=1,D=1) 1 1 1 1 1 1 A put includes one node from each subspace Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 15 / 40
32. Value-Dependent Chaining k key subspace A B subspace 1 C D subspace 2 put(k, A=0,B=0, C=1,D=1) 2 1 2 2 2 1 2 2 When updating an object, the value-dependent chain includes the servers which hold the old and new versions of the object Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 15 / 40
33. Value-Dependent Chaining k key subspace A B subspace 1 C D subspace 2 put(k, A=0,B=0, C=1,D=1) 2 2 2 2 2 2 Each put removes all state from the previous put Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 15 / 40
34. Value-Dependent Chaining k key subspace A B subspace 1 C D subspace 2 put(k, A=0,B=1, C=1,D=1) 3 2 3 3 3 2 3 3 Subsequent operations involve solely the most recent nodes Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 15 / 40
35. Value-Dependent Chaining k key subspace A B subspace 1 C D subspace 2 Servers are replicated in each region to provide fault tolerance Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 16 / 40
36. Value-Dependent Chaining k key subspace A B subspace 1 C D subspace 2 put(k, A=0,B=0, C=1,D=1) 2 1 2 2 2 2 1 1 2 2 2 2 The value-dependent chain includes all replicas Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 16 / 40
37. Value-Dependent Chaining k key subspace A B subspace 1 C D subspace 2 put(k, A=0,B=0, C=1,D=1) 2 1 2 2 2 2 1 1 2 2 2 2 Failed nodes are removed from the chain Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 16 / 40
38. Value-Dependent Chaining Implications No extra mechanism is necessary to provide Atomicity Ordering Replication Relocation Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 17 / 40
39. Multikey Transactions Hyperspace hashing enables HD to locate data quickly Value-dependent chaining enables HD to replicate data And this is sufficient for many applications But some apps require atomic, consistent updates to multiple items Options are: Spray and pray Use a heavyweight algorithm (e.g. Paxos) for ordering HyperDex Warp Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 18 / 40
40. Warp Properties Warp is a novel, optimistic, concurrent, distributed algorithm for ensuring isolated updates to a data store. Atomic – multiple operations on multiple keys are indivisable Consistent – application invariants are preserved Isolated – one copy serializability Durability – all transactions are propagated to f+1 replicas Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 19 / 40
41. Consistency Key Operations and Transactions: One copy serializability Search Consistency: All search operations observe all put operations that completed prior to the search Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 20 / 40
42. Implementation Fully implemented system with 52,000 LOC Bindings for C, C++, Python, Java, Ruby, Node.JS Open sourced under a BSD-like license Active user community with many contributors Implementation tricks: Hyperspace hashing maps objects to locations on disk Paxos-based RSM maintains the hyperspace mapping Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 21 / 40
43. Inserting/Retrieving an Object >>> c.put(’phonebook’, ’jsmith’, ... {’first’: ’John’, ’last’: ’Smith’, ... ’phone’: 6075551024}) True >>> c.get(’phonebook’, ’jsmith’) {’first’: ’John’, ’last’: ’Smith’, ’phone’: 6075551024} Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 22 / 40
44. Performing A Search >>> [x for x in c.search(’phonebook’, ... {’first’: ’John’})] [{’first’: ’John’, ’last’: ’Smith’, ’phone’: 6075551024, ’username’: ’jsmith’}] >>> [x for x in c.search(’phonebook’, ... {’phone’: (6070000000, 6080000000)})] [{’first’: ’John’, ’last’: ’Smith’, ’phone’: 6075551024, ’username’: ’jsmith’}] Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 23 / 40
45. Atomic Operations >>> c.condput(’phonebook’, ’jsmith’, ... {’phone’: 6075551024}, ... {’phone’: 6075552048}) True >>> c.condput(’phonebook’, ’jsmith’, ... {’phone’: 6075551024}, ... {’phone’: 6075552048}) False Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 24 / 40
46. Atomic Operations >>> c.atomic_add(’phonebook’, ’jsmith’, ... {’phone’: 1}) True >>> c.get(’phonebook’, ’jsmith’) {’first’: ’John’, ’last’: ’Smith’, ’phone’: 6075552049} Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 25 / 40
47. Asynchronous Operations >>> d = c.async_put(’phonebook’, ’jsmith’, ... {’first’: ’John’, ’last’: ’Smith’, ... ’phone’: 6075551024}) >>> d <hyperclient.DeferredInsert object at 0x7f2252d >>> do_work() >>> d.wait() True Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 26 / 40
48. A Space with Datastructures $ hyperdex-coordinator-control \ --host 127.0.0.1 \ --port 6970 \ add-space << EOF space socialnetwork dimensions username, first, last, pending (list(string)), hobbies (set(string)), unread_messages (map(string,string)) upvotes (map(string,int64)) key username auto 3 3 subspace first, last auto 3 3 # 8 regions, 3 replicas EOF Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 27 / 40
49. Manipulating Lists >>> c.list_rpush(’socialnetwork’, ’jsmith’, ... {’pending’: ’bjones1’}) True >>> c.get(’socialnetwork’, ’jsmith’)[’pending’] [’bjones1’] Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 28 / 40
50. Transactions >>> t = c.begin_transaction() >>> amount1 = t.get(’account’, ’egs’)[’balance’ >>> amount2 = t.get(’account’, ’rob’)[’balance’ >>> amount1 -= 1000 >>> amount2 += 1000 >>> t.put(’account’, ’egs’, {’balance’: amount1 >>> t.put(’account’, ’rob’, {’balance’: amount2 >>> t.commit() Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 29 / 40
51. Introduction Design and Implementation Evaluation Perspective Conclusion Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 30 / 40
52. Experimental Setup Use the Yahoo! Cloud Serving Benchmark Each system makes two replicas of the data MongoDB: Writes to the client’s outgoing socket buffer Cassandra: Writes to one storage node’s filesystem HyperDex: Writes to both replicas in three subspaces Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 31 / 40
53. YCSB Throughput 0 5 10 15 20 25 30 35 40 Cassandra MongoDB HyperDex Throughput (thousand op/s) Workload A Workload B Workload C Workload D Workload F Workload E Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 32 / 40
54. 95% get / 5% put Latency 0 20 40 60 80 100 1 10 100 1000 CDF (%) Latency (ms) YCSB Workload B Cassandra (R) Cassandra (U) MongoDB (R) MongoDB (U) HyperDex (R) HyperDex (U) Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 33 / 40
55. 100% put Latency 0 20 40 60 80 100 1 10 100 1000 CDF (%) Latency (ms) YCSB Load Dataset Cassandra MongoDB HyperDex Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 34 / 40
56. search Latency 0 20 40 60 80 100 1 10 100 1000 CDF (%) Latency (ms) YCSB Workload E Cassandra MongoDB HyperDex Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 35 / 40
57. Chain Length vs. Put Latency 0 1 2 3 4 5 6 7 2 4 6 8 10 12 14 16 18 20 22 Latency (ms) Chain Length (nodes) Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 36 / 40
58. Scalability 0 1 2 3 4 4 8 12 16 20 24 28 32 Throughput (million ops/s) Nodes Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 37 / 40
59. Performance Summary Outperforms other systems by 2–4× for get/put While offering stronger consistency and fault tolerance Outperforms other systems by 12–13× for search Despite operating solely on secondary attributes Latency for chain-operations is predictable Scales as resources are added Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 38 / 40
60. The CAP Theorem What CAP is simplified to: You must always give something up What the CAP theorem really says: If you cannot limit the number of faults and requests can be directed to any server and you insist on serving every request then you cannot possibly be consistent Most NoSQL systems are proud to preemptively give up desirable properties like consistency in the name of CAP — even in the case of no failures HyperDex allows for f failures without sacrificing consistency or availability Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 39 / 40
61. Conclusion HyperDex is a next generation NoSQL system Novel Techniques Hyperspace Hashing Value-Dependent Chaining ACID Transactions The next-generation of NoSQL systems should explore alternative designs that offer both an expanded API and strong guarantees http://hyperdex.org/ Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQL http://hyperdex.org/ 40 / 40
62. Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 1 / 6
63. YCSB Benchmark Workloads Name Workload Key Choice Application A 50% R Zipf Session Store 50% U B 95% R Zipf Photo Tagging 5% U C 100% R Zipf Profile Cache D 95% R Temporal Status Updates 5% I E 95% S Zipf Threads 5% I F 50% R Zipf User Database 50% R-M-U R = Read, U = Update, I = Insert, S = Scan/Search Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 2 / 6
64. Hash Functions and Load Balancing Out of the box, HyperDex supports hashing strings and integers What about non-uniform inputs? Select a better hash function Use forwarding pointers Create multiple dimensions in the hyperspace for a single attribute The default hash functions work well for workloads that we’ve seen in practice Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 3 / 6
65. Experimental Setup Lab Cluster 14 Machines Intel Xeon 2.5 GHz E5420 × 2 16 GB RAM 500 GB SATA HDD Debian 6.0 Linux 2.6.32 VICCI Cluster 70 Machines Intel Xeon 2.66 GHz X5650 × 2 48 GB RAM 1 TB SATA HDD × 3 Virtualized Fedora 12 Linux 2.6.32 Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 4 / 6
66. Cluster Size Netflix: App-specific clusters of 6-48 Cassandra instances Google BigTable: 66% of clusters < 20 tablet servers 84% of clusters < 100 tablet servers 96% of clusters < 500 tablet servers Justin Sheehy, Basho Inc.: Typical cluster is 6-12 Riak nodes Largest clusters < 100 Riak nodes Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 5 / 6
67. Related Work Multi-dimensional database systems on a single host Grid File, KD-Tree, Multi-dimensional BST, Quad-Tree, R-Tree, Universal B-Tree Distributed database systems maintain distributed indices Distributed B-Tree, P-Tree, Sinfonia Peer-to-peer systems are only eventually consistent Arpeggio, CAN, Chord, Consistent Hashing, Mercury, MURK, Pastry, SkipIndex, SWAM-V, Tapestry Space-filling curves suffer from the curse of dimensionality MAAN, SCRAP, Squid, ZNet NoSQL systems/key-value stores give up search, consistency or fault-tolerance CouchDB, MongoDB, Neo4j, PNUTS, Redis, TXCache, BigTable, Cassandra, COPS, Distributed Data Structures, Dynamo, Fawn KV, HBase, HyperTable, LazyBase, Masstree, Memcached, RAMCloud, Riak, SILT, Spanner, Spinnaker, TSSL, Voldemort Emin Gün Sirer Robert Escriva, Bernard Wong HyperDex: Next Generation NoSQLhttp://hyperdex.org/ 6 / 6