Supersize your Key-Value Store

Presented at ConFoo 2014

We've all heard in recent years about how Key-Value stores cast off the scaling problems of SQL-based solutions and give developers the flexibility to choose in-memory or disk-persistent, single-node or clustered options.

In this talk, we review design and performance of several key-value stores (Riak, LevelDB, and MySQL API), and several techniques such as efficient compression and schema extraction to get the most out of any KV store.

Transcript

1. Supersize Your Key-Value Store Sunny Gleason ConFoo February 28, 2014
2. whoami •distributed systems engineer
 sunnycloud - boston, ma •previous work @ Amazon, Ning •github: sunnygleason
 twitter: @sunnygleason
 speakerdeck: sunnygleason •don’t be a stranger!
3. what's this all about? • NoSQL is getting a lot of love right now • NoSQL core ideas: simplification and
 doing more with less • These techniques apply to any system • You can create future-proof APIs and still enjoy the benefits of your favorite data store
4. agenda • What is a Key-Value store? • 3 Sample KV Implementations • Some techniques for getting the most out of your KV store
5. key-value stores • kv is a model, not an implementation • goal: get improved scalability & performance characteristics by restricting the persistence model • GET(key), PUT(key), DELETE(key)
6. kv implementations
7. other KV implementations
8. leveldb • Open-Source (BSD License) Embedded Key-Value Store • Created by Jeff Dean and Sanjay Ghemawat at Google • Written in C/C++ • Original intent: embedded data store in Chromium browser
9. leveldb design source: http://dailyjs.com/2013/04/19/leveldb-and-node-1/
10. leveldb api • byte[] DB.get(byte[] key)
 void DB.put(byte[] key, byte[] value)
 void DB.delete(byte[] key)
 DBIterator DB.iterator() • DBIterator.seek(byte[] key)
 DBIterator.peekNext()
 DBIterator.prev()
 DBIterator.seekToFirst() / seekToLast() • Check out:
 https://code.google.com/p/leveldb/ (c/c++)
 https://github.com/dain/leveldb (pure java)
 https://github.com/fusesource/leveldbjni (java/jni)

11. leveldb performance source: http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html
12. leveldb thoughts • leveldb is a great general-purpose embedded KV store • backup / restore model is not great - close file and copy, or iterate over snapshot and write to new file • as with any embedded KV store, beware large data set sizes • OSS license and adoption are nice relative to alternatives like BDB, BDB-JE, Tokyo Cabinet • Promising newcomer: RocksDB (from Facebook)
13. • Open-Source (Apache 2) distributed data store • Based on the Amazon Dynamo model
 (as presented SOSP 2007) • Created by Basho Technologies as their primary product • Written in Erlang, with some JavaScript capabilities (MapReduce) • Pluggable data stores: most commonly eleveldb, also bitcask
14. design source: http://www.cs.rutgers.edu/~pxk/417/notes/images/dht-dynamo-vnode.png
15. api • Keyspaces are called “buckets”
 Bucket myBucket = client.fetchBucket("test").execute(); • Create:
 int val1 = 1;
 myBucket.store("one", val1).execute(); • Get:
 int fetched1 = 
 myBucket.fetch("one",Integer.class).execute();

16. • Update:
 StringIntMap fetched3 = myBucket.fetch("three",
 StringIntMap.class).execute();
 
 fetched3.put("myValue", 42);
 myBucket.store("three", fetched3).execute(); • Delete:
 myBucket.delete("one").execute(); api
17. riak performance
18. riak gotchas • Lower throughput compared to 
 embedded data store • Time is no longer a single-node, authoritative concept • Sibling resolution: concurrent updates happen, "last write wins" almost always loses - need to provide a Strategy • Backup/restore is challenging - not just one leveldb, several nodes with leveldbs • Erlang: can be tough to get under the hood
19. riak advantages • Battle-tested, kernel-level
 understanding of leveldb • Commercial distribution with support • Wide-area replication with very efficient synchronization (merkle trees) • Riak CS for large files / blobs • Erlang: a lot of power & reliability once you 
 get under the hood
20. mysql KV schema • Not a separate product: just use
 MySQL with a restricted schema • MySQL itself is Open Source (GPL2), 
 you will likely never need to modify it • Owned/maintained by Oracle • All the commercial support you will pay for • Other variants by Percona (recommended), MariaDB
21. mysql KV schema create table if not exists `_key_types` (! `_key_type` smallint unsigned not null,! `_type_name` varchar(100) not null,! PRIMARY KEY(`_key_type`),! UNIQUE KEY(`_type_name`)! ) ENGINE=InnoDB ROW_FORMAT=DYNAMIC! ! create table if not exists `_sequences` (! `_key_type` smallint unsigned not null,! `_next_id` bigint unsigned not null,! PRIMARY KEY(`_key_type`)! ) ENGINE=InnoDB ROW_FORMAT=DYNAMIC
22. mysql KV schema create table if not exists `_key_values` (! `_key_type` smallint unsigned not null,! `_key_id` bigint unsigned not null,! `_created_dt` int unsigned not null,! `_updated_dt` int unsigned not null,! `_version` bigint unsigned not null,! `_is_deleted` char(1) not null default 'N',! `_format` char(1) not null default 'S',! `_compression` char(1) not null default 'F',! `_value` blob not null,! PRIMARY KEY(`_key_type`, `_key_id`),! INDEX(`_updated_dt`)! )
 ENGINE=InnoDB ROW_FORMAT=DYNAMIC
23. mysql KV schema design • Key is 2-parts: type_id + sequence_id • Value is a blob (alternatively could be binary) • Separate sequence/id generation from DB implementation • Use int/long as identifier mapping for types 
 to reduce size
24. mysql KV api • get(key, value):
 SELECT … WHERE key_type = ? and key_id = ? • insert(key, value):
 INSERT INTO … VALUES (?, ?, ?) • update(key, value):
 UPDATE … SET value = ? • delete(key):
 DELETE FROM … WHERE key_type = ? and key_id = ?
 UPDATE … SET is_deleted = ‘Y’ WHERE …

25. mysql performance (RW, not KV) source: http://www.mysql.com/why-mysql/benchmarks/
26. mysql KV schema gotchas • Roll your own data store • Hey, this isn't NoSQL?! • Requires DB tuning and operational knowledge • Not distributed or fully symmetric like Riak:
 requires vertical scaling or sharding
27. mysql KV schema advantages • Utilize existing widely deployed
 technology: easy sell to DBA’s and
 manager types • Transactional guarantees provided by MySQL • Improve scaling & durability through replication (read slaves) • Strong durability using InnoDB, easy hot backup (full and incremental) using tools like Percona Xtrabackup • Works with other SQL databases as well: H2, Sqlite3, MSSQL, Postgres, Oracle, …
28. taking KV to the next level • Turn your “big data” into smaller data • Compression: apply compression algorithms to byte[] values • Binary Encoding: smaller binary representations of JSON (for example) • Schema Extraction: use limited, flexible schema information to reduce data storage requirements

29. compression • Instead of TEXT/LONGTEXT and huge JSON or XML values • Use BLOB/LONGBLOB, and compress the value prior to storage: • GZip • Snappy • LZF
30. compression performance source: http://ning.github.io/jvm-compressor-benchmark/results/canterbury/roundtrip-2013-06-08/index.html
31. compression notes • GZip, Bzip2 typically have great compression (70%+), (but) very high CPU utilization • Snappy, LZF have good compression (30-50%) and lower CPU utilization • However, all of these algorithms typically fall short for small values (< 1024 bytes)
32. huffman encoding • For small documents, consider using a Hufman Coding library for compression* • Analyze a representative set of data beforehand to create a statistical data model • Can yield 30-50% compression or more for small documents (2-1024 bytes) • *Also consider InnoDB page compression (however, its use of zlib puts CPU burden on the database)
33. binary encoding
34. schema extraction • Idea: Extract a subset of key-value pairs into a schema, represent as Array vs. Map • Additionally, use more efficient types to represent values 
 (such as boolean -> bit, enum -> int) • Benefits: explicit schema versioning, decouple schema evolution, more compact representation • Downside: approaching relational DB complexity without benefits of querying (yet) & table-level consistency
35. schema extraction • Instead of:
 {“id”:10,”first_name”:”William”,
 ”last_name”:”Gates”,”yob”:1955, ...}
 {“id”:11,”first_name”:”Steve”,
 ”last_name”:”Jobs”,”yob”:1955, …} • Schema:
 [{“id”:”int”},{“first_name”:”string”},
 {“last_name”:”string”},{“yob”:”int”}, …] • Values:
 [10,”William”,”Gates”,1955, ...]
 [11,”Steve”,”Jobs”,1955, ...]
36. what about sparse objects? • Include a value that encodes presence or absence of attribute values • Use a bitmap int at beginning of array • Sparse Values:
 [‘1010,10,”Gates”]
 [‘1101,11,”Steve”,1955]
37. what about schema evolution? • Include an int schema version in value array • Remember previous versions of schema to decode values • [{“id”:”int”,”v”:1},{“first_name”:”string”,”v”:1},
 {“last_name”:”string”,”v”:1},{“yob”:”int”,”v”:2}] • Versioned Values:
 [1,‘101,10,”Gates”]
 [2,‘1101,11,”Steve”,1955]
38. what about extension attributes? • Include a map of undeclared attributes at the end of the value array • Merge with declared attributes • Extended Values:
 [1,‘101,10,”Gates”,{“home_town”:”Redmond}] [2,‘1101,11,”Steve”,{“company”:”Apple”}]
39. techniques recap • Improve the scalability of your KV store by applying techniques like compression, binary encoding and schema extraction • These techniques can also have follow-on benefits in the caching and application layers as well, at the expense of tighter coupling • Please view these techniques more like a spectrum or continuum, and less like a shopping list
40. kazuki • Open-Source (Apache 2) Data Store written in Java • Collection of persistence patterns:
 KV Store, KV Cache, Full-text Index, Range-Based Indexes, Counters, Journal Store • Run as embedded library or REST service • Portable Implementations:
 KV Store: MySQL, H2DB, Sqlite3
 KV Store Soon: MSSQL, LevelDB, Riak, Cassandra
 KV Cache: Memcached, Redis
 Full-text Index: ElasticSearch, Solr
 Journal Store: KV Stores, plus Java Chronicle
41. kazuki features • now: persistence plus schema extraction, encoding (smile), compression (lzf, huffman soon) • coming up next: schema evolution, backup/ restore, plugins, more data store portability • ideas / participation welcome!
 github.com/kazukidb/kazuki
42. questions? thank you! source: http://static1.businessinsider.com/image/50801faeeab8ea7c42000000-960/superman.jpg