Slide 1
Slide 1 text
Throw Some Keys on It
Data Modeling for Key/Value Data Stores by Example
Slide 2
Slide 2 text
Hector Castro
@hectcastro
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
Relational databases are
not all bad. They actually
give us a lot.
Slide 5
Slide 5 text
Relationships
Transactions
Schemas
Ability to extend preexisting
structure easily
Ad-hoc queries (with SQL)
Slide 6
Slide 6 text
Relationships
Transactions
Schemas
Ability to extend preexisting
structure easily
Ad-hoc queries (with SQL)
Slide 7
Slide 7 text
Relationships
Transactions
Schemas
Ability to extend preexisting
structure easily
Ad-hoc queries (with SQL)
Slide 8
Slide 8 text
Relationships
Transactions
Schemas
Ability to extend preexisting
structure easily
Ad-hoc queries (with SQL)
Slide 9
Slide 9 text
Relationships
Transactions
Schemas
Ability to extend preexisting
structure easily
Ad-hoc queries (with SQL)
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
What if your application
doesn’t need most of that?
Slide 12
Slide 12 text
What if the latency, scale,
or high-availability
requirements you’re facing
make most of those
benefits disappear?
Slide 13
Slide 13 text
Enter key/value data stores.
Slide 14
Slide 14 text
Schema-less
Single-access reads
Great at write-heavy
workloads
Easier to scale
Familiar interface
Slide 15
Slide 15 text
Schema-less
Single-access reads
Great at write-heavy
workloads
Easier to scale
Familiar interface
Slide 16
Slide 16 text
Schema-less
Single-access reads
Great at write-heavy
workloads
Easier to scale
Familiar interface
Slide 17
Slide 17 text
Schema-less
Single-access reads
Great at write-heavy
workloads
Easier to scale
Familiar interface
Slide 18
Slide 18 text
Schema-less
Single-access reads
Great at write-heavy
workloads
Easier to scale
Familiar interface
Slide 19
Slide 19 text
mapping = { }
mapping["p_cool_databas"] = "riak"
p mapping["p_cool_databas"]
# => "riak"
Slide 20
Slide 20 text
mapping["p_cool_databas"] = ["riak", "postgres"].to_json
p mapping["p_cool_databas"]
# => "[\"riak\",\"postgres\"]"
Slide 21
Slide 21 text
mapping["p_cool_databas_logo"] = File.read("riak_logo.png")
p mapping["p_cool_databas_logo"]
# => "\x89PNG\r\n\u001A\n\u0000\u0000\u0000\rIHDR\u0000\...
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
Cool story, Hector, but my
real-world application is
too complex for a key/value
data store.
Slide 24
Slide 24 text
No content
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
No content
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
No content
Slide 30
Slide 30 text
No content
Slide 31
Slide 31 text
No content
Slide 32
Slide 32 text
No content
Slide 33
Slide 33 text
buckets/20140221133500/keys/1_-‐2
Slide 34
Slide 34 text
buckets/20140221133500/keys
20140221
Slide 35
Slide 35 text
buckets/20140221133500/keys
133500
Slide 36
Slide 36 text
buckets/20140221133500/keys
1_-‐2
Slide 37
Slide 37 text
[
"car1",
"car22",
"car7"
]
Slide 38
Slide 38 text
def emit_car_location(car_id, color, lat, lon)
current_timestamp = Time.now.utc
bucket = $client.bucket(current_timestamp.strftime($timestamp_format))
!
puts "%s[%s]: %s is at [ %s, %s ]" % [
color,
current_timestamp.strftime($timestamp_format),
car_id,
lat,
lon
]
!
cars = GSet.new
cars.add(car_id)
!
object = bucket.new("%s_%s" % [ lat, lon ])
object.content_type = "application/json"
object.data = cars.to_json
!
object.store(returnbody: false)
end
Slide 39
Slide 39 text
def
current_timestamp
bucket
!
puts
color,
current_timestamp
car_id,
lat,
lon
]
!
cars
cars
!
object
object
object
!
object
end
emit_car_location(car_id, color, lat, lon)
Slide 40
Slide 40 text
def
current_timestamp
bucket
!
puts
color,
current_timestamp
car_id,
lat,
lon
]
!
cars
cars
!
object
object
object
!
object
end
!
!
bucket = $client.bucket(current_timestamp.strftime($timestamp_format))
Slide 41
Slide 41 text
buckets/20140221133500/keys/
20140221133500
Slide 42
Slide 42 text
def
current_timestamp
bucket
!
puts
color,
current_timestamp
car_id,
lat,
lon
]
!
cars
cars
!
object
object
object
!
object
end
!
!
!
!
puts "%s[%s]: %s is at [ %s, %s ]" % [
color,
current_timestamp.strftime($timestamp_format),
car_id,
lat,
lon
]
Slide 43
Slide 43 text
def
current_timestamp
bucket
!
puts
color,
current_timestamp
car_id,
lat,
lon
]
!
cars
cars
!
object
object
object
!
object
end
!
!
!
!
!
!
!
!
!
!
!
!
cars = GSet.new
cars.add(car_id)
Slide 44
Slide 44 text
G[row-only]Set
Slide 45
Slide 45 text
Set union is commutative
and convergent; therefore it
is always safe to have
simultaneous writes to a set
which only allows addition.
Slide 46
Slide 46 text
Conflict-free Replicated
Data Type
Slide 47
Slide 47 text
1. An alternative to locking
2. A useful abstraction (Set, Counter, Map)
3. A way to resolve automatically toward a
single value
Provides:
Slide 48
Slide 48 text
def
current_timestamp
bucket
!
puts
color,
current_timestamp
car_id,
lat,
lon
]
!
cars
cars
!
object
object
object
!
object
end
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
object = bucket.new("%s_%s" % [ lat, lon ])
object.content_type = "application/json"
object.data = cars.to_json
Slide 49
Slide 49 text
{
"type": "GSet",
"a": ["car1"]
}
Slide 50
Slide 50 text
No content
Slide 51
Slide 51 text
No content
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
No content
Slide 54
Slide 54 text
def request_car(request_lat, request_lon)
local_grid = closest_blocks(request_lat, request_lon)
!
puts "%s\n%% Car requested in [ %s, %s ]!\n\n" %
ANSI.yellow,
request_lat,
request_lon
]
!
local_grid.keys.sort.each do |distance|
lat, lon = local_grid[distance].split("_")
closest_car = get_cars_at_location(lat, lon)
!
if closest_car.members.length > 0
puts "\n%% Cars closest to you:\n\n"
puts JSON.pretty_generate(closest_car.members.to_a)
puts
!
break
end
end
end
Slide 55
Slide 55 text
def
local_grid
!
puts
request_lat,
request_lon
]
!
local_grid
lat, lon
closest_car
!
!
end
end
request_car(request_lat, request_lon)
Slide 56
Slide 56 text
def
local_grid
!
puts
request_lat,
request_lon
]
!
local_grid
lat, lon
closest_car
!
!
end
end
!
local_grid = closest_blocks(request_lat, request_lon)
Slide 57
Slide 57 text
def
local_grid
!
puts
request_lat,
request_lon
]
!
local_grid
lat, lon
closest_car
!
!
end
end
!
!
!
puts "%s\n%% Car requested in [ %s, %s ]!\n\n" %
ANSI.yellow,
request_lat,
request_lon
]
Slide 58
Slide 58 text
def
local_grid
!
puts
request_lat,
request_lon
]
!
local_grid
lat, lon
closest_car
!
!
end
end
!
!
!
!
!
!
!
!
!
local_grid.keys.sort.each do |distance|
Slide 59
Slide 59 text
{
1.2424 => "2_1",
2.7335 => "2_2",
5.8733 => "5_-‐5"
}
Slide 60
Slide 60 text
def
local_grid
!
puts
request_lat,
request_lon
]
!
local_grid
lat, lon
closest_car
!
!
end
end
!
!
!
!
!
!
!
!
!
local_grid.keys.sort.each do |distance|
lat, lon = local_grid[distance].split("_")
closest_car = get_cars_at_location(lat, lon)
Slide 61
Slide 61 text
def
local_grid
!
puts
request_lat,
request_lon
]
!
local_grid
lat, lon
closest_car
!
!
end
end
!
!
!
!
!
!
!
!
!
!
!
closest_car = get_cars_at_location(lat, lon)
Slide 62
Slide 62 text
Still with me?
Slide 63
Slide 63 text
def get_cars_at_location(lat, lon)
current_timestamp = Time.now.utc -‐ 1
bucket = $client.bucket(current_timestamp.strftime($timestamp_format))
object = bucket.get_or_new("%s_%s" % [ lat, lon ])
!
cars = GSet.new
!
if object.siblings.length > 1
object.siblings.each do |sibling|
unless sibling.data.nil?
cars.merge_json(sibling.data)
end
end
!
resolved_object = bucket.new("%s_%s" % [ lat, lon ])
resolved_object.vclock = object.vclock
resolved_object.content_type = "application/json"
resolved_object.data = cars.to_json
!
resolved_object.store(returnbody: false)
elsif !object.data.nil?
cars.merge_json(object.data)
end
!
cars
end
Slide 64
Slide 64 text
get_cars_at_location(lat, lon)
def
current_timestamp
bucket
object
!
cars
!
if
object
cars
!
resolved_object
resolved_object
resolved_object
resolved_object
!
resolved_object
elsif
cars
end
!
cars
end
Slide 65
Slide 65 text
!
current_timestamp = Time.now.utc -‐ 1
bucket = $client.bucket(current_timestamp.strftime($timestamp_format))
object = bucket.get_or_new("%s_%s" % [ lat, lon ])
def
current_timestamp
bucket
object
!
cars
!
if
object
cars
!
resolved_object
resolved_object
resolved_object
resolved_object
!
resolved_object
elsif
cars
end
!
cars
end
Slide 66
Slide 66 text
!
!
!
!
!
cars = GSet.new
def
current_timestamp
bucket
object
!
cars
!
if
object
cars
!
resolved_object
resolved_object
resolved_object
resolved_object
!
resolved_object
elsif
cars
end
!
cars
end
Slide 67
Slide 67 text
def
current_timestamp
bucket
object
!
cars
!
if
object
cars
!
resolved_object
resolved_object
resolved_object
resolved_object
!
resolved_object
elsif
cars
end
!
cars
end
!
!
!
!
!
!
!
if object.siblings.length > 1
Slide 68
Slide 68 text
Siblings are a mechanism
to prevent data loss
during concurrent writes
or network partitions.
Slide 69
Slide 69 text
A A A
Slide 70
Slide 70 text
A A A
B C
Slide 71
Slide 71 text
A A A
B C
Slide 72
Slide 72 text
A A A
B C
{B,C} {B,C} {B,C}
Slide 73
Slide 73 text
{
"type": "GSet",
"a": ["car1", "car22"]
}
{
"type": "GSet",
"a": ["car7"]
}
Slide 74
Slide 74 text
!
!
!
!
!
!
!
!
object.siblings.each do |sibling|
unless sibling.data.nil?
cars.merge_json(sibling.data)
end
end
def
current_timestamp
bucket
object
!
cars
!
if
object
cars
!
resolved_object
resolved_object
resolved_object
resolved_object
!
resolved_object
elsif
cars
end
!
cars
end
Slide 75
Slide 75 text
!
!
!
!
!
!
!
!
!
!
!
!
!
!
resolved_object = bucket.new("%s_%s" % [ lat, lon ])
resolved_object.vclock = object.vclock
resolved_object.content_type = "application/json"
resolved_object.data = cars.to_json
!
resolved_object.store(returnbody: false)
def
current_timestamp
bucket
object
!
cars
!
if
object
cars
!
resolved_object
resolved_object
resolved_object
resolved_object
!
resolved_object
elsif
cars
end
!
cars
end
Slide 76
Slide 76 text
B C
{B,C} {B,C} {B,C}
Slide 77
Slide 77 text
D D D
B C
{B,C} {B,C} {B,C}
Slide 78
Slide 78 text
{
"type": "GSet",
"a": ["car1", "car22", "car7"]
}
Slide 79
Slide 79 text
def
local_grid
!
puts
request_lat,
request_lon
]
!
local_grid
lat, lon
closest_car
!
!
end
end
!
!
!
!
!
!
!
!
!
!
!
closest_car = get_cars_at_location(lat, lon)
Slide 80
Slide 80 text
def
local_grid
!
puts
request_lat,
request_lon
]
!
local_grid
lat, lon
closest_car
!
!
end
end
!
!
!
!
!
!
!
!
!
!
!
!
!
if closest_car.members.length > 0
puts "\n%% Cars closest to you:\n\n"
puts JSON.pretty_generate(closest_car.members.to_a)
puts
!
break
end
Slide 81
Slide 81 text
No content
Slide 82
Slide 82 text
Data types
Available in Riak 2.0.
Slide 83
Slide 83 text
bucket = $client.bucket(current_timestamp.strftime($timestamp_format))
cars = Riak::Crdt::Set.new(bucket, "%s_%s" % [ lat, lon ])
cars.add(car_id)
!
# -‐-‐
!
cars.members
!
#
Slide 84
Slide 84 text
No content
Slide 85
Slide 85 text
When would doing all of
this work be attractive?
Slide 86
Slide 86 text
What if the latency, scale,
or high-availability
requirements you’re facing
make most of those
benefits disappear?
Slide 87
Slide 87 text
Thanks for listening.
E-mail: [email protected]
Twitter: @hectcastro
Web: http://docs.basho.com