Under the Covers of DynamoDB

Slide 1

Slide 1 text

Under the Covers of DynamoDB Matt Wood Principal Data Scientist @mza

Slide 2

Slide 2 text

Hello.

Slide 3

Slide 3 text

1. Getting started 2. Data modeling 3. Partitioning 4. Replication & Analytics Overview 5. Customer story: Localytics

Slide 4

Slide 4 text

Getting started 1

Slide 5

Slide 5 text

DynamoDB is a managed NoSQL database service. Store and retrieve any amount of data. Serve any level of request traffic.

Slide 6

Slide 6 text

Without the operational burden.

Slide 7

Slide 7 text

Consistent, predictable performance. Single digit millisecond latency. Backed on solid-state drives.

Slide 8

Slide 8 text

Flexible data model. Key/attribute pairs. No schema required. Easy to create. Easy to adjust.

Slide 9

Slide 9 text

Seamless scalability. No table size limits. Unlimited storage. No downtime.

Slide 10

Slide 10 text

Durable. Consistent, disk only writes. Replication across data centers and availability zones.

Slide 11

Slide 11 text

Without the operational burden.

Slide 12

Slide 12 text

Focus on your app.

Slide 13

Slide 13 text

Two decisions + three clicks = ready for use

Slide 14

Slide 14 text

Two decisions + three clicks = ready for use Primary keys Level of throughput

Slide 15

Slide 15 text

Two decisions + three clicks = ready for use Primary keys Level of throughput

Slide 16

Slide 16 text

Provisioned throughput. Reserve IOPS for reads and writes. Scale up for down at any time.

Slide 17

Slide 17 text

Pay per capacity unit. Priced per hour of provisioned throughput.

Slide 18

Slide 18 text

Write throughput. Size of item x writes per second $0.0065 for 10 write units

Slide 19

Slide 19 text

Consistent writes. Atomic increment and decrement. Optimistic concurrency control: conditional writes.

Slide 20

Slide 20 text

Transactions. Item level transactions only. Puts, updates and deletes are ACID.

Slide 21

Slide 21 text

Read throughput. Strong or eventual consistency

Slide 22

Slide 22 text

Read throughput. Strong or eventual consistency Provisioned units = size of item x reads per second $0.0065 per hour for 50 units

Slide 23

Slide 23 text

Read throughput. Strong or eventual consistency Provisioned units = size of item x reads per second $0.0065 per hour for 100 units 2

Slide 24

Slide 24 text

Read throughput. Strong or eventual consistency Same latency expectations. Mix and match at ‘read time’.

Slide 25

Slide 25 text

Provisioned throughput is managed by DynamoDB.

Slide 26

Slide 26 text

Data is partitioned and managed by DynamoDB.

Slide 27

Slide 27 text

Indexed data storage. $0.25 per GB per month. Tiered bandwidth pricing: aws.amazon.com/dynamodb/pricing

Slide 28

Slide 28 text

Reserved capacity. Up to 53% for 1 year reservation. Up to 76% for 3 year reservation.

Slide 29

Slide 29 text

Authentication. Session based to minimize latency. Uses the Amazon Security Token Service. Handled by AWS SDKs. Integrates with IAM.

Slide 30

Slide 30 text

Monitoring. CloudWatch metrics: latency, consumed read and write throughput, errors and throttling.

Slide 31

Slide 31 text

Libraries, mappers and mocks. ColdFusion, Django, Erlang, Java, .Net, Node.js, Perl, PHP, Python, Ruby http://j.mp/dynamodb-libs

Slide 32

Slide 32 text

Data modeling 2

Slide 33

Slide 33 text

id = 100 date = 2012-05-16-09-00-10 total = 25.00 id = 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00

Slide 34

Slide 34 text

id = 100 date = 2012-05-16-09-00-10 total = 25.00 id = 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 Table

Slide 35

Slide 35 text

id = 100 date = 2012-05-16-09-00-10 total = 25.00 id = 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 Item

Slide 36

Slide 36 text

id = 100 date = 2012-05-16-09-00-10 total = 25.00 id = 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 Attribute

Slide 37

Slide 37 text

Where is the schema? Tables do not require a formal schema. Items are an arbitrarily sized hash.

Slide 38

Slide 38 text

Indexing. Items are indexed by primary and secondary keys. Primary keys can be composite. Secondary keys are local to the table.

Slide 39

Slide 39 text

ID Date Total

Slide 40

Slide 40 text

ID Date Total Hash key

Slide 41

Slide 41 text

ID Date Total Hash key Range key Composite primary key

Slide 42

Slide 42 text

ID Date Total Hash key Range key Secondary range key

Slide 43

Slide 43 text

Programming DynamoDB. Small but perfectly formed API.

Slide 44

Slide 44 text

CreateTable UpdateTable DeleteTable DescribeTable ListTables Query Scan PutItem GetItem UpdateItem DeleteItem BatchGetItem BatchWriteItem

Slide 45

Slide 45 text

CreateTable UpdateTable DeleteTable DescribeTable ListTables Query Scan PutItem GetItem UpdateItem DeleteItem BatchGetItem BatchWriteItem

Slide 46

Slide 46 text

CreateTable UpdateTable DeleteTable DescribeTable ListTables Query Scan PutItem GetItem UpdateItem DeleteItem BatchGetItem BatchWriteItem

Slide 47

Slide 47 text

Conditional updates. PutItem, UpdateItem, DeleteItem can take optional conditions for operation. UpdateItem performs atomic increments.

Slide 48

Slide 48 text

One API call, multiple items BatchGet returns multiple items by key. Throughput is measured by IO, not API calls. BatchWrite performs up to 25 put or delete operations.

Slide 49

Slide 49 text

CreateTable UpdateTable DeleteTable DescribeTable ListTables Query Scan PutItem GetItem UpdateItem DeleteItem BatchGetItem BatchWriteItem

Slide 50

Slide 50 text

Query vs Scan Query returns items by key. Scan reads the whole table sequentially.

Slide 51

Slide 51 text

Query patterns Retrieve all items by hash key. Range key conditions: ==, <, >, >=, <=, begins with, between. Counts. Top and bottom n values. Paged responses.

Slide 52

Slide 52 text

Mapping relationships. E X A M P L E 1 :

Slide 53

Slide 53 text

Players user_id = mza location = Cambridge joined = 2011-07-04 user_id = jeffbarr location = Seattle joined = 2012-01-20 user_id = werner location = Worldwide joined = 2011-05-15

Slide 54

Slide 54 text

Slide 55

Slide 55 text

Players user_id = mza location = Cambridge joined = 2011-07-04 user_id = jeffbarr location = Seattle joined = 2012-01-20 user_id = werner location = Worldwide joined = 2011-05-15 Scores Leader boards user_id = mza game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = werner location = bejewelled score = 55,000 game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = mza game = tetris score = 9,000,000 user_id = jeffbarr

Slide 56

Slide 56 text

Players user_id = mza location = Cambridge joined = 2011-07-04 user_id = jeffbarr location = Seattle joined = 2012-01-20 user_id = werner location = Worldwide joined = 2011-05-15 user_id = mza game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = werner location = bejewelled score = 55,000 Scores game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = mza game = tetris score = 9,000,000 user_id = jeffbarr Leader boards Query for scores by user

Slide 57

Slide 57 text

Slide 58

Slide 58 text

Storing large items. E X A M P L E 2 :

Slide 59

Slide 59 text

Unlimited storage. Unlimited attributes per item. Unlimited items per table. Maximum of 64k per item.

Slide 60

Slide 60 text

message_id = 1 part = 1 message = message_id = 1 part = 2 message = message_id = 1 part = 3 joined = Split across items.

Slide 61

Slide 61 text

message_id = 1 message = http://s3.amazonaws.com... message_id = 2 message = http://s3.amazonaws.com... message_id = 3 message = http://s3.amazonaws.com... Store a pointer to S3.

Slide 62

Slide 62 text

Time series data E X A M P L E 3 :

Slide 63

Slide 63 text

event_id = 1000 timestamp = 2013-04-16-09-59-01 key = value event_id = 1001 timestamp = 2013-04-16-09-59-02 key = value event_id = 1002 timestamp = 2013-04-16-09-59-02 key = value Hot and cold tables. April March event_id = 1000 timestamp = 2013-03-01-09-59-01 key = value event_id = 1001 timestamp = 2013-03-01-09-59-02 key = value event_id = timestamp = key =

Slide 64

Slide 64 text

April March February January December

Slide 65

Slide 65 text

Archive data. Move old data to S3: lower cost. Still available for analytics. Run queries across hot and cold data with Elastic MapReduce.

Slide 66

Slide 66 text

Partitioning 3

Slide 67

Slide 67 text

Uniform workload. Data stored across multiple partitions. Data is primarily distributed by primary key. Provisioned throughput is divided evenly across partitions.

Slide 68

Slide 68 text

To achieve and maintain full provisioned throughput, spread workload evenly across hash keys.

Slide 69

Slide 69 text

Non-Uniform workload. Might be throttled, even at high levels of throughput.

Slide 70

Slide 70 text

Distinct values for hash keys. B E S T P R A C T I C E 1 : Hash key elements should have a high number of distinct values.

Slide 71

Slide 71 text

user_id = mza first_name = Matt last_name = Wood user_id = jeffbarr first_name = Jeff last_name = Barr user_id = werner first_name = Werner last_name = Vogels user_id = simone first_name = Simone last_name = Brunozzi ... ... ... Lots of users with unique user_id. Workload well distributed across hash key.

Slide 72

Slide 72 text

Avoid limited hash key values. B E S T P R A C T I C E 2 : Hash key elements should have a high number of distinct values.

Slide 73

Slide 73 text

status = 200 date = 2012-04-01-00-00-01 status = 404 date = 2012-04-01-00-00-01 status 404 date = 2012-04-01-00-00-01 status = 404 date = 2012-04-01-00-00-01 Small number of status codes. Unevenly, non-uniform workload.

Slide 74

Slide 74 text

Model for even distribution. B E S T P R A C T I C E 3 : Access by hash key value should be evenly distributed across the dataset.

Slide 75

Slide 75 text

mobile_id = 100 access_date = 2012-04-01-00-00-01 mobile_id = 100 access_date = 2012-04-01-00-00-02 mobile_id = 100 access_date = 2012-04-01-00-00-03 mobile_id = 100 access_date = 2012-04-01-00-00-04 ... ... Large number of devices. Small number which are much more popular than others. Workload unevenly distributed.

Slide 76

Slide 76 text

mobile_id = 100.1 access_date = 2012-04-01-00-00-01 mobile_id = 100.2 access_date = 2012-04-01-00-00-02 mobile_id = 100.3 access_date = 2012-04-01-00-00-03 mobile_id = 100.4 access_date = 2012-04-01-00-00-04 ... ... Sample access pattern. Workload randomized by hash key.

Slide 77

Slide 77 text

Replication & Analytics 4

Slide 78

Slide 78 text

Seamless scale. Scalable methods for data processing. Scalable methods for backup/restore.

Slide 79

Slide 79 text

Amazon Elastic MapReduce. Managed Hadoop service for data-intensive workflows. aws.amazon.com/emr

Slide 80

Slide 80 text

create external table items_db (id string, votes bigint, views bigint) stored by 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler' tblproperties ("dynamodb.table.name" = "items", "dynamodb.column.mapping" = "id:id,votes:votes,views:views");

Slide 81

Slide 81 text

select id, likes, views from items_db order by views desc;

Slide 82

Slide 82 text

Slide 83

Slide 83 text

Mohit Dilawari Director of Engineering @mdilawari DynamoDB @ Localytics

Slide 84

Slide 84 text

About Localytics 84 • Mobile App Analytics Service • 750+ Million Devices and over 20,000 Apps • Customers Include: …and many more.

Slide 85

Slide 85 text

About the Development Team 85 • Small team of four managing entire AWS infrastructure - 100 EC2 Instances • Experts in BigData • Leveraging Amazon's service has been the key to our success • Large scale users of: • SQS • S3 • ELB • RDS • Route53 • Elastic Cache • EMR …and of course DynamoDB

Slide 86

Slide 86 text

Why DynamoDB? 86 Set it and Forget it

Slide 87

Slide 87 text

Our use-case: Dedup Data 87 • Each datapoint includes a globally unique ID • Mobile traffic over 2G/3G will upload periodic duplicate data • We accept data up to a 28 day window

Slide 88

Slide 88 text

First Design for Dedup table 88 Unique ID: aaaaaaaaaaaaaaaaaaaaaaaaa333333333333333 Table Name = dedup_table ID aaaaaaaaaaaaaaaaaaaaaaaaa111111111111111 aaaaaaaaaaaaaaaaaaaaaaaaa222222222222222 "Test and Set" in a single operation aaaaaaaaaaaaaaaaaaaaaaaaa333333333333333

Slide 89

Slide 89 text

Optimization One - Data Aging 89 • Partition by Month • Create new table day before the month • Need to keep two months of data

Slide 90

Slide 90 text

Optimization One - Data Aging 90 Unique ID: bbbbbbbbbbbbbbbbbbbbbbbbb333333333333333 Check Previous month: Table Name = March2013_dedup ID aaaaaaaaaaaaaaaaaaaaaaaaa111111111111111 aaaaaaaaaaaaaaaaaaaaaaaaa222222222222222 Not Here!

Slide 91

Slide 91 text

Optimization One - Data Aging 91 Unique ID: bbbbbbbbbbbbbbbbbbbbbbbbb333333333333333 Test and Set in current month: Inserted Table Name = April2013_dedup ID bbbbbbbbbbbbbbbbbbbbbbbbb111111111111111 bbbbbbbbbbbbbbbbbbbbbbbbb222222222222222 bbbbbbbbbbbbbbbbbbbbbbbbb333333333333333

Slide 92

Slide 92 text

Optimization Two 92 • Reduce the index size - Reduces costs • Each item has a 100 byte overhead which is substantial • Combine multiple IDs together to one record • Split each ID into two halves o First half is the key. Second Half is added to the set

Slide 93

Slide 93 text

Optimization Two - Use Sets 93 Unique ID: ccccccccccccccccccccccccccc999999999999999 Prefix Values aaaaaaaaaaaaaaaaaaaaaaaaa [111111111111111, 222222222222222, 333333333333333] bbbbbbbbbbbbbbbbbbbbbbbbb [444444444444444, 555555555555555, 666666666666666] ccccccccccccccccccccccccccc [777777777777777, 888888888888888, ] ccccccccccccccccccccccccccc 999999999999999

Slide 94

Slide 94 text

Optimization Three - Combine Months 94 • Go back to a single table Prefix March2013 April2013 aaaaaaaaaa... [111111111111111, 22222222222... [1212121212121212, 3434343434.... bbbbbbbbbb... [444444444444444, 555555555.... [4545454545454545, 6767676767..... ccccccccccc... [777777777777777, 888888888... [8989898989898989, 1313131313.... One Operation 1. Delete February2013 Field 2. Check ID in March2013 3. Test and Set into April 2013

Slide 95

Slide 95 text

Recap 95 Compare Plans for 20 Billion IDs per month Plan Storage Costs Read Costs Write Costs Total Savings Naive (after a year) $8400 0 $4000 $12400 Data Age $900 $350 $4000 $5250 57% Using Sets $150 $350 $4000 $4500 64% Multiple Months $150 0 $4000 $4150 67%

Slide 96

Slide 96 text

96 Thank You @mdilawari

Slide 97

Slide 97 text

1. Getting started 2. Data modeling 3. Partitioning 4. Replication & Analytics Summary 5. Customer story: Localytics