Understanding DynamoDB (Tokyo NoSQL Night)

Understanding + using DynamoDB @alexbdebrie

@alexbdebrie Understanding DynamoDB • AWS Data Hero • Wrote The
DynamoDB Book • Created DynamoDBGuide.com • AWS / serverless fan Alex DeBrie

@alexbdebrie Understanding DynamoDB • DynamoDB background • Basic terminology •
SQL vs. NoSQL • Data modeling process • Hard patterns in DynamoDB Overview

DynamoDB background @alexbdebrie

@alexbdebrie Understanding DynamoDB • Key characteristics • Data model DynamoDB
background

@alexbdebrie Understanding DynamoDB • Fully-managed & proprietary to AWS •
Infinite scaling with no performance degradation • Serverless-friendly: • HTTP connection model • IAM authentication • Infrastructure-as-code • Flexible pricing model: • Pay for reads & writes directly • Provisioned vs. on-demand mode Key characteristics

@alexbdebrie Understanding DynamoDB • Key-value or wide-column data model •
Key-value: distributed hash table (dictionary, map) • Wide-column: hash-table + B-tree • B-tree == phone book • Wide-column == Bookshelf full of phone books Data model

DynamoDB Basics @alexbdebrie

@alexbdebrie Understanding DynamoDB • Table • Item • Primary key
• Attribute DynamoDB basics

@alexbdebrie Understanding DynamoDB

@alexbdebrie Understanding DynamoDB Table

@alexbdebrie Understanding DynamoDB Item

@alexbdebrie Understanding DynamoDB Primary key

@alexbdebrie Understanding DynamoDB Attributes

@alexbdebrie Understanding DynamoDB • Simple primary key (partition key) •
Composite primary key (partition key + sort key) Primary key

@alexbdebrie Understanding DynamoDB Simple primary key

@alexbdebrie Understanding DynamoDB Composite primary key

@alexbdebrie Understanding DynamoDB Composite primary key Must be unique!

From SQL to NoSQL @alexbdebrie

@alexbdebrie Understanding DynamoDB • Goals • Choices • Implications From
SQL to NoSQL

@alexbdebrie Understanding DynamoDB Goals

@alexbdebrie Understanding DynamoDB • Consistency Goals

@alexbdebrie Understanding DynamoDB 😍😍

@alexbdebrie Understanding DynamoDB Investigation 😍😍

@alexbdebrie Understanding DynamoDB Investigation Re-architect 😍😍

@alexbdebrie Understanding DynamoDB • Consistency • Same response time, regardless
of: Goals

of: • Database size Goals

of: • Database size • Concurrent queries Goals

of: • Database size • Concurrent queries • Reduce need for expertise Goals

@alexbdebrie Understanding DynamoDB Nelson Elhage, Some opinionated thoughts on SQL
databases https://blog.nelhage.com/post/some-opinionated-sql-takes/

of: • Database size • Concurrent queries • Reduce need for expertise Goals

@alexbdebrie Understanding DynamoDB • Consistency • Predictability Goals

@alexbdebrie Understanding DynamoDB • Consistency • Predictability • Performance Goals

@alexbdebrie Understanding DynamoDB • Consistency • Predictability • Performance •
Billing Goals

@alexbdebrie Understanding DynamoDB • Pay for reads and writes directly
rather than resources • Traditional databases: CPU, RAM, IOPS • DynamoDB: RCUs & WCUs • Provisioned capacity vs. pay-per-use • Do the math! Billing model

@alexbdebrie Understanding DynamoDB • Consistency • Predictability • Performance •
Billing Goals

@alexbdebrie Understanding DynamoDB • Consistency • Predictability Goals

@alexbdebrie Understanding DynamoDB • Consistency • Predictability • Reliability Goals

@alexbdebrie Understanding DynamoDB • Consistency • Predictability • Reliability •
Fully managed service Goals

Fully managed service • Few outages Goals

Fully managed service • Few outages • DynamoDB global tables Goals

@alexbdebrie Understanding DynamoDB • Consistency • Predictability • Reliability Goals

SQL to NoSQL

@alexbdebrie Understanding DynamoDB Choices

@alexbdebrie Understanding DynamoDB • Importance of primary key Choices

@alexbdebrie Understanding DynamoDB Query with these

@alexbdebrie Understanding DynamoDB Query with these Not with these

@alexbdebrie Understanding DynamoDB Partition 1 Partition 2

@alexbdebrie Understanding DynamoDB Partition 1 Partition 2 PutItem: Email: “alexdebrie1..”
OrderId:“01F37K…”

@alexbdebrie Understanding DynamoDB Partition 1 Partition 2 PutItem: Email: “alexdebrie1..”
OrderId:“01F37K…” fx(Email): This item belongs to Partition 1

@alexbdebrie Understanding DynamoDB Partition 1 Partition 2 Partition 3 PutItem:
Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1

@alexbdebrie Understanding DynamoDB Partition 1 Partition 2 Partition 3 Partition
4 Partition 5 Partition 6 Partition 7 Partition 8 Partition 9 PutItem: Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1

4 Partition 5 Partition 6 Partition 7 Partition 8 Partition 9 Constant time (O(1)) PutItem: Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1

@alexbdebrie Understanding DynamoDB • Importance of primary key Choices

@alexbdebrie Understanding DynamoDB • Importance of primary key • Clear,
harsh limits Choices

harsh limits • 1MB page size limit for Query + Scan Choices

@alexbdebrie Understanding DynamoDB Query: CustomerEmail = “[email protected]”

@alexbdebrie Understanding DynamoDB 1MB max

harsh limits • 1MB page size limit for Query + Scan Choices

harsh limits • 1MB page size limit for Query + Scan • Partition throughput limits Choices

4 Partition 5 Partition 6 Partition 7 Partition 8 Partition 9 PutItem: Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1

4 Partition 5 Partition 6 Partition 7 Partition 8 Partition 9 PutItem: Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1 1k WCU/s 3k RCU/s

harsh limits • 1MB page size limit for Query + Scan • Partition throughput limits Choices

harsh limits Choices

harsh limits • No joins, no aggregations Choices

@alexbdebrie Understanding DynamoDB Nelson Elhage, Some opinionated thoughts on SQL
databases https://blog.nelhage.com/post/some-opinionated-sql-takes/

SQL to NoSQL

@alexbdebrie Understanding DynamoDB Implications

@alexbdebrie Understanding DynamoDB • Hard work upfront Implications

@alexbdebrie Understanding DynamoDB • Hard work upfront • Design for
your access patterns Implications

your access patterns • Maintain aggregations Implications

your access patterns • Maintain aggregations • Consider pagination + sorting Implications

your access patterns • Maintain aggregations • Consider pagination + sorting • Shard if needed Implications

@alexbdebrie Understanding DynamoDB • Hard work upfront Implications

@alexbdebrie Understanding DynamoDB • Hard work upfront • Use secondary
indexes Implications

@alexbdebrie Understanding DynamoDB • Data from base table is copied
into secondary index with new primary key • Can use read-based operations on secondary indexes • No writes Secondary indexes

indexes Implications

indexes • Get comfortable with denormalization Implications

indexes • Get comfortable with denormalization • Duplication Implications

indexes • Get comfortable with denormalization • Duplication • Nested relationships Implications

Data modeling process @alexbdebrie

Key point: With DynamoDB, you must design your table for
your access patterns. @alexbdebrie

@alexbdebrie Understanding DynamoDB • Understand your application • Create your
entity-relationship diagram (ERD) • List your access patterns • Design your primary keys to handle your access patterns Data modeling process

@alexbdebrie Understanding DynamoDB • Online gaming application • Users sign
up for a game • Can hold inventory items • Can go on quests • Can belong to a guild Example:

@alexbdebrie Understanding DynamoDB List your access patterns

@alexbdebrie Understanding DynamoDB • Think: • What REST API endpoints
do I expose? • Or: what methods does my service offer? • Be thorough and specific List your access patterns

@alexbdebrie Understanding DynamoDB List your access patterns

@alexbdebrie Understanding DynamoDB • Think of types of patterns •
Single-item write operations Handling your access patterns

@alexbdebrie Understanding DynamoDB List your access patterns Single Item patterns

Single-item write operations Handling your access patterns

Single-item write operations (PutItem, UpdateItem, DeleteItem) Handling your access patterns

Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations Handling your access patterns

@alexbdebrie Understanding DynamoDB List your access patterns Multi- item pattern

Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations Handling your access patterns

Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) Handling your access patterns

Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) • Single-item read operations Handling your access patterns

@alexbdebrie Understanding DynamoDB List your access patterns Single Item patterns

Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) • Single-item read operations Handling your access patterns

Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) • Single-item read operations (GetItem) Handling your access patterns

Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) • Single-item read operations (GetItem) • Multi-item read operations Handling your access patterns

@alexbdebrie Understanding DynamoDB List your access patterns Multi- Item patterns

Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) • Single-item read operations (GetItem) • Multi-item read operations Handling your access patterns

Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) • Single-item read operations (GetItem) • Multi-item read operations (Query) Handling your access patterns

@alexbdebrie Understanding DynamoDB • Think of types of patterns Handling
your access patterns

Think of what context is available when making a request Handling your access patterns

Think of what context is available when making a request • E.g. Username, TransactionId Handling your access patterns

Think of what context is available when making a request • E.g. Username, TransactionId • Which patterns to handle first? Handling your access patterns

Think of what context is available when making a request • E.g. Username, TransactionId • Which patterns to handle first? • Writes (with conditions) Handling your access patterns

Think of what context is available when making a request • E.g. Username, TransactionId • Which patterns to handle first? • Writes (with conditions) • ‘Core’ entities Handling your access patterns

Hard patterns in DynamoDB @alexbdebrie

@alexbdebrie Understanding DynamoDB • Hard patterns at any time •
Aggregations • “How many transactions has Customer X done each month?” • “What is the largest purchase done by Customer Y?” Hard patterns in DynamoDB

Aggregations • “How many transactions has Customer X done each month?” • “What is the largest purchase done by Customer Y?” Hard patterns in DynamoDB

Aggregations Hard patterns in DynamoDB

Aggregations • Complex filtering Hard patterns in DynamoDB

Aggregations • Complex filtering • Filtering by 2 or more properties, all of which are optional Hard patterns in DynamoDB

Aggregations • Complex filtering • Filtering by 2 or more properties, all of which are optional • ”Find me all transactions by Customer X Hard patterns in DynamoDB

Aggregations • Complex filtering • Filtering by 2 or more properties, all of which are optional • ”Find me all transactions by Customer X where token = ‘ETH’ Hard patterns in DynamoDB

Aggregations • Complex filtering • Filtering by 2 or more properties, all of which are optional • ”Find me all transactions by Customer X where token = ‘ETH’ and amount > 100 Hard patterns in DynamoDB

Aggregations • Complex filtering • Filtering by 2 or more properties, all of which are optional • ”Find me all transactions by Customer X where token = ‘ETH’ and amount > 100 and date between 2022-01-01 and 2023-01-01 Hard patterns in DynamoDB

Aggregations • Complex filtering • Filtering by 2 or more properties, all of which are optional • ”Find me all transactions by Customer X where token = ‘ETH’ and amount > 100 and date between 2022-01-01 and 2023-01-01 • DynamoDB likes exact matches! Hard patterns in DynamoDB

Aggregations • Complex filtering Hard patterns in DynamoDB

Aggregations • Complex filtering • Search Hard patterns in DynamoDB

Aggregations • Complex filtering • Search • Queries across entire table Hard patterns in DynamoDB

Aggregations • Complex filtering • Search • Queries across entire table • “Find the most recent 10 transactions on our platform” Hard patterns in DynamoDB

Hard to change / add later Hard patterns in DynamoDB

Hard to change / add later • New conditions Hard patterns in DynamoDB

Hard to change / add later • New conditions • Changing primary key Hard patterns in DynamoDB

Hard to change / add later • New conditions • Changing primary key • New indexing if new attribute – ETL process Hard patterns in DynamoDB

Summary @alexbdebrie

@alexbdebrie Understanding DynamoDB • DynamoDB background • Basic terminology •
SQL vs. NoSQL • Data modeling process • Hard patterns in DynamoDB Overview

@alexbdebrie Understanding DynamoDB • @alexbdebrie • [email protected] • dynamodbbook.com Questions??

Understanding DynamoDB (Tokyo NoSQL Night)

Understanding DynamoDB (Tokyo NoSQL Night)

Featured

Transcript