Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding DynamoDB (Tokyo NoSQL Night)

Avatar for Alex DeBrie Alex DeBrie
September 26, 2023
54

Understanding DynamoDB (Tokyo NoSQL Night)

Avatar for Alex DeBrie

Alex DeBrie

September 26, 2023
Tweet

Transcript

  1. @alexbdebrie Understanding DynamoDB • AWS Data Hero • Wrote The

    DynamoDB Book • Created DynamoDBGuide.com • AWS / serverless fan Alex DeBrie
  2. @alexbdebrie Understanding DynamoDB • DynamoDB background • Basic terminology •

    SQL vs. NoSQL • Data modeling process • Hard patterns in DynamoDB Overview
  3. @alexbdebrie Understanding DynamoDB • Fully-managed & proprietary to AWS •

    Infinite scaling with no performance degradation • Serverless-friendly: • HTTP connection model • IAM authentication • Infrastructure-as-code • Flexible pricing model: • Pay for reads & writes directly • Provisioned vs. on-demand mode Key characteristics
  4. @alexbdebrie Understanding DynamoDB • Key-value or wide-column data model •

    Key-value: distributed hash table (dictionary, map) • Wide-column: hash-table + B-tree • B-tree == phone book • Wide-column == Bookshelf full of phone books Data model
  5. @alexbdebrie Understanding DynamoDB • Simple primary key (partition key) •

    Composite primary key (partition key + sort key) Primary key
  6. @alexbdebrie Understanding DynamoDB • Consistency • Same response time, regardless

    of: • Database size • Concurrent queries • Reduce need for expertise Goals
  7. @alexbdebrie Understanding DynamoDB Nelson Elhage, Some opinionated thoughts on SQL

    databases https://blog.nelhage.com/post/some-opinionated-sql-takes/
  8. @alexbdebrie Understanding DynamoDB • Consistency • Same response time, regardless

    of: • Database size • Concurrent queries • Reduce need for expertise Goals
  9. @alexbdebrie Understanding DynamoDB • Pay for reads and writes directly

    rather than resources • Traditional databases: CPU, RAM, IOPS • DynamoDB: RCUs & WCUs • Provisioned capacity vs. pay-per-use • Do the math! Billing model
  10. @alexbdebrie Understanding DynamoDB • Consistency • Predictability • Reliability •

    Fully managed service • Few outages • DynamoDB global tables Goals
  11. @alexbdebrie Understanding DynamoDB Partition 1 Partition 2 PutItem: Email: “alexdebrie1..”

    OrderId:“01F37K…” fx(Email): This item belongs to Partition 1
  12. @alexbdebrie Understanding DynamoDB Partition 1 Partition 2 PutItem: Email: “alexdebrie1..”

    OrderId:“01F37K…” fx(Email): This item belongs to Partition 1
  13. @alexbdebrie Understanding DynamoDB Partition 1 Partition 2 Partition 3 PutItem:

    Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1
  14. @alexbdebrie Understanding DynamoDB Partition 1 Partition 2 Partition 3 Partition

    4 Partition 5 Partition 6 Partition 7 Partition 8 Partition 9 PutItem: Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1
  15. @alexbdebrie Understanding DynamoDB Partition 1 Partition 2 Partition 3 Partition

    4 Partition 5 Partition 6 Partition 7 Partition 8 Partition 9 Constant time (O(1)) PutItem: Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1
  16. @alexbdebrie Understanding DynamoDB • Importance of primary key • Clear,

    harsh limits • 1MB page size limit for Query + Scan Choices
  17. @alexbdebrie Understanding DynamoDB • Importance of primary key • Clear,

    harsh limits • 1MB page size limit for Query + Scan Choices
  18. @alexbdebrie Understanding DynamoDB • Importance of primary key • Clear,

    harsh limits • 1MB page size limit for Query + Scan • Partition throughput limits Choices
  19. @alexbdebrie Understanding DynamoDB Partition 1 Partition 2 Partition 3 Partition

    4 Partition 5 Partition 6 Partition 7 Partition 8 Partition 9 PutItem: Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1
  20. @alexbdebrie Understanding DynamoDB Partition 1 Partition 2 Partition 3 Partition

    4 Partition 5 Partition 6 Partition 7 Partition 8 Partition 9 PutItem: Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1
  21. @alexbdebrie Understanding DynamoDB Partition 1 Partition 2 Partition 3 Partition

    4 Partition 5 Partition 6 Partition 7 Partition 8 Partition 9 PutItem: Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1 1k WCU/s 3k RCU/s
  22. @alexbdebrie Understanding DynamoDB Partition 1 Partition 2 Partition 3 Partition

    4 Partition 5 Partition 6 Partition 7 Partition 8 Partition 9 PutItem: Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1 1k WCU/s 3k RCU/s
  23. @alexbdebrie Understanding DynamoDB • Importance of primary key • Clear,

    harsh limits • 1MB page size limit for Query + Scan • Partition throughput limits Choices
  24. @alexbdebrie Understanding DynamoDB • Importance of primary key • Clear,

    harsh limits • No joins, no aggregations Choices
  25. @alexbdebrie Understanding DynamoDB Nelson Elhage, Some opinionated thoughts on SQL

    databases https://blog.nelhage.com/post/some-opinionated-sql-takes/
  26. @alexbdebrie Understanding DynamoDB • Importance of primary key • Clear,

    harsh limits • No joins, no aggregations Choices
  27. @alexbdebrie Understanding DynamoDB • Hard work upfront • Design for

    your access patterns • Maintain aggregations Implications
  28. @alexbdebrie Understanding DynamoDB • Hard work upfront • Design for

    your access patterns • Maintain aggregations • Consider pagination + sorting Implications
  29. @alexbdebrie Understanding DynamoDB • Hard work upfront • Design for

    your access patterns • Maintain aggregations • Consider pagination + sorting • Shard if needed Implications
  30. @alexbdebrie Understanding DynamoDB Partition 1 Partition 2 Partition 3 Partition

    4 Partition 5 Partition 6 Partition 7 Partition 8 Partition 9 Constant time (O(1)) PutItem: Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1
  31. @alexbdebrie Understanding DynamoDB • Data from base table is copied

    into secondary index with new primary key • Can use read-based operations on secondary indexes • No writes Secondary indexes
  32. @alexbdebrie Understanding DynamoDB • Hard work upfront • Use secondary

    indexes • Get comfortable with denormalization Implications
  33. @alexbdebrie Understanding DynamoDB • Hard work upfront • Use secondary

    indexes • Get comfortable with denormalization • Duplication Implications
  34. @alexbdebrie Understanding DynamoDB • Hard work upfront • Use secondary

    indexes • Get comfortable with denormalization • Duplication • Nested relationships Implications
  35. @alexbdebrie Understanding DynamoDB • Understand your application • Create your

    entity-relationship diagram (ERD) • List your access patterns • Design your primary keys to handle your access patterns Data modeling process
  36. @alexbdebrie Understanding DynamoDB • Online gaming application • Users sign

    up for a game • Can hold inventory items • Can go on quests • Can belong to a guild Example:
  37. @alexbdebrie Understanding DynamoDB • Understand your application • Create your

    entity-relationship diagram (ERD) • List your access patterns • Design your primary keys to handle your access patterns Data modeling process
  38. @alexbdebrie Understanding DynamoDB • Understand your application • Create your

    entity-relationship diagram (ERD) • List your access patterns • Design your primary keys to handle your access patterns Data modeling process
  39. @alexbdebrie Understanding DynamoDB • Understand your application • Create your

    entity-relationship diagram (ERD) • List your access patterns • Design your primary keys to handle your access patterns Data modeling process
  40. @alexbdebrie Understanding DynamoDB • Understand your application • Create your

    entity-relationship diagram (ERD) • List your access patterns • Design your primary keys to handle your access patterns Data modeling process
  41. @alexbdebrie Understanding DynamoDB • Think: • What REST API endpoints

    do I expose? • Or: what methods does my service offer? • Be thorough and specific List your access patterns
  42. @alexbdebrie Understanding DynamoDB • Understand your application • Create your

    entity-relationship diagram (ERD) • List your access patterns • Design your primary keys to handle your access patterns Data modeling process
  43. @alexbdebrie Understanding DynamoDB • Understand your application • Create your

    entity-relationship diagram (ERD) • List your access patterns • Design your primary keys to handle your access patterns Data modeling process
  44. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Single-item write operations Handling your access patterns
  45. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Single-item write operations Handling your access patterns
  46. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Single-item write operations (PutItem, UpdateItem, DeleteItem) Handling your access patterns
  47. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations Handling your access patterns
  48. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations Handling your access patterns
  49. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) Handling your access patterns
  50. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) • Single-item read operations Handling your access patterns
  51. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) • Single-item read operations Handling your access patterns
  52. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) • Single-item read operations (GetItem) Handling your access patterns
  53. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) • Single-item read operations (GetItem) • Multi-item read operations Handling your access patterns
  54. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) • Single-item read operations (GetItem) • Multi-item read operations Handling your access patterns
  55. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Single-item write operations (PutItem, UpdateItem, DeleteItem) • Multi-item write operations (TransactWriteItems) • Single-item read operations (GetItem) • Multi-item read operations (Query) Handling your access patterns
  56. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Think of what context is available when making a request Handling your access patterns
  57. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Think of what context is available when making a request • E.g. Username, TransactionId Handling your access patterns
  58. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Think of what context is available when making a request • E.g. Username, TransactionId • Which patterns to handle first? Handling your access patterns
  59. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Think of what context is available when making a request • E.g. Username, TransactionId • Which patterns to handle first? • Writes (with conditions) Handling your access patterns
  60. @alexbdebrie Understanding DynamoDB • Think of types of patterns •

    Think of what context is available when making a request • E.g. Username, TransactionId • Which patterns to handle first? • Writes (with conditions) • ‘Core’ entities Handling your access patterns
  61. @alexbdebrie Understanding DynamoDB • Understand your application • Create your

    entity-relationship diagram (ERD) • List your access patterns • Design your primary keys to handle your access patterns Data modeling process
  62. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • “How many transactions has Customer X done each month?” • “What is the largest purchase done by Customer Y?” Hard patterns in DynamoDB
  63. @alexbdebrie Understanding DynamoDB • Importance of primary key • Clear,

    harsh limits • No joins, no aggregations Choices
  64. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • “How many transactions has Customer X done each month?” • “What is the largest purchase done by Customer Y?” Hard patterns in DynamoDB
  65. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • Complex filtering Hard patterns in DynamoDB
  66. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • Complex filtering • Filtering by 2 or more properties, all of which are optional Hard patterns in DynamoDB
  67. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • Complex filtering • Filtering by 2 or more properties, all of which are optional • ”Find me all transactions by Customer X Hard patterns in DynamoDB
  68. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • Complex filtering • Filtering by 2 or more properties, all of which are optional • ”Find me all transactions by Customer X where token = ‘ETH’ Hard patterns in DynamoDB
  69. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • Complex filtering • Filtering by 2 or more properties, all of which are optional • ”Find me all transactions by Customer X where token = ‘ETH’ and amount > 100 Hard patterns in DynamoDB
  70. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • Complex filtering • Filtering by 2 or more properties, all of which are optional • ”Find me all transactions by Customer X where token = ‘ETH’ and amount > 100 and date between 2022-01-01 and 2023-01-01 Hard patterns in DynamoDB
  71. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • Complex filtering • Filtering by 2 or more properties, all of which are optional • ”Find me all transactions by Customer X where token = ‘ETH’ and amount > 100 and date between 2022-01-01 and 2023-01-01 • DynamoDB likes exact matches! Hard patterns in DynamoDB
  72. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • Complex filtering Hard patterns in DynamoDB
  73. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • Complex filtering • Search Hard patterns in DynamoDB
  74. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • Complex filtering • Search • Queries across entire table Hard patterns in DynamoDB
  75. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • Complex filtering • Search • Queries across entire table • “Find the most recent 10 transactions on our platform” Hard patterns in DynamoDB
  76. @alexbdebrie Understanding DynamoDB Partition 1 Partition 2 Partition 3 Partition

    4 Partition 5 Partition 6 Partition 7 Partition 8 Partition 9 Constant time (O(1)) PutItem: Email: “alexdebrie1..” OrderId:“01F37K…” fx(Email): This item belongs to Partition 1
  77. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Aggregations • Complex filtering • Search • Queries across entire table • “Find the most recent 10 transactions on our platform” Hard patterns in DynamoDB
  78. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Hard to change / add later Hard patterns in DynamoDB
  79. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Hard to change / add later • New conditions Hard patterns in DynamoDB
  80. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Hard to change / add later • New conditions • Changing primary key Hard patterns in DynamoDB
  81. @alexbdebrie Understanding DynamoDB • Hard patterns at any time •

    Hard to change / add later • New conditions • Changing primary key • New indexing if new attribute – ETL process Hard patterns in DynamoDB
  82. @alexbdebrie Understanding DynamoDB • DynamoDB background • Basic terminology •

    SQL vs. NoSQL • Data modeling process • Hard patterns in DynamoDB Overview