Serverlessを極めるためにDynamoDBデータモデリングを極めよう / Let’s become the master of DynamoDB Data Modeling to become the master of Serverless

Serverlessを極めるためにDynamoDBデータモデリングを極めよう / Let’s become the master of DynamoDB Data Modeling to become the master of Serverless

AWS Dev Day Tokyo 2018の登壇資料です
https://aws.amazon.com/jp/aws-devday-tokyo-2018/

5cd1b99b9950b26e8ee005bd6cd1a860?s=128

Masashi Terui

October 31, 2018
Tweet

Transcript

  1. LET’S BECOME THE MASTER OF DYNAMODB DATA MODELING TO BECOME

    THE MASTER OF SERVERLESS MASASHI TERUI @ AWS DEV DAY TOKYO 2018
  2. SERVERWORKS CO.,LTD. + FREELANCER • Serverless Oji-san • Serverless Framework

    Plugin Developer • Serverlessconf Tokyo 2016,2017,2018 speaker • Remote worker (in Sapporo-shi, Hokkaido) • http://marcy.hatenablog.com/entry/2018/08/04/231241 MASASHI TERUI ARCHITECT / DEVELOPER
  3. SERVERLESSͳΜͯ০ΓͰ͢ Ғ͍ਓʹ͸ͦΕ͕Θ͔ΒΜͷͰ͢Α ͜͜͸Ұ଍ઌʹ(উखʹ)DatabaseτϥοΫͰ͢ʂʂ ⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠ ͜ͷηογϣϯ͸ SERVERLESSཁૉ͕ ΄ͱΜͲ͋Γ·ͤΜʂʂ ⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠

  4. SERVERLESS΍ͬͯ·͔͢ʁ DynamoDB࢖ͬͯ·͔͢ʁ

  5. • ޷͖ • ී௨ • ݏ͍ • LambdaͱRDSͷ૬ੑ͕ѱ͍͔Β࢓ํͳ͘࢖͍ͬͯΔ DynamoDB͸޷͖Ͱ͔͢ʁ

  6. • ίωΫγϣϯϞσϧ • Ӭଓత • ηΩϡϦςΟ • ૊ΈࠐΈͷऑ͍ೝূ • VPC಺΁ͷΞΫηείετʢENIੜ੒ॲཧʣ

    LambdaͱRDSͷ૬ੑ͕ѱ͍ʁ ͱͯ΋OLTPͰ͸࢖͑ͳ͍(਺ඵʙे਺ඵ)
  7. ࢓ํ͕ͳ͍͔ΒDynamoDB…… ຊ౰͸RDBΛ࢖͍͍͚ͨͲ

  8. DynamoDBΛੵۃతʹ࢖͓͏ ͦΜͳϚΠϯυͰେৎ෉ʁ

  9. • ୺తʹݴͬͯόΠϒϧ • ͜ΕΛಡΜͰશͯཧղͰ͖ͨਓ͸
 ͨͿΜ͜ͷઌΛฉ͘ඞཁ͸͋Γ·ͤΜʂʂ • https://docs.aws.amazon.com/ja_jp/amazondynamodb/latest/ developerguide/best-practices.html DynamoDB Best

    Practices
  10. • ςʔϒϧઃܭͱ͔ΠϯσοΫεઃܭͬͯͲ͏΍Δͷʁ • ACIDτϥϯβΫγϣϯແ͍ͷʹେৎ෉ͳͷʁ • ݁Ռ੔߹ੑͬͯେৎ෉ͳͷʁ • ্ख͘ੑೳΛҾ͖ग़͢ʹ͸Ͳ͏͢Ε͹ྑ͍ʁ DynamoDBͷΑ͋͘Δෆ҆

  11. • ςʔϒϧઃܭͱ͔ΠϯσοΫεઃܭͬͯͲ͏΍Δͷʁ • ACIDτϥϯβΫγϣϯແ͍ͷʹେৎ෉ͳͷʁ • ݁Ռ੔߹ੑͬͯେৎ෉ͳͷʁ • ্ख͘ੑೳΛҾ͖ग़͢ʹ͸Ͳ͏͢Ε͹ྑ͍ʁ DynamoDBͷΑ͋͘Δෆ҆ ෼͔Βͳ͍͔Βෆ҆ʹͳΔ

  12. • ͋͑ͯ෼ྨ͢ΔͳΒ෼ࢄKVSͱ͍͏δϟϯϧͷNoSQL • KeyʹΑΔ׬શҰகͱIndex୳ࡧ͕Մೳ • ϑϧϚωʔδυ • ແ੍ݶͷࣗಈ֦ுετϨʔδʢ࢖ͬͨ෼͚ͩʣ • Πϯελϯεͷ؅ཧෆཁɺWrite/Readͷੑೳ͚ͩΛࢦఆ͢Δ

    DynamoDBͷجຊ
  13. DynamoDB͸ͲΜͳ ”AWSαʔϏε” ͔ Ͱ͸ͳ͘

  14. DynamoDB͸ͲΜͳ ”σʔλϕʔε” ͔ ஌ͬͯ·͔͢ʁ

  15. • Consistent Hashing • Vector clocks with reconciliation during reads

    • Sloppy Quorum and hinted handoff • Anti-entropy using Merkle trees • Gossip-based membership protocol and failure detection Amazon’s Dynamo ৄ͘͠஌Γ͍ͨͳΒʢBetterͰ͋ͬͯMustͰ͸ͳ͍ʣ https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
  16. • Ωʔͷϋογϡ஋Ͱ෼ࢄ • όοΫάϥϯυͰߋ৽ΛޙউͪͰಉظ • ݁Ռ੔߹ੑϕʔε • ॻ͖ࠐΈɾಡΈࠐΈͷQuorum਺Ͱ੔߹ੑ͕ܾ·Δ • ΩʔΛࢦఆͨ͠ಡΈࠐΈ͸ڧ੔߹ੑ͕બ୒Մೳ

    ཈͑Δ΂͖ϙΠϯτ
  17. Partition Keyͷϋογϡ஋Ͱ෼ࢄ Like Hash Table(Index)

  18. • Partition Keyʹ࿈൪Λ࢖͏ͷ͸Կͷҙຯ΋ͳ͍͜ͱ͕෼͔Δ • ࿈൪Χ΢ϯλʔΛ࡞ΔͱPartition͕ภΔ • Max + 1ͳΜͯҎͯͷ֎ •

    ࿦ཧతʹҰҙͱͳΔଐੑ͕͋ΔͳΒͦΕΛ࢖͏ • ͳ͚Ε͹UUIDͷΑ͏ͳিಥՄೳੑ͕ۃখͳϥϯμϜ஋Ͱྑ͍ • Sort΋Ͱ͖ͳ͍ͷͰSnowflakeͷΑ͏ͳॱংੑ΋ཁΒͳ͍ • User ID͸CognitoͰೝূ͍ͯ͠ΔͳΒCognito͕ൃߦ͢ΔIDΛ࢖͏ • ΞΫηε੍ޚʹ΋࢖͑Δ ͜ͷલఏΛ౿·͑Δͱ
  19. • 3000RCU, 1000WCU, Index Size 10GB (2018-10-31ݱࡏ) • ͲΕ͔ҰͭͰ΋௒ա͢ΔͱPartition͕෼ׂ͞ΕΔ •

    ෼ׂ͞ΕΔͱ֬อͨ͠Ϣχοτ͕ۉ౳ʹׂΓৼΒΕΔ • εϩοτϦϯά࣌ͷҰ࣌తͳόʔετ͕͋Δ • ݶఆతͳόʔετ • ༨৒෼Λճ͢ (Adaptive Capacity) Partition͝ͱͷΩϟύγςΟ
  20. B+TREE INDEX B-treeͳͷ͔B+treeͳͷ͔͸ެද͞Ε͍ͯͳ͍͚Ͳ Range Keyͱ͔ݺͿ͘Β͍͔ͩΒ…Ͷʁ

  21. TARGETING QUERY B+treeͷߏ଄Λ஌͍ͬͯΕ͹ ૊Έ߹ΘͤΩʔͷΠϯσοΫε͕ແͯ͘΋ ͜͏΍Ε͹ྑ͍͜ͱ͕෼͔Δ

  22. Partition KeyͰ෼ࢄͨ͠தͰSort KeyͰ੔ྻ ͜ͷΠϝʔδΛ࣋ͭ͜ͱ͕ͱͯ΋ॏཁ

  23. • Partition KeyͰ෼ࢄͨ͠ΠϯελϯεʹIndex͕ߏங͞ΕΔ • ҰͭͷΠϯελϯε಺Ͱ͸ಉظతʹIndex΋ߋ৽͞ΕΔ • ڧ੔߹ੑ͕બ୒Մೳ • Primary Sort

    KeyͷϦʔϑϊʔυʹ࣮σʔλͷΞυϨε͕
 ֨ೲ͞Ε͍ͯΔΠϝʔδʢͱࢥΘΕΔʣ • LSI͸࣮σʔλΛ࣋ͨͳ͍ͷͰॏෳՄೳ Primary Sort Key, LSI
  24. GSI͸࣮ςʔϒϧͷࣹӨ

  25. • ू߹࿦ʹ͓͚ΔࣹӨ • GSIͷPartition KeyΛݩʹ෼ࢄ
 ʢߋ৽͸ඇಉظͰద༻ʣ • ϦʔϑϊʔυʹଐੑͱPrimary Key
 ʢৄ͘͠͸ޙड़ʣ

    • ιʔτࡁΈͷσʔλͷ෦෼ෳ੡
 ʢҟ࿦͸ೝΊΔʣ ࣹӨ ࡶͳMySQLύϑΥʔϚϯενϡʔχϯά from yoku0825 https://www.slideshare.net/yoku0825/mysql-57449062/62
  26. • KEYS_ONLY: Primary Keyͷ஋ͷΈ • INCLUDE: Primary Keyͷ஋ + બ୒ͨ͠ଐੑ

    • ALL: શͯͷଐੑ • GSIʹؚΊͨଐੑ͸Primary KeyΛҾ͖௚͢ඞཁ͕ͳ͘ͳΔ • ಡΈࠐΈͷޮ཰Խ • ߋ৽ίετͱͷτϨʔυΦϑ • MySQL΍PostgreSQLͰݴ͏Covering Index / Index Only ScanͷΠϝʔδ GSI͕ࣹӨ͢ΔଐੑͷΦϓγϣϯ
  27. • ϝΠϯςʔϒϧͱ͸ผͷPartitionͰ෼ࢄ͞ΕΔ • ߋ৽͸ඇಉظͰ൓ө • ݁Ռ੔߹ੑͬͯେৎ෉ͳͷʁ • RDSͩͬͯRead ReplicaಡΜͩΒ݁Ռ੔߹ੑͰ͢Αʁʁ •

    GSI΁ͷΫΤϦ͸͍͍ͩͨRead Replicaʹ౤͛ΔΑ͏ͳΫΤϦ GSIʹର͢ΔΫΤϦ͸݁Ռ੔߹ੑ
  28. ࢓૊ΈΛ஌Ε͹࣮ʹγϯϓϧ શ෦RDBͰطʹ͋ͬͨ࢓૊ΈͰ͢Αʁʁ

  29. ☓ DynamoDB͸บ͕͋Δ ◦ RDBͷบ͕͋Γ͗͢Δ →γϯϓϧ͗͢ΔDynamoDBͱͷΪϟοϓ͕ڧͯͦ͘͏ײ͡Δ͚ͩ ɹRDBʹચ೴͞Ε͍ͯΔΜ͡Όͳ͍Ͱ͔͢ʁʁ

  30. DynamoDBͷσʔλϞσϦϯάͱ͸ ͭ·Δͱ͜Ζ

  31. σʔλͷ഑ஔΛܾΊΔ͜ͱ ҟ࿦͸ೝΊΔʂʂ

  32. σʔλϞσϦϯάͷߟ͑ํͷϙΠϯτ ഑ஔΛܾΊΔͨΊͷ

  33. • εΩʔϚϨεͷҙຯΛߟ͑Δ • List, Map͕࣋ͯΔ͜ͱʁ • ޙ͔ΒଐੑΛγʔϜϨεʹ௥ՃͰ͖Δ͜ͱʁ • A. ͳΜͰ΋ಥͬࠐΊΔ͜ͱ

    • Partition KeyͰ෼ࢄ • ςʔϒϧΛ෼͚ͳͯ͘΋ෛՙ͸෼ࢄ͢Δ • ۃ࿦͢Δͱ෼͚Δҙຯ͕ͳ͍ɺ෼͚ͳ͍ํ͕ΩϟύγςΟ͕؅ཧ͠΍͍͢ • “ઃܭ͕༏ΕͨΞϓϦέʔγϣϯͰ͸ɺඞཁͳςʔϒϧ͸ 1 ͭͷΈͰ͢” • GSIͷݸ਺੍ݶ(5ݸ)ͳͲ΋͋ΔͷͰۀ຿υϝΠϯຖʹҰͭ͘Β͍ͷؾ࣋ͪ ςʔϒϧͷ෼͚ํ
  34. • RDB • ·ͣεΩʔϚΛߟ͑Δ • ਖ਼نԽ • ͦΕʹରͯ͠Ͳ͏ΞΫηε͢Δ͔(SQL)ߟ͑Δ • DynamoDB

    • εΩʔϚϨε • ඇਖ਼نԽ • ·ͣΞΫηεΛߟ͑ͯɺͦΕʹ߹ΘͤͨσʔλΛ࡞Δ • ΞϓϦέʔγϣϯͱσʔλϞσϧ͸ಉ࣌ʹઃܭ͢Δ RDBͱ͸Ξϓϩʔν͕શ͘ҟͳΔ
  35. ACIDτϥϯβΫγϣϯ͕ແ͍ͷ͸Ͳ͏͢Δͷʁ ͱ͸͍͑

  36. • ݪࢠੑ (Atomicity) • Ұ؏(੔߹)ੑ (Consistency) • ಠཱੑ (Isolation) •

    Ӭଓੑ (Durability) ͦ΋ͦ΋ACIDͬͯͳΜ͚ͩͬʁ ͜ΕΒΛ୲อ͢Δ͜ͱ͕໨తͰ͋ͬͯ τϥϯβΫγϣϯγεςϜͰ͋Δඞཁ͸ແ͍
  37. • RDB • ਖ਼نԽ͍ͯ͘͠ͱςʔϒϧͷ਺͸جຊతʹ૿͑Δ • ࿦ཧతʹ෼͔Εͨςʔϒϧ(Ϩίʔυ)ؒͷ੔߹ੑΛऔΔ࢓૊Έ͕͋Δ • ϦϨʔγϣϯγοϓ • ֎෦Ωʔ੍໿

    • ACIDτϥϯβΫγϣϯ • DynamoDB • ඇਖ਼نԽͯ͠1ͭͷ࣮ମʹ͍ͭͯ1ͭͷΞΠςϜʹऩΊΔ • 1ΞΠςϜͷߋ৽͸ΞτϛοΫ ACIDͷ୲อ
  38. • PutItem • ΞΠςϜΛ·Δ͝ͱߋ৽͢Δɺແ͔ͬͨΒ࡞Δ • Get -> ॻ͖׵͑ -> Putͩͱಠཱੑ่͕ΕΔ

    • UpdateItem • ΞΠςϜΛ෦෼ߋ৽ɺແ͔ͬͨΒ࡞Δ • List, Mapͷ෦෼௥Ճɺ࡟আͳͲ΋Մೳ • ଐੑͷ஋Λར༻ͨ͠ߋ৽΋Մೳʢ͋Δଐੑʹ+1͢Δ౳) • ଐੑͷ஋Λར༻ͨ͠৚݅෇͖ॻ͖ࠐΈ DynamoDBͷߋ৽ૢ࡞
  39. ඇਖ਼نԽ͢ΔͱಡΈࠐΈ͕ͭΒ͍ ϑϧεΩϟϯ͢Δ͔͠ͳ͍ શ෦GSIషΔͷʁ

  40. CQRSͰ͢Α ͦ͜Ͱ

  41. • ॻ͖ࠐΉσʔλͱಡΈࠐΉσʔλ͸ಉ͡Ͱ͋Δඞཁ͸ແ͍ • ݁Ռ੔߹ੑΛड͚ೖΕͯඇಉظͰಡΈࠐΈσʔλΛ࡞ΕΔ • Commandͷ׬ྃΛ΋ͬͯQuery༻σʔλΛ࡞ΕΔ • DynamoDB Streams͕Ϛον͢Δ •

    CQRSΛૉ௚ʹ࣮૷͢Δͱ·ͣKinesis Streams͚ͩͲ • Materialized View CQRS (Command Query Responsibility Segregation)
  42. ARCHITECTURE PATTERN AppSync͕͜Ε͔ΒॏཁͳҐஔΛ઎ΊΔ͸ͣ

  43. DATA ATOMICITY MicroserviesతͳΠϝʔδͰ͸͜ͷݪ୯Ґ͕ॏཁ σʔλετΞɾςʔϒϧͷ෼͚ํͰ͸ͳ͍ ލ͕͍ͤͨͳΒMaterialized ViewΛ࡞Δ User Profile User Status

    Name Email Read/Write Search Read/Write
  44. ͜ΜͳΠϝʔδ 1ͭͷGSIͰ༷ʑͳݕࡧ͕Ͱ͖Δ

  45. • ෳ਺ͷ࣮ମʹލΔτϥϯβΫγϣϯ • DynamoDB୯ମͩͱ͜͜͸Ͳ͏ͯ͠΋ͭΒ͍ • τϥϯβΫγϣϯ෼཭Ϩϕϧͷ࠷ߴ͸Selializable • ͭ·Γ௚ྻԽ • Kinesis

    StreamsͰ௚ྻԽ • ίέΔ͔΋͠Εͳ͍ͷͰႈ౳ʹ࣮૷͢Δ • ৚݅෇ॻ͖ࠐΈͰطʹߦͬͨॲཧΛSkip͢Δͷ΋ႈ౳ • ͱ͍͏͔ɺKinesis StreamsͰಉ࣌઀ଓ਺Λίϯτϩʔϧ͢Ε͹RDS࢖ͬͨͬͯྑ͍Θ͚Ͱ… ͱ͸͍͑ݶք͸͋Δ
  46. ADVANCED TIPS ͍͔ͭ͘͝঺հ

  47. • Partition Keyͷύλʔϯ͕গͳ͍৔߹ • ෼ࢄ͠ͳ͘ͳͬͯ͠·͏ • 1PartitionͷΩϟύγςΟΛӽ͑ͦ͏ͳΒγϟʔσΟϯά͢Δ • KeyҎ֎ͷࣗ໌ͳଐੑ͔ΒܭࢉՄೳͳSuffixΛ෇༩͢Δ •

    ूܭ΍ൣғݕࡧͷඞཁ͕ͳ͚Ε͹Hash • ඞཁ͕͋Ε͹Mod౳ GSIͷγϟʔσΟϯά
  48. ͜ΜͳΠϝʔδ Unicode PointΛ߹ܭͯ͠200Ͱ ׂͬͨ༨ΓΛ࢖༻͍ͯ͠Δ

  49. • DynamoDBͷIndex͸Sparse • Indexଐੑ͕ଘࡏ͠ͳ͍৔߹ɺIndexʹͦͷσʔλ͸ؚ·ͳ͍ • ಛఆͷΞΠςϜͷΈʹଘࡏ͢ΔଐੑΛࢦఆ͢Δ͜ͱͰ
 Index༰ྔ͕େ͖͘ѹॖͰ͖Δ Sparse Index

  50. Sort΋ཁΒͳ͍ GSIʹ͸҉໧ͰPrimary Keyؚ͕·ΕΔ

  51. • Α͋͘Δ֊૚ߏ଄Λ࣋ͭσʔλ • ૊৫ߏ଄ • ωετͨ͠ΧςΰϥΠζ • ͳͲͳͲ • ݕࡧΛߟ͑ͳ͚Ε͹֊૚ߏ଄Λ·Δͬͱ1ΞΠςϜʹԡ͠ࠐΉ

    • Sort KeyͰ֊૚Λදݱ͢Δ Hierarchical data
  52. Sort KeyͰલํҰகݕࡧ ͋ɺ͜ΕSQLΞϯνύλʔϯͰʢ͈́ ঎඼ઃܭ͕ࡶͳͷ͸͝Ѫᇷ

  53. “DynamoDB ދͷר” [ݕࡧ] http://marcy.hatenablog.com/entry/2018/07/31/213705 ҰԠະ׬ͳͷͰ·ͩॻ͔͘΋

  54. “LET’S BECOME THE MASTER OF DYNAMODB” THANKS!! “TO BECOME THE

    MASTER OF SERVERLESS”