Serverlessを極めるためにDynamoDBデータモデリングを極めよう / Let’s become the master of DynamoDB Data Modeling to become the master of Serverless

AWS Dev Day Tokyo 2018の登壇資料です
https://aws.amazon.com/jp/aws-devday-tokyo-2018/

Transcript

1. LET’S BECOME THE MASTER
   OF DYNAMODB DATA MODELING
   TO BECOME THE MASTER OF SERVERLESS
   MASASHI TERUI @ AWS DEV DAY TOKYO 2018
   

   View Slide

2. SERVERWORKS CO.,LTD.
   + FREELANCER
   • Serverless Oji-san
   • Serverless Framework Plugin Developer
   • Serverlessconf Tokyo 2016,2017,2018 speaker
   • Remote worker (in Sapporo-shi, Hokkaido)
   • http://marcy.hatenablog.com/entry/2018/08/04/231241
   MASASHI TERUI
   ARCHITECT / DEVELOPER
   

   View Slide

3. SERVERLESSͳΜͯ০ΓͰ͢
   Ғ͍ਓʹ͸ͦΕ͕Θ͔ΒΜͷͰ͢Α
   ͜͜͸Ұ଍ઌʹ(উखʹ)DatabaseτϥοΫͰ͢ʂʂ
   ⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠
   ͜ͷηογϣϯ͸
   SERVERLESSཁૉ͕
   ΄ͱΜͲ͋Γ·ͤΜʂʂ
   ⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠
   

   View Slide

4. SERVERLESS΍ͬͯ·͔͢ʁ
   DynamoDB࢖ͬͯ·͔͢ʁ
   

   View Slide

5. • ޷͖
   • ී௨
   • ݏ͍
   • LambdaͱRDSͷ૬ੑ͕ѱ͍͔Β࢓ํͳ͘࢖͍ͬͯΔ
   DynamoDB͸޷͖Ͱ͔͢ʁ
   

   View Slide

6. • ίωΫγϣϯϞσϧ
   • Ӭଓత
   • ηΩϡϦςΟ
   • ૊ΈࠐΈͷऑ͍ೝূ
   • VPC಺΁ͷΞΫηείετʢENIੜ੒ॲཧʣ
   LambdaͱRDSͷ૬ੑ͕ѱ͍ʁ
   ͱͯ΋OLTPͰ͸࢖͑ͳ͍(਺ඵʙे਺ඵ)
   

   View Slide

7. ࢓ํ͕ͳ͍͔ΒDynamoDB……
   ຊ౰͸RDBΛ࢖͍͍͚ͨͲ
   

   View Slide

8. DynamoDBΛੵۃతʹ࢖͓͏
   ͦΜͳϚΠϯυͰେৎ෉ʁ
   

   View Slide

9. • ୺తʹݴͬͯόΠϒϧ
   • ͜ΕΛಡΜͰશͯཧղͰ͖ͨਓ͸

   ͨͿΜ͜ͷઌΛฉ͘ඞཁ͸͋Γ·ͤΜʂʂ
   • https://docs.aws.amazon.com/ja_jp/amazondynamodb/latest/
   developerguide/best-practices.html
   DynamoDB Best Practices
   

   View Slide

10. • ςʔϒϧઃܭͱ͔ΠϯσοΫεઃܭͬͯͲ͏΍Δͷʁ
    • ACIDτϥϯβΫγϣϯແ͍ͷʹେৎ෉ͳͷʁ
    • ݁Ռ੔߹ੑͬͯେৎ෉ͳͷʁ
    • ্ख͘ੑೳΛҾ͖ग़͢ʹ͸Ͳ͏͢Ε͹ྑ͍ʁ
    DynamoDBͷΑ͋͘Δෆ҆
    

    View Slide

11. • ςʔϒϧઃܭͱ͔ΠϯσοΫεઃܭͬͯͲ͏΍Δͷʁ
    • ACIDτϥϯβΫγϣϯແ͍ͷʹେৎ෉ͳͷʁ
    • ݁Ռ੔߹ੑͬͯେৎ෉ͳͷʁ
    • ্ख͘ੑೳΛҾ͖ग़͢ʹ͸Ͳ͏͢Ε͹ྑ͍ʁ
    DynamoDBͷΑ͋͘Δෆ҆
    ෼͔Βͳ͍͔Βෆ҆ʹͳΔ
    

    View Slide

12. • ͋͑ͯ෼ྨ͢ΔͳΒ෼ࢄKVSͱ͍͏δϟϯϧͷNoSQL
    • KeyʹΑΔ׬શҰகͱIndex୳ࡧ͕Մೳ
    • ϑϧϚωʔδυ
    • ແ੍ݶͷࣗಈ֦ுετϨʔδʢ࢖ͬͨ෼͚ͩʣ
    • Πϯελϯεͷ؅ཧෆཁɺWrite/Readͷੑೳ͚ͩΛࢦఆ͢Δ
    DynamoDBͷجຊ
    

    View Slide

13. DynamoDB͸ͲΜͳ ”AWSαʔϏε” ͔
    Ͱ͸ͳ͘
    

    View Slide

14. DynamoDB͸ͲΜͳ ”σʔλϕʔε” ͔
    ஌ͬͯ·͔͢ʁ
    

    View Slide

15. • Consistent Hashing
    • Vector clocks with reconciliation during reads
    • Sloppy Quorum and hinted handoff
    • Anti-entropy using Merkle trees
    • Gossip-based membership protocol and failure detection
    Amazon’s Dynamo
    ৄ͘͠஌Γ͍ͨͳΒʢBetterͰ͋ͬͯMustͰ͸ͳ͍ʣ
    https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
    

    View Slide

16. • Ωʔͷϋογϡ஋Ͱ෼ࢄ
    • όοΫάϥϯυͰߋ৽ΛޙউͪͰಉظ
    • ݁Ռ੔߹ੑϕʔε
    • ॻ͖ࠐΈɾಡΈࠐΈͷQuorum਺Ͱ੔߹ੑ͕ܾ·Δ
    • ΩʔΛࢦఆͨ͠ಡΈࠐΈ͸ڧ੔߹ੑ͕બ୒Մೳ
    ཈͑Δ΂͖ϙΠϯτ
    

    View Slide

17. Partition Keyͷϋογϡ஋Ͱ෼ࢄ
    Like Hash Table(Index)
    

    View Slide

18. • Partition Keyʹ࿈൪Λ࢖͏ͷ͸Կͷҙຯ΋ͳ͍͜ͱ͕෼͔Δ
    • ࿈൪Χ΢ϯλʔΛ࡞ΔͱPartition͕ภΔ
    • Max + 1ͳΜͯҎͯͷ֎
    • ࿦ཧతʹҰҙͱͳΔଐੑ͕͋ΔͳΒͦΕΛ࢖͏
    • ͳ͚Ε͹UUIDͷΑ͏ͳিಥՄೳੑ͕ۃখͳϥϯμϜ஋Ͱྑ͍
    • Sort΋Ͱ͖ͳ͍ͷͰSnowflakeͷΑ͏ͳॱংੑ΋ཁΒͳ͍
    • User ID͸CognitoͰೝূ͍ͯ͠ΔͳΒCognito͕ൃߦ͢ΔIDΛ࢖͏
    • ΞΫηε੍ޚʹ΋࢖͑Δ
    ͜ͷલఏΛ౿·͑Δͱ
    

    View Slide

19. • 3000RCU, 1000WCU, Index Size 10GB (2018-10-31ݱࡏ)
    • ͲΕ͔ҰͭͰ΋௒ա͢ΔͱPartition͕෼ׂ͞ΕΔ
    • ෼ׂ͞ΕΔͱ֬อͨ͠Ϣχοτ͕ۉ౳ʹׂΓৼΒΕΔ
    • εϩοτϦϯά࣌ͷҰ࣌తͳόʔετ͕͋Δ
    • ݶఆతͳόʔετ
    • ༨৒෼Λճ͢ (Adaptive Capacity)
    Partition͝ͱͷΩϟύγςΟ
    

    View Slide

20. B+TREE INDEX B-treeͳͷ͔B+treeͳͷ͔͸ެද͞Ε͍ͯͳ͍͚Ͳ
    Range Keyͱ͔ݺͿ͘Β͍͔ͩΒ…Ͷʁ
    

    View Slide

21. TARGETING QUERY B+treeͷߏ଄Λ஌͍ͬͯΕ͹
    ૊Έ߹ΘͤΩʔͷΠϯσοΫε͕ແͯ͘΋
    ͜͏΍Ε͹ྑ͍͜ͱ͕෼͔Δ
    

    View Slide

22. Partition KeyͰ෼ࢄͨ͠தͰSort KeyͰ੔ྻ
    ͜ͷΠϝʔδΛ࣋ͭ͜ͱ͕ͱͯ΋ॏཁ
    

    View Slide

23. • Partition KeyͰ෼ࢄͨ͠ΠϯελϯεʹIndex͕ߏங͞ΕΔ
    • ҰͭͷΠϯελϯε಺Ͱ͸ಉظతʹIndex΋ߋ৽͞ΕΔ
    • ڧ੔߹ੑ͕બ୒Մೳ
    • Primary Sort KeyͷϦʔϑϊʔυʹ࣮σʔλͷΞυϨε͕

    ֨ೲ͞Ε͍ͯΔΠϝʔδʢͱࢥΘΕΔʣ
    • LSI͸࣮σʔλΛ࣋ͨͳ͍ͷͰॏෳՄೳ
    Primary Sort Key, LSI
    

    View Slide

24. GSI͸࣮ςʔϒϧͷࣹӨ
    

    View Slide

25. • ू߹࿦ʹ͓͚ΔࣹӨ
    • GSIͷPartition KeyΛݩʹ෼ࢄ

    ʢߋ৽͸ඇಉظͰద༻ʣ
    • ϦʔϑϊʔυʹଐੑͱPrimary Key

    ʢৄ͘͠͸ޙड़ʣ
    • ιʔτࡁΈͷσʔλͷ෦෼ෳ੡

    ʢҟ࿦͸ೝΊΔʣ
    ࣹӨ
    ࡶͳMySQLύϑΥʔϚϯενϡʔχϯά from yoku0825
    https://www.slideshare.net/yoku0825/mysql-57449062/62
    

    View Slide

26. • KEYS_ONLY: Primary Keyͷ஋ͷΈ
    • INCLUDE: Primary Keyͷ஋ + બ୒ͨ͠ଐੑ
    • ALL: શͯͷଐੑ
    • GSIʹؚΊͨଐੑ͸Primary KeyΛҾ͖௚͢ඞཁ͕ͳ͘ͳΔ
    • ಡΈࠐΈͷޮ཰Խ
    • ߋ৽ίετͱͷτϨʔυΦϑ
    • MySQL΍PostgreSQLͰݴ͏Covering Index / Index Only ScanͷΠϝʔδ
    GSI͕ࣹӨ͢ΔଐੑͷΦϓγϣϯ
    

    View Slide

27. • ϝΠϯςʔϒϧͱ͸ผͷPartitionͰ෼ࢄ͞ΕΔ
    • ߋ৽͸ඇಉظͰ൓ө
    • ݁Ռ੔߹ੑͬͯେৎ෉ͳͷʁ
    • RDSͩͬͯRead ReplicaಡΜͩΒ݁Ռ੔߹ੑͰ͢Αʁʁ
    • GSI΁ͷΫΤϦ͸͍͍ͩͨRead Replicaʹ౤͛ΔΑ͏ͳΫΤϦ
    GSIʹର͢ΔΫΤϦ͸݁Ռ੔߹ੑ
    

    View Slide

28. ࢓૊ΈΛ஌Ε͹࣮ʹγϯϓϧ
    શ෦RDBͰطʹ͋ͬͨ࢓૊ΈͰ͢Αʁʁ
    

    View Slide

29. ☓ DynamoDB͸บ͕͋Δ
    ○ RDBͷบ͕͋Γ͗͢Δ
    →γϯϓϧ͗͢ΔDynamoDBͱͷΪϟοϓ͕ڧͯͦ͘͏ײ͡Δ͚ͩ
    ɹRDBʹચ೴͞Ε͍ͯΔΜ͡Όͳ͍Ͱ͔͢ʁʁ
    

    View Slide

30. DynamoDBͷσʔλϞσϦϯάͱ͸
    ͭ·Δͱ͜Ζ
    

    View Slide

31. σʔλͷ഑ஔΛܾΊΔ͜ͱ
    ҟ࿦͸ೝΊΔʂʂ
    

    View Slide

32. σʔλϞσϦϯάͷߟ͑ํͷϙΠϯτ
    ഑ஔΛܾΊΔͨΊͷ
    

    View Slide

33. • εΩʔϚϨεͷҙຯΛߟ͑Δ
    • List, Map͕࣋ͯΔ͜ͱʁ
    • ޙ͔ΒଐੑΛγʔϜϨεʹ௥ՃͰ͖Δ͜ͱʁ
    • A. ͳΜͰ΋ಥͬࠐΊΔ͜ͱ
    • Partition KeyͰ෼ࢄ
    • ςʔϒϧΛ෼͚ͳͯ͘΋ෛՙ͸෼ࢄ͢Δ
    • ۃ࿦͢Δͱ෼͚Δҙຯ͕ͳ͍ɺ෼͚ͳ͍ํ͕ΩϟύγςΟ͕؅ཧ͠΍͍͢
    • “ઃܭ͕༏ΕͨΞϓϦέʔγϣϯͰ͸ɺඞཁͳςʔϒϧ͸ 1 ͭͷΈͰ͢”
    • GSIͷݸ਺੍ݶ(5ݸ)ͳͲ΋͋ΔͷͰۀ຿υϝΠϯຖʹҰͭ͘Β͍ͷؾ࣋ͪ
    ςʔϒϧͷ෼͚ํ
    

    View Slide

34. • RDB
    • ·ͣεΩʔϚΛߟ͑Δ
    • ਖ਼نԽ
    • ͦΕʹରͯ͠Ͳ͏ΞΫηε͢Δ͔(SQL)ߟ͑Δ
    • DynamoDB
    • εΩʔϚϨε
    • ඇਖ਼نԽ
    • ·ͣΞΫηεΛߟ͑ͯɺͦΕʹ߹ΘͤͨσʔλΛ࡞Δ
    • ΞϓϦέʔγϣϯͱσʔλϞσϧ͸ಉ࣌ʹઃܭ͢Δ
    RDBͱ͸Ξϓϩʔν͕શ͘ҟͳΔ
    

    View Slide

35. ACIDτϥϯβΫγϣϯ͕ແ͍ͷ͸Ͳ͏͢Δͷʁ
    ͱ͸͍͑
    

    View Slide

36. • ݪࢠੑ (Atomicity)
    • Ұ؏(੔߹)ੑ (Consistency)
    • ಠཱੑ (Isolation)
    • Ӭଓੑ (Durability)
    ͦ΋ͦ΋ACIDͬͯͳΜ͚ͩͬʁ
    ͜ΕΒΛ୲อ͢Δ͜ͱ͕໨తͰ͋ͬͯ
    τϥϯβΫγϣϯγεςϜͰ͋Δඞཁ͸ແ͍
    

    View Slide

37. • RDB
    • ਖ਼نԽ͍ͯ͘͠ͱςʔϒϧͷ਺͸جຊతʹ૿͑Δ
    • ࿦ཧతʹ෼͔Εͨςʔϒϧ(Ϩίʔυ)ؒͷ੔߹ੑΛऔΔ࢓૊Έ͕͋Δ
    • ϦϨʔγϣϯγοϓ
    • ֎෦Ωʔ੍໿
    • ACIDτϥϯβΫγϣϯ
    • DynamoDB
    • ඇਖ਼نԽͯ͠1ͭͷ࣮ମʹ͍ͭͯ1ͭͷΞΠςϜʹऩΊΔ
    • 1ΞΠςϜͷߋ৽͸ΞτϛοΫ
    ACIDͷ୲อ
    

    View Slide

38. • PutItem
    • ΞΠςϜΛ·Δ͝ͱߋ৽͢Δɺແ͔ͬͨΒ࡞Δ
    • Get -> ॻ͖׵͑ -> Putͩͱಠཱੑ่͕ΕΔ
    • UpdateItem
    • ΞΠςϜΛ෦෼ߋ৽ɺແ͔ͬͨΒ࡞Δ
    • List, Mapͷ෦෼௥Ճɺ࡟আͳͲ΋Մೳ
    • ଐੑͷ஋Λར༻ͨ͠ߋ৽΋Մೳʢ͋Δଐੑʹ+1͢Δ౳)
    • ଐੑͷ஋Λར༻ͨ͠৚݅෇͖ॻ͖ࠐΈ
    DynamoDBͷߋ৽ૢ࡞
    

    View Slide

39. ඇਖ਼نԽ͢ΔͱಡΈࠐΈ͕ͭΒ͍
    ϑϧεΩϟϯ͢Δ͔͠ͳ͍
    શ෦GSIషΔͷʁ
    

    View Slide

40. CQRSͰ͢Α
    ͦ͜Ͱ
    

    View Slide

41. • ॻ͖ࠐΉσʔλͱಡΈࠐΉσʔλ͸ಉ͡Ͱ͋Δඞཁ͸ແ͍
    • ݁Ռ੔߹ੑΛड͚ೖΕͯඇಉظͰಡΈࠐΈσʔλΛ࡞ΕΔ
    • Commandͷ׬ྃΛ΋ͬͯQuery༻σʔλΛ࡞ΕΔ
    • DynamoDB Streams͕Ϛον͢Δ
    • CQRSΛૉ௚ʹ࣮૷͢Δͱ·ͣKinesis Streams͚ͩͲ
    • Materialized View
    CQRS (Command Query Responsibility Segregation)
    

    View Slide

42. ARCHITECTURE PATTERN
    AppSync͕͜Ε͔ΒॏཁͳҐஔΛ઎ΊΔ͸ͣ
    

    View Slide

43. DATA ATOMICITY MicroserviesతͳΠϝʔδͰ͸͜ͷݪ୯Ґ͕ॏཁ
    σʔλετΞɾςʔϒϧͷ෼͚ํͰ͸ͳ͍
    ލ͕͍ͤͨͳΒMaterialized ViewΛ࡞Δ
    User
    Profile
    User
    Status
    Name Email
    Read/Write
    Search
    Read/Write
    

    View Slide

44. ͜ΜͳΠϝʔδ 1ͭͷGSIͰ༷ʑͳݕࡧ͕Ͱ͖Δ
    

    View Slide

45. • ෳ਺ͷ࣮ମʹލΔτϥϯβΫγϣϯ
    • DynamoDB୯ମͩͱ͜͜͸Ͳ͏ͯ͠΋ͭΒ͍
    • τϥϯβΫγϣϯ෼཭Ϩϕϧͷ࠷ߴ͸Selializable
    • ͭ·Γ௚ྻԽ
    • Kinesis StreamsͰ௚ྻԽ
    • ίέΔ͔΋͠Εͳ͍ͷͰႈ౳ʹ࣮૷͢Δ
    • ৚݅෇ॻ͖ࠐΈͰطʹߦͬͨॲཧΛSkip͢Δͷ΋ႈ౳
    • ͱ͍͏͔ɺKinesis StreamsͰಉ࣌઀ଓ਺Λίϯτϩʔϧ͢Ε͹RDS࢖ͬͨͬͯྑ͍Θ͚Ͱ…
    ͱ͸͍͑ݶք͸͋Δ
    

    View Slide

46. ADVANCED TIPS
    ͍͔ͭ͘͝঺հ
    

    View Slide

47. • Partition Keyͷύλʔϯ͕গͳ͍৔߹
    • ෼ࢄ͠ͳ͘ͳͬͯ͠·͏
    • 1PartitionͷΩϟύγςΟΛӽ͑ͦ͏ͳΒγϟʔσΟϯά͢Δ
    • KeyҎ֎ͷࣗ໌ͳଐੑ͔ΒܭࢉՄೳͳSuffixΛ෇༩͢Δ
    • ूܭ΍ൣғݕࡧͷඞཁ͕ͳ͚Ε͹Hash
    • ඞཁ͕͋Ε͹Mod౳
    GSIͷγϟʔσΟϯά
    

    View Slide

48. ͜ΜͳΠϝʔδ Unicode PointΛ߹ܭͯ͠200Ͱ
    ׂͬͨ༨ΓΛ࢖༻͍ͯ͠Δ
    

    View Slide

49. • DynamoDBͷIndex͸Sparse
    • Indexଐੑ͕ଘࡏ͠ͳ͍৔߹ɺIndexʹͦͷσʔλ͸ؚ·ͳ͍
    • ಛఆͷΞΠςϜͷΈʹଘࡏ͢ΔଐੑΛࢦఆ͢Δ͜ͱͰ

    Index༰ྔ͕େ͖͘ѹॖͰ͖Δ
    Sparse Index
    

    View Slide

50. Sort΋ཁΒͳ͍ GSIʹ͸҉໧ͰPrimary Keyؚ͕·ΕΔ
    

    View Slide

51. • Α͋͘Δ֊૚ߏ଄Λ࣋ͭσʔλ
    • ૊৫ߏ଄
    • ωετͨ͠ΧςΰϥΠζ
    • ͳͲͳͲ
    • ݕࡧΛߟ͑ͳ͚Ε͹֊૚ߏ଄Λ·Δͬͱ1ΞΠςϜʹԡ͠ࠐΉ
    • Sort KeyͰ֊૚Λදݱ͢Δ
    Hierarchical data
    

    View Slide

52. Sort KeyͰલํҰகݕࡧ ͋ɺ͜ΕSQLΞϯνύλʔϯͰʢ͈́
    ঎඼ઃܭ͕ࡶͳͷ͸͝Ѫᇷ
    

    View Slide

53. “DynamoDB ދͷר” [ݕࡧ]
    http://marcy.hatenablog.com/entry/2018/07/31/213705
    ҰԠະ׬ͳͷͰ·ͩॻ͔͘΋
    

    View Slide

54. “LET’S BECOME THE MASTER OF DYNAMODB”
    THANKS!!
    “TO BECOME THE MASTER OF SERVERLESS”
    

    View Slide