Serverlessを極めるためにDynamoDBデータモデリングを極めよう / Let’s become the master of DynamoDB Data Modeling to become the master of Serverless

Transcript

1. LET’S BECOME THE MASTER OF DYNAMODB DATA MODELING TO BECOME

   THE MASTER OF SERVERLESS MASASHI TERUI @ AWS DEV DAY TOKYO 2018

2. SERVERWORKS CO.,LTD. + FREELANCER • Serverless Oji-san • Serverless Framework

   Plugin Developer • Serverlessconf Tokyo 2016,2017,2018 speaker • Remote worker (in Sapporo-shi, Hokkaido) • http://marcy.hatenablog.com/entry/2018/08/04/231241 MASASHI TERUI ARCHITECT / DEVELOPER

3. SERVERLESSͳΜͯ০ΓͰ͢ Ғ͍ਓʹ͸ͦΕ͕Θ͔ΒΜͷͰ͢Α ͜͜͸Ұ଍ઌʹ(উखʹ)DatabaseτϥοΫͰ͢ʂʂ ⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠ ͜ͷηογϣϯ͸ SERVERLESSཁૉ͕ ΄ͱΜͲ͋Γ·ͤΜʂʂ ⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠

4. SERVERLESS΍ͬͯ·͔͢ʁ DynamoDB࢖ͬͯ·͔͢ʁ

5. • ޷͖ • ී௨ • ݏ͍ • LambdaͱRDSͷ૬ੑ͕ѱ͍͔Β࢓ํͳ͘࢖͍ͬͯΔ DynamoDB͸޷͖Ͱ͔͢ʁ

6. • ίωΫγϣϯϞσϧ • Ӭଓత • ηΩϡϦςΟ • ૊ΈࠐΈͷऑ͍ೝূ • VPC಺΁ͷΞΫηείετʢENIੜ੒ॲཧʣ

   LambdaͱRDSͷ૬ੑ͕ѱ͍ʁ ͱͯ΋OLTPͰ͸࢖͑ͳ͍(਺ඵʙे਺ඵ)

7. ࢓ํ͕ͳ͍͔ΒDynamoDB…… ຊ౰͸RDBΛ࢖͍͍͚ͨͲ

8. DynamoDBΛੵۃతʹ࢖͓͏ ͦΜͳϚΠϯυͰେৎ෉ʁ

9. • ୺తʹݴͬͯόΠϒϧ • ͜ΕΛಡΜͰશͯཧղͰ͖ͨਓ͸
 ͨͿΜ͜ͷઌΛฉ͘ඞཁ͸͋Γ·ͤΜʂʂ • https://docs.aws.amazon.com/ja_jp/amazondynamodb/latest/ developerguide/best-practices.html DynamoDB Best

   Practices

10. • ςʔϒϧઃܭͱ͔ΠϯσοΫεઃܭͬͯͲ͏΍Δͷʁ • ACIDτϥϯβΫγϣϯແ͍ͷʹେৎ෉ͳͷʁ • ݁Ռ੔߹ੑͬͯେৎ෉ͳͷʁ • ্ख͘ੑೳΛҾ͖ग़͢ʹ͸Ͳ͏͢Ε͹ྑ͍ʁ DynamoDBͷΑ͋͘Δෆ҆

11. • ςʔϒϧઃܭͱ͔ΠϯσοΫεઃܭͬͯͲ͏΍Δͷʁ • ACIDτϥϯβΫγϣϯແ͍ͷʹେৎ෉ͳͷʁ • ݁Ռ੔߹ੑͬͯେৎ෉ͳͷʁ • ্ख͘ੑೳΛҾ͖ग़͢ʹ͸Ͳ͏͢Ε͹ྑ͍ʁ DynamoDBͷΑ͋͘Δෆ҆ ෼͔Βͳ͍͔Βෆ҆ʹͳΔ

12. • ͋͑ͯ෼ྨ͢ΔͳΒ෼ࢄKVSͱ͍͏δϟϯϧͷNoSQL • KeyʹΑΔ׬શҰகͱIndex୳ࡧ͕Մೳ • ϑϧϚωʔδυ • ແ੍ݶͷࣗಈ֦ுετϨʔδʢ࢖ͬͨ෼͚ͩʣ • Πϯελϯεͷ؅ཧෆཁɺWrite/Readͷੑೳ͚ͩΛࢦఆ͢Δ

    DynamoDBͷجຊ

13. DynamoDB͸ͲΜͳ ”AWSαʔϏε” ͔ Ͱ͸ͳ͘

14. DynamoDB͸ͲΜͳ ”σʔλϕʔε” ͔ ஌ͬͯ·͔͢ʁ

15. • Consistent Hashing • Vector clocks with reconciliation during reads

    • Sloppy Quorum and hinted handoff • Anti-entropy using Merkle trees • Gossip-based membership protocol and failure detection Amazon’s Dynamo ৄ͘͠஌Γ͍ͨͳΒʢBetterͰ͋ͬͯMustͰ͸ͳ͍ʣ https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

16. • Ωʔͷϋογϡ஋Ͱ෼ࢄ • όοΫάϥϯυͰߋ৽ΛޙউͪͰಉظ • ݁Ռ੔߹ੑϕʔε • ॻ͖ࠐΈɾಡΈࠐΈͷQuorum਺Ͱ੔߹ੑ͕ܾ·Δ • ΩʔΛࢦఆͨ͠ಡΈࠐΈ͸ڧ੔߹ੑ͕બ୒Մೳ

    ཈͑Δ΂͖ϙΠϯτ

17. Partition Keyͷϋογϡ஋Ͱ෼ࢄ Like Hash Table(Index)

18. • Partition Keyʹ࿈൪Λ࢖͏ͷ͸Կͷҙຯ΋ͳ͍͜ͱ͕෼͔Δ • ࿈൪Χ΢ϯλʔΛ࡞ΔͱPartition͕ภΔ • Max + 1ͳΜͯҎͯͷ֎ •

    ࿦ཧతʹҰҙͱͳΔଐੑ͕͋ΔͳΒͦΕΛ࢖͏ • ͳ͚Ε͹UUIDͷΑ͏ͳিಥՄೳੑ͕ۃখͳϥϯμϜ஋Ͱྑ͍ • Sort΋Ͱ͖ͳ͍ͷͰSnowflakeͷΑ͏ͳॱংੑ΋ཁΒͳ͍ • User ID͸CognitoͰೝূ͍ͯ͠ΔͳΒCognito͕ൃߦ͢ΔIDΛ࢖͏ • ΞΫηε੍ޚʹ΋࢖͑Δ ͜ͷલఏΛ౿·͑Δͱ

19. • 3000RCU, 1000WCU, Index Size 10GB (2018-10-31ݱࡏ) • ͲΕ͔ҰͭͰ΋௒ա͢ΔͱPartition͕෼ׂ͞ΕΔ •

    ෼ׂ͞ΕΔͱ֬อͨ͠Ϣχοτ͕ۉ౳ʹׂΓৼΒΕΔ • εϩοτϦϯά࣌ͷҰ࣌తͳόʔετ͕͋Δ • ݶఆతͳόʔετ • ༨৒෼Λճ͢ (Adaptive Capacity) Partition͝ͱͷΩϟύγςΟ

20. B+TREE INDEX B-treeͳͷ͔B+treeͳͷ͔͸ެද͞Ε͍ͯͳ͍͚Ͳ Range Keyͱ͔ݺͿ͘Β͍͔ͩΒ…Ͷʁ

21. TARGETING QUERY B+treeͷߏ଄Λ஌͍ͬͯΕ͹ ૊Έ߹ΘͤΩʔͷΠϯσοΫε͕ແͯ͘΋ ͜͏΍Ε͹ྑ͍͜ͱ͕෼͔Δ

22. Partition KeyͰ෼ࢄͨ͠தͰSort KeyͰ੔ྻ ͜ͷΠϝʔδΛ࣋ͭ͜ͱ͕ͱͯ΋ॏཁ

23. • Partition KeyͰ෼ࢄͨ͠ΠϯελϯεʹIndex͕ߏங͞ΕΔ • ҰͭͷΠϯελϯε಺Ͱ͸ಉظతʹIndex΋ߋ৽͞ΕΔ • ڧ੔߹ੑ͕બ୒Մೳ • Primary Sort

    KeyͷϦʔϑϊʔυʹ࣮σʔλͷΞυϨε͕
 ֨ೲ͞Ε͍ͯΔΠϝʔδʢͱࢥΘΕΔʣ • LSI͸࣮σʔλΛ࣋ͨͳ͍ͷͰॏෳՄೳ Primary Sort Key, LSI

24. GSI͸࣮ςʔϒϧͷࣹӨ

25. • ू߹࿦ʹ͓͚ΔࣹӨ • GSIͷPartition KeyΛݩʹ෼ࢄ
 ʢߋ৽͸ඇಉظͰద༻ʣ • ϦʔϑϊʔυʹଐੑͱPrimary Key
 ʢৄ͘͠͸ޙड़ʣ

    • ιʔτࡁΈͷσʔλͷ෦෼ෳ੡
 ʢҟ࿦͸ೝΊΔʣ ࣹӨ ࡶͳMySQLύϑΥʔϚϯενϡʔχϯά from yoku0825 https://www.slideshare.net/yoku0825/mysql-57449062/62

26. • KEYS_ONLY: Primary Keyͷ஋ͷΈ • INCLUDE: Primary Keyͷ஋ + બ୒ͨ͠ଐੑ

    • ALL: શͯͷଐੑ • GSIʹؚΊͨଐੑ͸Primary KeyΛҾ͖௚͢ඞཁ͕ͳ͘ͳΔ • ಡΈࠐΈͷޮ཰Խ • ߋ৽ίετͱͷτϨʔυΦϑ • MySQL΍PostgreSQLͰݴ͏Covering Index / Index Only ScanͷΠϝʔδ GSI͕ࣹӨ͢ΔଐੑͷΦϓγϣϯ

27. • ϝΠϯςʔϒϧͱ͸ผͷPartitionͰ෼ࢄ͞ΕΔ • ߋ৽͸ඇಉظͰ൓ө • ݁Ռ੔߹ੑͬͯେৎ෉ͳͷʁ • RDSͩͬͯRead ReplicaಡΜͩΒ݁Ռ੔߹ੑͰ͢Αʁʁ •

    GSI΁ͷΫΤϦ͸͍͍ͩͨRead Replicaʹ౤͛ΔΑ͏ͳΫΤϦ GSIʹର͢ΔΫΤϦ͸݁Ռ੔߹ੑ

28. ࢓૊ΈΛ஌Ε͹࣮ʹγϯϓϧ શ෦RDBͰطʹ͋ͬͨ࢓૊ΈͰ͢Αʁʁ

29. ☓ DynamoDB͸บ͕͋Δ ◦ RDBͷบ͕͋Γ͗͢Δ →γϯϓϧ͗͢ΔDynamoDBͱͷΪϟοϓ͕ڧͯͦ͘͏ײ͡Δ͚ͩ ɹRDBʹચ೴͞Ε͍ͯΔΜ͡Όͳ͍Ͱ͔͢ʁʁ

30. DynamoDBͷσʔλϞσϦϯάͱ͸ ͭ·Δͱ͜Ζ

31. σʔλͷ഑ஔΛܾΊΔ͜ͱ ҟ࿦͸ೝΊΔʂʂ

32. σʔλϞσϦϯάͷߟ͑ํͷϙΠϯτ ഑ஔΛܾΊΔͨΊͷ

33. • εΩʔϚϨεͷҙຯΛߟ͑Δ • List, Map͕࣋ͯΔ͜ͱʁ • ޙ͔ΒଐੑΛγʔϜϨεʹ௥ՃͰ͖Δ͜ͱʁ • A. ͳΜͰ΋ಥͬࠐΊΔ͜ͱ

    • Partition KeyͰ෼ࢄ • ςʔϒϧΛ෼͚ͳͯ͘΋ෛՙ͸෼ࢄ͢Δ • ۃ࿦͢Δͱ෼͚Δҙຯ͕ͳ͍ɺ෼͚ͳ͍ํ͕ΩϟύγςΟ͕؅ཧ͠΍͍͢ • “ઃܭ͕༏ΕͨΞϓϦέʔγϣϯͰ͸ɺඞཁͳςʔϒϧ͸ 1 ͭͷΈͰ͢” • GSIͷݸ਺੍ݶ(5ݸ)ͳͲ΋͋ΔͷͰۀ຿υϝΠϯຖʹҰͭ͘Β͍ͷؾ࣋ͪ ςʔϒϧͷ෼͚ํ

34. • RDB • ·ͣεΩʔϚΛߟ͑Δ • ਖ਼نԽ • ͦΕʹରͯ͠Ͳ͏ΞΫηε͢Δ͔(SQL)ߟ͑Δ • DynamoDB

    • εΩʔϚϨε • ඇਖ਼نԽ • ·ͣΞΫηεΛߟ͑ͯɺͦΕʹ߹ΘͤͨσʔλΛ࡞Δ • ΞϓϦέʔγϣϯͱσʔλϞσϧ͸ಉ࣌ʹઃܭ͢Δ RDBͱ͸Ξϓϩʔν͕શ͘ҟͳΔ

35. ACIDτϥϯβΫγϣϯ͕ແ͍ͷ͸Ͳ͏͢Δͷʁ ͱ͸͍͑

36. • ݪࢠੑ (Atomicity) • Ұ؏(੔߹)ੑ (Consistency) • ಠཱੑ (Isolation) •

    Ӭଓੑ (Durability) ͦ΋ͦ΋ACIDͬͯͳΜ͚ͩͬʁ ͜ΕΒΛ୲อ͢Δ͜ͱ͕໨తͰ͋ͬͯ τϥϯβΫγϣϯγεςϜͰ͋Δඞཁ͸ແ͍

37. • RDB • ਖ਼نԽ͍ͯ͘͠ͱςʔϒϧͷ਺͸جຊతʹ૿͑Δ • ࿦ཧతʹ෼͔Εͨςʔϒϧ(Ϩίʔυ)ؒͷ੔߹ੑΛऔΔ࢓૊Έ͕͋Δ • ϦϨʔγϣϯγοϓ • ֎෦Ωʔ੍໿

    • ACIDτϥϯβΫγϣϯ • DynamoDB • ඇਖ਼نԽͯ͠1ͭͷ࣮ମʹ͍ͭͯ1ͭͷΞΠςϜʹऩΊΔ • 1ΞΠςϜͷߋ৽͸ΞτϛοΫ ACIDͷ୲อ

38. • PutItem • ΞΠςϜΛ·Δ͝ͱߋ৽͢Δɺແ͔ͬͨΒ࡞Δ • Get -> ॻ͖׵͑ -> Putͩͱಠཱੑ่͕ΕΔ

    • UpdateItem • ΞΠςϜΛ෦෼ߋ৽ɺແ͔ͬͨΒ࡞Δ • List, Mapͷ෦෼௥Ճɺ࡟আͳͲ΋Մೳ • ଐੑͷ஋Λར༻ͨ͠ߋ৽΋Մೳʢ͋Δଐੑʹ+1͢Δ౳) • ଐੑͷ஋Λར༻ͨ͠৚݅෇͖ॻ͖ࠐΈ DynamoDBͷߋ৽ૢ࡞

39. ඇਖ਼نԽ͢ΔͱಡΈࠐΈ͕ͭΒ͍ ϑϧεΩϟϯ͢Δ͔͠ͳ͍ શ෦GSIషΔͷʁ

40. CQRSͰ͢Α ͦ͜Ͱ

41. • ॻ͖ࠐΉσʔλͱಡΈࠐΉσʔλ͸ಉ͡Ͱ͋Δඞཁ͸ແ͍ • ݁Ռ੔߹ੑΛड͚ೖΕͯඇಉظͰಡΈࠐΈσʔλΛ࡞ΕΔ • Commandͷ׬ྃΛ΋ͬͯQuery༻σʔλΛ࡞ΕΔ • DynamoDB Streams͕Ϛον͢Δ •

    CQRSΛૉ௚ʹ࣮૷͢Δͱ·ͣKinesis Streams͚ͩͲ • Materialized View CQRS (Command Query Responsibility Segregation)

42. ARCHITECTURE PATTERN AppSync͕͜Ε͔ΒॏཁͳҐஔΛ઎ΊΔ͸ͣ

43. DATA ATOMICITY MicroserviesతͳΠϝʔδͰ͸͜ͷݪ୯Ґ͕ॏཁ σʔλετΞɾςʔϒϧͷ෼͚ํͰ͸ͳ͍ ލ͕͍ͤͨͳΒMaterialized ViewΛ࡞Δ User Profile User Status

    Name Email Read/Write Search Read/Write

44. ͜ΜͳΠϝʔδ 1ͭͷGSIͰ༷ʑͳݕࡧ͕Ͱ͖Δ

45. • ෳ਺ͷ࣮ମʹލΔτϥϯβΫγϣϯ • DynamoDB୯ମͩͱ͜͜͸Ͳ͏ͯ͠΋ͭΒ͍ • τϥϯβΫγϣϯ෼཭Ϩϕϧͷ࠷ߴ͸Selializable • ͭ·Γ௚ྻԽ • Kinesis

    StreamsͰ௚ྻԽ • ίέΔ͔΋͠Εͳ͍ͷͰႈ౳ʹ࣮૷͢Δ • ৚݅෇ॻ͖ࠐΈͰطʹߦͬͨॲཧΛSkip͢Δͷ΋ႈ౳ • ͱ͍͏͔ɺKinesis StreamsͰಉ࣌઀ଓ਺Λίϯτϩʔϧ͢Ε͹RDS࢖ͬͨͬͯྑ͍Θ͚Ͱ… ͱ͸͍͑ݶք͸͋Δ

46. ADVANCED TIPS ͍͔ͭ͘͝঺հ

47. • Partition Keyͷύλʔϯ͕গͳ͍৔߹ • ෼ࢄ͠ͳ͘ͳͬͯ͠·͏ • 1PartitionͷΩϟύγςΟΛӽ͑ͦ͏ͳΒγϟʔσΟϯά͢Δ • KeyҎ֎ͷࣗ໌ͳଐੑ͔ΒܭࢉՄೳͳSuffixΛ෇༩͢Δ •

    ूܭ΍ൣғݕࡧͷඞཁ͕ͳ͚Ε͹Hash • ඞཁ͕͋Ε͹Mod౳ GSIͷγϟʔσΟϯά

48. ͜ΜͳΠϝʔδ Unicode PointΛ߹ܭͯ͠200Ͱ ׂͬͨ༨ΓΛ࢖༻͍ͯ͠Δ

49. • DynamoDBͷIndex͸Sparse • Indexଐੑ͕ଘࡏ͠ͳ͍৔߹ɺIndexʹͦͷσʔλ͸ؚ·ͳ͍ • ಛఆͷΞΠςϜͷΈʹଘࡏ͢ΔଐੑΛࢦఆ͢Δ͜ͱͰ
 Index༰ྔ͕େ͖͘ѹॖͰ͖Δ Sparse Index

50. Sort΋ཁΒͳ͍ GSIʹ͸҉໧ͰPrimary Keyؚ͕·ΕΔ

51. • Α͋͘Δ֊૚ߏ଄Λ࣋ͭσʔλ • ૊৫ߏ଄ • ωετͨ͠ΧςΰϥΠζ • ͳͲͳͲ • ݕࡧΛߟ͑ͳ͚Ε͹֊૚ߏ଄Λ·Δͬͱ1ΞΠςϜʹԡ͠ࠐΉ

    • Sort KeyͰ֊૚Λදݱ͢Δ Hierarchical data

52. Sort KeyͰલํҰகݕࡧ ͋ɺ͜ΕSQLΞϯνύλʔϯͰʢ͈́ ঎඼ઃܭ͕ࡶͳͷ͸͝Ѫᇷ

53. “DynamoDB ދͷר” [ݕࡧ] http://marcy.hatenablog.com/entry/2018/07/31/213705 ҰԠະ׬ͳͷͰ·ͩॻ͔͘΋

54. “LET’S BECOME THE MASTER OF DYNAMODB” THANKS!! “TO BECOME THE

    MASTER OF SERVERLESS”