$30 off During Our Annual Pro Sale. View Details »

Serverlessを極めるためにDynamoDBデータモデリングを極めよう / Let’s become the master of DynamoDB Data Modeling to become the master of Serverless

Serverlessを極めるためにDynamoDBデータモデリングを極めよう / Let’s become the master of DynamoDB Data Modeling to become the master of Serverless

AWS Dev Day Tokyo 2018の登壇資料です
https://aws.amazon.com/jp/aws-devday-tokyo-2018/

Masashi Terui

October 31, 2018
Tweet

More Decks by Masashi Terui

Other Decks in Technology

Transcript

  1. LET’S BECOME THE MASTER
    OF DYNAMODB DATA MODELING
    TO BECOME THE MASTER OF SERVERLESS
    MASASHI TERUI @ AWS DEV DAY TOKYO 2018

    View Slide

  2. SERVERWORKS CO.,LTD.
    + FREELANCER
    • Serverless Oji-san
    • Serverless Framework Plugin Developer
    • Serverlessconf Tokyo 2016,2017,2018 speaker
    • Remote worker (in Sapporo-shi, Hokkaido)
    • http://marcy.hatenablog.com/entry/2018/08/04/231241
    MASASHI TERUI
    ARCHITECT / DEVELOPER

    View Slide

  3. SERVERLESSͳΜͯ০ΓͰ͢
    Ғ͍ਓʹ͸ͦΕ͕Θ͔ΒΜͷͰ͢Α
    ͜͜͸Ұ଍ઌʹ(উखʹ)DatabaseτϥοΫͰ͢ʂʂ
    ⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠
    ͜ͷηογϣϯ͸
    SERVERLESSཁૉ͕
    ΄ͱΜͲ͋Γ·ͤΜʂʂ
    ⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠

    View Slide

  4. SERVERLESS΍ͬͯ·͔͢ʁ
    DynamoDB࢖ͬͯ·͔͢ʁ

    View Slide

  5. • ޷͖
    • ී௨
    • ݏ͍
    • LambdaͱRDSͷ૬ੑ͕ѱ͍͔Β࢓ํͳ͘࢖͍ͬͯΔ
    DynamoDB͸޷͖Ͱ͔͢ʁ

    View Slide

  6. • ίωΫγϣϯϞσϧ
    • Ӭଓత
    • ηΩϡϦςΟ
    • ૊ΈࠐΈͷऑ͍ೝূ
    • VPC಺΁ͷΞΫηείετʢENIੜ੒ॲཧʣ
    LambdaͱRDSͷ૬ੑ͕ѱ͍ʁ
    ͱͯ΋OLTPͰ͸࢖͑ͳ͍(਺ඵʙे਺ඵ)

    View Slide

  7. ࢓ํ͕ͳ͍͔ΒDynamoDB……
    ຊ౰͸RDBΛ࢖͍͍͚ͨͲ

    View Slide

  8. DynamoDBΛੵۃతʹ࢖͓͏
    ͦΜͳϚΠϯυͰେৎ෉ʁ

    View Slide

  9. • ୺తʹݴͬͯόΠϒϧ
    • ͜ΕΛಡΜͰશͯཧղͰ͖ͨਓ͸

    ͨͿΜ͜ͷઌΛฉ͘ඞཁ͸͋Γ·ͤΜʂʂ
    • https://docs.aws.amazon.com/ja_jp/amazondynamodb/latest/
    developerguide/best-practices.html
    DynamoDB Best Practices

    View Slide

  10. • ςʔϒϧઃܭͱ͔ΠϯσοΫεઃܭͬͯͲ͏΍Δͷʁ
    • ACIDτϥϯβΫγϣϯແ͍ͷʹେৎ෉ͳͷʁ
    • ݁Ռ੔߹ੑͬͯେৎ෉ͳͷʁ
    • ্ख͘ੑೳΛҾ͖ग़͢ʹ͸Ͳ͏͢Ε͹ྑ͍ʁ
    DynamoDBͷΑ͋͘Δෆ҆

    View Slide

  11. • ςʔϒϧઃܭͱ͔ΠϯσοΫεઃܭͬͯͲ͏΍Δͷʁ
    • ACIDτϥϯβΫγϣϯແ͍ͷʹେৎ෉ͳͷʁ
    • ݁Ռ੔߹ੑͬͯେৎ෉ͳͷʁ
    • ্ख͘ੑೳΛҾ͖ग़͢ʹ͸Ͳ͏͢Ε͹ྑ͍ʁ
    DynamoDBͷΑ͋͘Δෆ҆
    ෼͔Βͳ͍͔Βෆ҆ʹͳΔ

    View Slide

  12. • ͋͑ͯ෼ྨ͢ΔͳΒ෼ࢄKVSͱ͍͏δϟϯϧͷNoSQL
    • KeyʹΑΔ׬શҰகͱIndex୳ࡧ͕Մೳ
    • ϑϧϚωʔδυ
    • ແ੍ݶͷࣗಈ֦ுετϨʔδʢ࢖ͬͨ෼͚ͩʣ
    • Πϯελϯεͷ؅ཧෆཁɺWrite/Readͷੑೳ͚ͩΛࢦఆ͢Δ
    DynamoDBͷجຊ

    View Slide

  13. DynamoDB͸ͲΜͳ ”AWSαʔϏε” ͔
    Ͱ͸ͳ͘

    View Slide

  14. DynamoDB͸ͲΜͳ ”σʔλϕʔε” ͔
    ஌ͬͯ·͔͢ʁ

    View Slide

  15. • Consistent Hashing
    • Vector clocks with reconciliation during reads
    • Sloppy Quorum and hinted handoff
    • Anti-entropy using Merkle trees
    • Gossip-based membership protocol and failure detection
    Amazon’s Dynamo
    ৄ͘͠஌Γ͍ͨͳΒʢBetterͰ͋ͬͯMustͰ͸ͳ͍ʣ
    https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

    View Slide

  16. • Ωʔͷϋογϡ஋Ͱ෼ࢄ
    • όοΫάϥϯυͰߋ৽ΛޙউͪͰಉظ
    • ݁Ռ੔߹ੑϕʔε
    • ॻ͖ࠐΈɾಡΈࠐΈͷQuorum਺Ͱ੔߹ੑ͕ܾ·Δ
    • ΩʔΛࢦఆͨ͠ಡΈࠐΈ͸ڧ੔߹ੑ͕બ୒Մೳ
    ཈͑Δ΂͖ϙΠϯτ

    View Slide

  17. Partition Keyͷϋογϡ஋Ͱ෼ࢄ
    Like Hash Table(Index)

    View Slide

  18. • Partition Keyʹ࿈൪Λ࢖͏ͷ͸Կͷҙຯ΋ͳ͍͜ͱ͕෼͔Δ
    • ࿈൪Χ΢ϯλʔΛ࡞ΔͱPartition͕ภΔ
    • Max + 1ͳΜͯҎͯͷ֎
    • ࿦ཧతʹҰҙͱͳΔଐੑ͕͋ΔͳΒͦΕΛ࢖͏
    • ͳ͚Ε͹UUIDͷΑ͏ͳিಥՄೳੑ͕ۃখͳϥϯμϜ஋Ͱྑ͍
    • Sort΋Ͱ͖ͳ͍ͷͰSnowflakeͷΑ͏ͳॱংੑ΋ཁΒͳ͍
    • User ID͸CognitoͰೝূ͍ͯ͠ΔͳΒCognito͕ൃߦ͢ΔIDΛ࢖͏
    • ΞΫηε੍ޚʹ΋࢖͑Δ
    ͜ͷલఏΛ౿·͑Δͱ

    View Slide

  19. • 3000RCU, 1000WCU, Index Size 10GB (2018-10-31ݱࡏ)
    • ͲΕ͔ҰͭͰ΋௒ա͢ΔͱPartition͕෼ׂ͞ΕΔ
    • ෼ׂ͞ΕΔͱ֬อͨ͠Ϣχοτ͕ۉ౳ʹׂΓৼΒΕΔ
    • εϩοτϦϯά࣌ͷҰ࣌తͳόʔετ͕͋Δ
    • ݶఆతͳόʔετ
    • ༨৒෼Λճ͢ (Adaptive Capacity)
    Partition͝ͱͷΩϟύγςΟ

    View Slide

  20. B+TREE INDEX B-treeͳͷ͔B+treeͳͷ͔͸ެද͞Ε͍ͯͳ͍͚Ͳ
    Range Keyͱ͔ݺͿ͘Β͍͔ͩΒ…Ͷʁ

    View Slide

  21. TARGETING QUERY B+treeͷߏ଄Λ஌͍ͬͯΕ͹
    ૊Έ߹ΘͤΩʔͷΠϯσοΫε͕ແͯ͘΋
    ͜͏΍Ε͹ྑ͍͜ͱ͕෼͔Δ

    View Slide

  22. Partition KeyͰ෼ࢄͨ͠தͰSort KeyͰ੔ྻ
    ͜ͷΠϝʔδΛ࣋ͭ͜ͱ͕ͱͯ΋ॏཁ

    View Slide

  23. • Partition KeyͰ෼ࢄͨ͠ΠϯελϯεʹIndex͕ߏங͞ΕΔ
    • ҰͭͷΠϯελϯε಺Ͱ͸ಉظతʹIndex΋ߋ৽͞ΕΔ
    • ڧ੔߹ੑ͕બ୒Մೳ
    • Primary Sort KeyͷϦʔϑϊʔυʹ࣮σʔλͷΞυϨε͕

    ֨ೲ͞Ε͍ͯΔΠϝʔδʢͱࢥΘΕΔʣ
    • LSI͸࣮σʔλΛ࣋ͨͳ͍ͷͰॏෳՄೳ
    Primary Sort Key, LSI

    View Slide

  24. GSI͸࣮ςʔϒϧͷࣹӨ

    View Slide

  25. • ू߹࿦ʹ͓͚ΔࣹӨ
    • GSIͷPartition KeyΛݩʹ෼ࢄ

    ʢߋ৽͸ඇಉظͰద༻ʣ
    • ϦʔϑϊʔυʹଐੑͱPrimary Key

    ʢৄ͘͠͸ޙड़ʣ
    • ιʔτࡁΈͷσʔλͷ෦෼ෳ੡

    ʢҟ࿦͸ೝΊΔʣ
    ࣹӨ
    ࡶͳMySQLύϑΥʔϚϯενϡʔχϯά from yoku0825
    https://www.slideshare.net/yoku0825/mysql-57449062/62

    View Slide

  26. • KEYS_ONLY: Primary Keyͷ஋ͷΈ
    • INCLUDE: Primary Keyͷ஋ + બ୒ͨ͠ଐੑ
    • ALL: શͯͷଐੑ
    • GSIʹؚΊͨଐੑ͸Primary KeyΛҾ͖௚͢ඞཁ͕ͳ͘ͳΔ
    • ಡΈࠐΈͷޮ཰Խ
    • ߋ৽ίετͱͷτϨʔυΦϑ
    • MySQL΍PostgreSQLͰݴ͏Covering Index / Index Only ScanͷΠϝʔδ
    GSI͕ࣹӨ͢ΔଐੑͷΦϓγϣϯ

    View Slide

  27. • ϝΠϯςʔϒϧͱ͸ผͷPartitionͰ෼ࢄ͞ΕΔ
    • ߋ৽͸ඇಉظͰ൓ө
    • ݁Ռ੔߹ੑͬͯେৎ෉ͳͷʁ
    • RDSͩͬͯRead ReplicaಡΜͩΒ݁Ռ੔߹ੑͰ͢Αʁʁ
    • GSI΁ͷΫΤϦ͸͍͍ͩͨRead Replicaʹ౤͛ΔΑ͏ͳΫΤϦ
    GSIʹର͢ΔΫΤϦ͸݁Ռ੔߹ੑ

    View Slide

  28. ࢓૊ΈΛ஌Ε͹࣮ʹγϯϓϧ
    શ෦RDBͰطʹ͋ͬͨ࢓૊ΈͰ͢Αʁʁ

    View Slide

  29. ☓ DynamoDB͸บ͕͋Δ
    ○ RDBͷบ͕͋Γ͗͢Δ
    →γϯϓϧ͗͢ΔDynamoDBͱͷΪϟοϓ͕ڧͯͦ͘͏ײ͡Δ͚ͩ
    ɹRDBʹચ೴͞Ε͍ͯΔΜ͡Όͳ͍Ͱ͔͢ʁʁ

    View Slide

  30. DynamoDBͷσʔλϞσϦϯάͱ͸
    ͭ·Δͱ͜Ζ

    View Slide

  31. σʔλͷ഑ஔΛܾΊΔ͜ͱ
    ҟ࿦͸ೝΊΔʂʂ

    View Slide

  32. σʔλϞσϦϯάͷߟ͑ํͷϙΠϯτ
    ഑ஔΛܾΊΔͨΊͷ

    View Slide

  33. • εΩʔϚϨεͷҙຯΛߟ͑Δ
    • List, Map͕࣋ͯΔ͜ͱʁ
    • ޙ͔ΒଐੑΛγʔϜϨεʹ௥ՃͰ͖Δ͜ͱʁ
    • A. ͳΜͰ΋ಥͬࠐΊΔ͜ͱ
    • Partition KeyͰ෼ࢄ
    • ςʔϒϧΛ෼͚ͳͯ͘΋ෛՙ͸෼ࢄ͢Δ
    • ۃ࿦͢Δͱ෼͚Δҙຯ͕ͳ͍ɺ෼͚ͳ͍ํ͕ΩϟύγςΟ͕؅ཧ͠΍͍͢
    • “ઃܭ͕༏ΕͨΞϓϦέʔγϣϯͰ͸ɺඞཁͳςʔϒϧ͸ 1 ͭͷΈͰ͢”
    • GSIͷݸ਺੍ݶ(5ݸ)ͳͲ΋͋ΔͷͰۀ຿υϝΠϯຖʹҰͭ͘Β͍ͷؾ࣋ͪ
    ςʔϒϧͷ෼͚ํ

    View Slide

  34. • RDB
    • ·ͣεΩʔϚΛߟ͑Δ
    • ਖ਼نԽ
    • ͦΕʹରͯ͠Ͳ͏ΞΫηε͢Δ͔(SQL)ߟ͑Δ
    • DynamoDB
    • εΩʔϚϨε
    • ඇਖ਼نԽ
    • ·ͣΞΫηεΛߟ͑ͯɺͦΕʹ߹ΘͤͨσʔλΛ࡞Δ
    • ΞϓϦέʔγϣϯͱσʔλϞσϧ͸ಉ࣌ʹઃܭ͢Δ
    RDBͱ͸Ξϓϩʔν͕શ͘ҟͳΔ

    View Slide

  35. ACIDτϥϯβΫγϣϯ͕ແ͍ͷ͸Ͳ͏͢Δͷʁ
    ͱ͸͍͑

    View Slide

  36. • ݪࢠੑ (Atomicity)
    • Ұ؏(੔߹)ੑ (Consistency)
    • ಠཱੑ (Isolation)
    • Ӭଓੑ (Durability)
    ͦ΋ͦ΋ACIDͬͯͳΜ͚ͩͬʁ
    ͜ΕΒΛ୲อ͢Δ͜ͱ͕໨తͰ͋ͬͯ
    τϥϯβΫγϣϯγεςϜͰ͋Δඞཁ͸ແ͍

    View Slide

  37. • RDB
    • ਖ਼نԽ͍ͯ͘͠ͱςʔϒϧͷ਺͸جຊతʹ૿͑Δ
    • ࿦ཧతʹ෼͔Εͨςʔϒϧ(Ϩίʔυ)ؒͷ੔߹ੑΛऔΔ࢓૊Έ͕͋Δ
    • ϦϨʔγϣϯγοϓ
    • ֎෦Ωʔ੍໿
    • ACIDτϥϯβΫγϣϯ
    • DynamoDB
    • ඇਖ਼نԽͯ͠1ͭͷ࣮ମʹ͍ͭͯ1ͭͷΞΠςϜʹऩΊΔ
    • 1ΞΠςϜͷߋ৽͸ΞτϛοΫ
    ACIDͷ୲อ

    View Slide

  38. • PutItem
    • ΞΠςϜΛ·Δ͝ͱߋ৽͢Δɺແ͔ͬͨΒ࡞Δ
    • Get -> ॻ͖׵͑ -> Putͩͱಠཱੑ่͕ΕΔ
    • UpdateItem
    • ΞΠςϜΛ෦෼ߋ৽ɺແ͔ͬͨΒ࡞Δ
    • List, Mapͷ෦෼௥Ճɺ࡟আͳͲ΋Մೳ
    • ଐੑͷ஋Λར༻ͨ͠ߋ৽΋Մೳʢ͋Δଐੑʹ+1͢Δ౳)
    • ଐੑͷ஋Λར༻ͨ͠৚݅෇͖ॻ͖ࠐΈ
    DynamoDBͷߋ৽ૢ࡞

    View Slide

  39. ඇਖ਼نԽ͢ΔͱಡΈࠐΈ͕ͭΒ͍
    ϑϧεΩϟϯ͢Δ͔͠ͳ͍
    શ෦GSIషΔͷʁ

    View Slide

  40. CQRSͰ͢Α
    ͦ͜Ͱ

    View Slide

  41. • ॻ͖ࠐΉσʔλͱಡΈࠐΉσʔλ͸ಉ͡Ͱ͋Δඞཁ͸ແ͍
    • ݁Ռ੔߹ੑΛड͚ೖΕͯඇಉظͰಡΈࠐΈσʔλΛ࡞ΕΔ
    • Commandͷ׬ྃΛ΋ͬͯQuery༻σʔλΛ࡞ΕΔ
    • DynamoDB Streams͕Ϛον͢Δ
    • CQRSΛૉ௚ʹ࣮૷͢Δͱ·ͣKinesis Streams͚ͩͲ
    • Materialized View
    CQRS (Command Query Responsibility Segregation)

    View Slide

  42. ARCHITECTURE PATTERN
    AppSync͕͜Ε͔ΒॏཁͳҐஔΛ઎ΊΔ͸ͣ

    View Slide

  43. DATA ATOMICITY MicroserviesతͳΠϝʔδͰ͸͜ͷݪ୯Ґ͕ॏཁ
    σʔλετΞɾςʔϒϧͷ෼͚ํͰ͸ͳ͍
    ލ͕͍ͤͨͳΒMaterialized ViewΛ࡞Δ
    User
    Profile
    User
    Status
    Name Email
    Read/Write
    Search
    Read/Write

    View Slide

  44. ͜ΜͳΠϝʔδ 1ͭͷGSIͰ༷ʑͳݕࡧ͕Ͱ͖Δ

    View Slide

  45. • ෳ਺ͷ࣮ମʹލΔτϥϯβΫγϣϯ
    • DynamoDB୯ମͩͱ͜͜͸Ͳ͏ͯ͠΋ͭΒ͍
    • τϥϯβΫγϣϯ෼཭Ϩϕϧͷ࠷ߴ͸Selializable
    • ͭ·Γ௚ྻԽ
    • Kinesis StreamsͰ௚ྻԽ
    • ίέΔ͔΋͠Εͳ͍ͷͰႈ౳ʹ࣮૷͢Δ
    • ৚݅෇ॻ͖ࠐΈͰطʹߦͬͨॲཧΛSkip͢Δͷ΋ႈ౳
    • ͱ͍͏͔ɺKinesis StreamsͰಉ࣌઀ଓ਺Λίϯτϩʔϧ͢Ε͹RDS࢖ͬͨͬͯྑ͍Θ͚Ͱ…
    ͱ͸͍͑ݶք͸͋Δ

    View Slide

  46. ADVANCED TIPS
    ͍͔ͭ͘͝঺հ

    View Slide

  47. • Partition Keyͷύλʔϯ͕গͳ͍৔߹
    • ෼ࢄ͠ͳ͘ͳͬͯ͠·͏
    • 1PartitionͷΩϟύγςΟΛӽ͑ͦ͏ͳΒγϟʔσΟϯά͢Δ
    • KeyҎ֎ͷࣗ໌ͳଐੑ͔ΒܭࢉՄೳͳSuffixΛ෇༩͢Δ
    • ूܭ΍ൣғݕࡧͷඞཁ͕ͳ͚Ε͹Hash
    • ඞཁ͕͋Ε͹Mod౳
    GSIͷγϟʔσΟϯά

    View Slide

  48. ͜ΜͳΠϝʔδ Unicode PointΛ߹ܭͯ͠200Ͱ
    ׂͬͨ༨ΓΛ࢖༻͍ͯ͠Δ

    View Slide

  49. • DynamoDBͷIndex͸Sparse
    • Indexଐੑ͕ଘࡏ͠ͳ͍৔߹ɺIndexʹͦͷσʔλ͸ؚ·ͳ͍
    • ಛఆͷΞΠςϜͷΈʹଘࡏ͢ΔଐੑΛࢦఆ͢Δ͜ͱͰ

    Index༰ྔ͕େ͖͘ѹॖͰ͖Δ
    Sparse Index

    View Slide

  50. Sort΋ཁΒͳ͍ GSIʹ͸҉໧ͰPrimary Keyؚ͕·ΕΔ

    View Slide

  51. • Α͋͘Δ֊૚ߏ଄Λ࣋ͭσʔλ
    • ૊৫ߏ଄
    • ωετͨ͠ΧςΰϥΠζ
    • ͳͲͳͲ
    • ݕࡧΛߟ͑ͳ͚Ε͹֊૚ߏ଄Λ·Δͬͱ1ΞΠςϜʹԡ͠ࠐΉ
    • Sort KeyͰ֊૚Λදݱ͢Δ
    Hierarchical data

    View Slide

  52. Sort KeyͰલํҰகݕࡧ ͋ɺ͜ΕSQLΞϯνύλʔϯͰʢ͈́
    ঎඼ઃܭ͕ࡶͳͷ͸͝Ѫᇷ

    View Slide

  53. “DynamoDB ދͷר” [ݕࡧ]
    http://marcy.hatenablog.com/entry/2018/07/31/213705
    ҰԠະ׬ͳͷͰ·ͩॻ͔͘΋

    View Slide

  54. “LET’S BECOME THE MASTER OF DYNAMODB”
    THANKS!!
    “TO BECOME THE MASTER OF SERVERLESS”

    View Slide