Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Serverlessを極めるためにDynamoDBデータモデリングを極めよう / Let’s ...
Search
Masashi Terui
October 31, 2018
Technology
12
17k
Serverlessを極めるためにDynamoDBデータモデリングを極めよう / Let’s become the master of DynamoDB Data Modeling to become the master of Serverless
AWS Dev Day Tokyo 2018の登壇資料です
https://aws.amazon.com/jp/aws-devday-tokyo-2018/
Masashi Terui
October 31, 2018
Tweet
Share
More Decks by Masashi Terui
See All by Masashi Terui
What is Jeffy ?? - Why did we create a new framework
marcyterui
2
810
Serverlessの今とこれから / Everything will be Serverless
marcyterui
7
3.3k
見せてやろう…!! Serverlessの本当の力を…!! / Let’s show you the real power of serverless…!!
marcyterui
6
2.4k
re:Invent 2018でServerlessの世界は何が変わったか / What has changed in the world of Serverless at re:Invent 2018??
marcyterui
4
2.8k
実践サーバレス + α / Serverless Practices
marcyterui
2
580
Serverless??そんなことよりDBだ!! / We should know Databases before Serverless
marcyterui
3
3.5k
マイクロサービスを成功させるためのサーバーレスアーキテクチャ設計とNoSQLデータモデリング / Serverless Architecting and NoSQL Data Modeling for Successful Microservices
marcyterui
13
6.6k
何故私達はサーバレスアプリケーションをイベントドリブンで作るべきなのか / Why should we make serverless application as event driven?
marcyterui
8
1.2k
The theory of data modeling to become the friend of DynamoDB
marcyterui
2
210
Other Decks in Technology
See All in Technology
成果のためのコミュニケーション - 語彙を育てよう -/communication-for-good-outcome-developing-vocabulary
hassaku63
4
150
ドメインと向き合う - 旅行予約編
hidenorigoto
4
550
Efficient zero-copy networking using io_uring
ennael
PRO
0
320
軽いノリで"自動化"に取り組んではいけないという話
tetsuyaooooo
1
430
Rubyはなぜ「たのしい」のか? / Why is Ruby a programmers' best friend? #tqrk15
expajp
4
1.8k
「ばん・さく・つき・たー!」にならないためにSHIROBAKOから 学んだこと
ysknsid25
3
630
普通の Web エンジニアのための様相論理入門 #yapcjapan / YAPC Hakodate 2024
ytaka23
5
1.2k
Pythonを活用したLLMによる構造的データ生成の手法と実践
brainpadpr
3
280
入門 KRR
donkomura
0
110
いまからでも遅くない! コンテナでWebアプリケーションを 動かしてみよう(2-1)WebAPI座学
nomu
0
150
[JAWS-UG GameTech] 第6回 各種事例紹介_18TRIPにおけるAWSサービスを活用した負荷テスト・障害テスト
naoto_yasuda
0
140
Hazard pointers with reference counter
ennael
PRO
0
120
Featured
See All Featured
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
249
21k
The Power of CSS Pseudo Elements
geoffreycrofte
71
5.3k
Facilitating Awesome Meetings
lara
49
6k
Visualization
eitanlees
143
15k
How GitHub Uses GitHub to Build GitHub
holman
473
290k
Build The Right Thing And Hit Your Dates
maggiecrowley
31
2.3k
Keith and Marios Guide to Fast Websites
keithpitt
408
22k
Building Flexible Design Systems
yeseniaperezcruz
327
38k
Documentation Writing (for coders)
carmenintech
65
4.3k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
29
1.7k
A designer walks into a library…
pauljervisheath
201
24k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
38
2.1k
Transcript
LET’S BECOME THE MASTER OF DYNAMODB DATA MODELING TO BECOME
THE MASTER OF SERVERLESS MASASHI TERUI @ AWS DEV DAY TOKYO 2018
SERVERWORKS CO.,LTD. + FREELANCER • Serverless Oji-san • Serverless Framework
Plugin Developer • Serverlessconf Tokyo 2016,2017,2018 speaker • Remote worker (in Sapporo-shi, Hokkaido) • http://marcy.hatenablog.com/entry/2018/08/04/231241 MASASHI TERUI ARCHITECT / DEVELOPER
SERVERLESSͳΜͯ০ΓͰ͢ Ғ͍ਓʹͦΕ͕Θ͔ΒΜͷͰ͢Α ͜͜Ұઌʹ(উखʹ)DatabaseτϥοΫͰ͢ʂʂ ⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠ ͜ͷηογϣϯ SERVERLESSཁૉ͕ ΄ͱΜͲ͋Γ·ͤΜʂʂ ⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠⾠
SERVERLESSͬͯ·͔͢ʁ DynamoDBͬͯ·͔͢ʁ
• ͖ • ී௨ • ݏ͍ • LambdaͱRDSͷ૬ੑ͕ѱ͍͔Βํͳ͍ͬͯ͘Δ DynamoDB͖Ͱ͔͢ʁ
• ίωΫγϣϯϞσϧ • Ӭଓత • ηΩϡϦςΟ • ΈࠐΈͷऑ͍ೝূ • VPCͷΞΫηείετʢENIੜॲཧʣ
LambdaͱRDSͷ૬ੑ͕ѱ͍ʁ ͱͯOLTPͰ͑ͳ͍(ඵʙेඵ)
ํ͕ͳ͍͔ΒDynamoDB…… ຊRDBΛ͍͍͚ͨͲ
DynamoDBΛੵۃతʹ͓͏ ͦΜͳϚΠϯυͰେৎʁ
• తʹݴͬͯόΠϒϧ • ͜ΕΛಡΜͰશͯཧղͰ͖ͨਓ ͨͿΜ͜ͷઌΛฉ͘ඞཁ͋Γ·ͤΜʂʂ • https://docs.aws.amazon.com/ja_jp/amazondynamodb/latest/ developerguide/best-practices.html DynamoDB Best
Practices
• ςʔϒϧઃܭͱ͔ΠϯσοΫεઃܭͬͯͲ͏Δͷʁ • ACIDτϥϯβΫγϣϯແ͍ͷʹେৎͳͷʁ • ݁Ռ߹ੑͬͯେৎͳͷʁ • ্ख͘ੑೳΛҾ͖ग़͢ʹͲ͏͢Εྑ͍ʁ DynamoDBͷΑ͋͘Δෆ҆
• ςʔϒϧઃܭͱ͔ΠϯσοΫεઃܭͬͯͲ͏Δͷʁ • ACIDτϥϯβΫγϣϯແ͍ͷʹେৎͳͷʁ • ݁Ռ߹ੑͬͯେৎͳͷʁ • ্ख͘ੑೳΛҾ͖ग़͢ʹͲ͏͢Εྑ͍ʁ DynamoDBͷΑ͋͘Δෆ҆ ͔Βͳ͍͔Βෆ҆ʹͳΔ
• ͋͑ͯྨ͢ΔͳΒࢄKVSͱ͍͏δϟϯϧͷNoSQL • KeyʹΑΔશҰகͱIndex୳ࡧ͕Մೳ • ϑϧϚωʔδυ • ແ੍ݶͷࣗಈ֦ுετϨʔδʢ͚ͬͨͩʣ • ΠϯελϯεͷཧෆཁɺWrite/Readͷੑೳ͚ͩΛࢦఆ͢Δ
DynamoDBͷجຊ
DynamoDBͲΜͳ ”AWSαʔϏε” ͔ Ͱͳ͘
DynamoDBͲΜͳ ”σʔλϕʔε” ͔ ͬͯ·͔͢ʁ
• Consistent Hashing • Vector clocks with reconciliation during reads
• Sloppy Quorum and hinted handoff • Anti-entropy using Merkle trees • Gossip-based membership protocol and failure detection Amazon’s Dynamo ৄ͘͠Γ͍ͨͳΒʢBetterͰ͋ͬͯMustͰͳ͍ʣ https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
• ΩʔͷϋογϡͰࢄ • όοΫάϥϯυͰߋ৽ΛޙউͪͰಉظ • ݁Ռ߹ੑϕʔε • ॻ͖ࠐΈɾಡΈࠐΈͷQuorumͰ߹ੑ͕ܾ·Δ • ΩʔΛࢦఆͨ͠ಡΈࠐΈڧ߹ੑ͕બՄೳ
͑Δ͖ϙΠϯτ
Partition KeyͷϋογϡͰࢄ Like Hash Table(Index)
• Partition Keyʹ࿈൪Λ͏ͷԿͷҙຯͳ͍͜ͱ͕͔Δ • ࿈൪ΧϯλʔΛ࡞ΔͱPartition͕ภΔ • Max + 1ͳΜͯҎͯͷ֎ •
ཧతʹҰҙͱͳΔଐੑ͕͋ΔͳΒͦΕΛ͏ • ͳ͚ΕUUIDͷΑ͏ͳিಥՄೳੑ͕ۃখͳϥϯμϜͰྑ͍ • SortͰ͖ͳ͍ͷͰSnowflakeͷΑ͏ͳॱংੑཁΒͳ͍ • User IDCognitoͰೝূ͍ͯ͠ΔͳΒCognito͕ൃߦ͢ΔIDΛ͏ • ΞΫηε੍ޚʹ͑Δ ͜ͷલఏΛ౿·͑Δͱ
• 3000RCU, 1000WCU, Index Size 10GB (2018-10-31ݱࡏ) • ͲΕ͔ҰͭͰա͢ΔͱPartitionׂ͕͞ΕΔ •
ׂ͞ΕΔͱ֬อͨ͠Ϣχοτ͕ۉʹׂΓৼΒΕΔ • εϩοτϦϯά࣌ͷҰ࣌తͳόʔετ͕͋Δ • ݶఆతͳόʔετ • ༨Λճ͢ (Adaptive Capacity) Partition͝ͱͷΩϟύγςΟ
B+TREE INDEX B-treeͳͷ͔B+treeͳͷ͔ެද͞Ε͍ͯͳ͍͚Ͳ Range Keyͱ͔ݺͿ͘Β͍͔ͩΒ…Ͷʁ
TARGETING QUERY B+treeͷߏΛ͍ͬͯΕ Έ߹ΘͤΩʔͷΠϯσοΫε͕ແͯ͘ ͜͏Εྑ͍͜ͱ͕͔Δ
Partition KeyͰࢄͨ͠தͰSort KeyͰྻ ͜ͷΠϝʔδΛ࣋ͭ͜ͱ͕ͱͯॏཁ
• Partition KeyͰࢄͨ͠ΠϯελϯεʹIndex͕ߏங͞ΕΔ • ҰͭͷΠϯελϯεͰಉظతʹIndexߋ৽͞ΕΔ • ڧ߹ੑ͕બՄೳ • Primary Sort
KeyͷϦʔϑϊʔυʹ࣮σʔλͷΞυϨε͕ ֨ೲ͞Ε͍ͯΔΠϝʔδʢͱࢥΘΕΔʣ • LSI࣮σʔλΛ࣋ͨͳ͍ͷͰॏෳՄೳ Primary Sort Key, LSI
GSI࣮ςʔϒϧͷࣹӨ
• ू߹ʹ͓͚ΔࣹӨ • GSIͷPartition KeyΛݩʹࢄ ʢߋ৽ඇಉظͰద༻ʣ • ϦʔϑϊʔυʹଐੑͱPrimary Key ʢৄ͘͠ޙड़ʣ
• ιʔτࡁΈͷσʔλͷ෦ෳ ʢҟೝΊΔʣ ࣹӨ ࡶͳMySQLύϑΥʔϚϯενϡʔχϯά from yoku0825 https://www.slideshare.net/yoku0825/mysql-57449062/62
• KEYS_ONLY: Primary KeyͷͷΈ • INCLUDE: Primary Keyͷ + બͨ͠ଐੑ
• ALL: શͯͷଐੑ • GSIʹؚΊͨଐੑPrimary KeyΛҾ͖͢ඞཁ͕ͳ͘ͳΔ • ಡΈࠐΈͷޮԽ • ߋ৽ίετͱͷτϨʔυΦϑ • MySQLPostgreSQLͰݴ͏Covering Index / Index Only ScanͷΠϝʔδ GSI͕ࣹӨ͢ΔଐੑͷΦϓγϣϯ
• ϝΠϯςʔϒϧͱผͷPartitionͰࢄ͞ΕΔ • ߋ৽ඇಉظͰө • ݁Ռ߹ੑͬͯେৎͳͷʁ • RDSͩͬͯRead ReplicaಡΜͩΒ݁Ռ߹ੑͰ͢Αʁʁ •
GSIͷΫΤϦ͍͍ͩͨRead Replicaʹ͛ΔΑ͏ͳΫΤϦ GSIʹର͢ΔΫΤϦ݁Ռ߹ੑ
ΈΛΕ࣮ʹγϯϓϧ શ෦RDBͰطʹ͋ͬͨΈͰ͢Αʁʁ
☓ DynamoDBบ͕͋Δ ◦ RDBͷบ͕͋Γ͗͢Δ →γϯϓϧ͗͢ΔDynamoDBͱͷΪϟοϓ͕ڧͯͦ͘͏ײ͡Δ͚ͩ ɹRDBʹચ͞Ε͍ͯΔΜ͡Όͳ͍Ͱ͔͢ʁʁ
DynamoDBͷσʔλϞσϦϯάͱ ͭ·Δͱ͜Ζ
σʔλͷஔΛܾΊΔ͜ͱ ҟೝΊΔʂʂ
σʔλϞσϦϯάͷߟ͑ํͷϙΠϯτ ஔΛܾΊΔͨΊͷ
• εΩʔϚϨεͷҙຯΛߟ͑Δ • List, Map͕࣋ͯΔ͜ͱʁ • ޙ͔ΒଐੑΛγʔϜϨεʹՃͰ͖Δ͜ͱʁ • A. ͳΜͰಥͬࠐΊΔ͜ͱ
• Partition KeyͰࢄ • ςʔϒϧΛ͚ͳͯ͘ෛՙࢄ͢Δ • ۃ͢Δͱ͚Δҙຯ͕ͳ͍ɺ͚ͳ͍ํ͕ΩϟύγςΟ͕ཧ͍͢͠ • “ઃܭ͕༏ΕͨΞϓϦέʔγϣϯͰɺඞཁͳςʔϒϧ 1 ͭͷΈͰ͢” • GSIͷݸ੍ݶ(5ݸ)ͳͲ͋ΔͷͰۀυϝΠϯຖʹҰͭ͘Β͍ͷؾ࣋ͪ ςʔϒϧͷ͚ํ
• RDB • ·ͣεΩʔϚΛߟ͑Δ • ਖ਼نԽ • ͦΕʹରͯ͠Ͳ͏ΞΫηε͢Δ͔(SQL)ߟ͑Δ • DynamoDB
• εΩʔϚϨε • ඇਖ਼نԽ • ·ͣΞΫηεΛߟ͑ͯɺͦΕʹ߹ΘͤͨσʔλΛ࡞Δ • ΞϓϦέʔγϣϯͱσʔλϞσϧಉ࣌ʹઃܭ͢Δ RDBͱΞϓϩʔν͕શ͘ҟͳΔ
ACIDτϥϯβΫγϣϯ͕ແ͍ͷͲ͏͢Δͷʁ ͱ͍͑
• ݪࢠੑ (Atomicity) • Ұ؏(߹)ੑ (Consistency) • ಠཱੑ (Isolation) •
Ӭଓੑ (Durability) ͦͦACIDͬͯͳΜ͚ͩͬʁ ͜ΕΒΛ୲อ͢Δ͜ͱ͕తͰ͋ͬͯ τϥϯβΫγϣϯγεςϜͰ͋Δඞཁແ͍
• RDB • ਖ਼نԽ͍ͯ͘͠ͱςʔϒϧͷجຊతʹ૿͑Δ • ཧతʹ͔Εͨςʔϒϧ(Ϩίʔυ)ؒͷ߹ੑΛऔΔΈ͕͋Δ • ϦϨʔγϣϯγοϓ • ֎෦Ωʔ੍
• ACIDτϥϯβΫγϣϯ • DynamoDB • ඇਖ਼نԽͯ͠1ͭͷ࣮ମʹ͍ͭͯ1ͭͷΞΠςϜʹऩΊΔ • 1ΞΠςϜͷߋ৽ΞτϛοΫ ACIDͷ୲อ
• PutItem • ΞΠςϜΛ·Δ͝ͱߋ৽͢Δɺແ͔ͬͨΒ࡞Δ • Get -> ॻ͖͑ -> Putͩͱಠཱੑ่͕ΕΔ
• UpdateItem • ΞΠςϜΛ෦ߋ৽ɺແ͔ͬͨΒ࡞Δ • List, Mapͷ෦ՃɺআͳͲՄೳ • ଐੑͷΛར༻ͨ͠ߋ৽Մೳʢ͋Δଐੑʹ+1͢Δ) • ଐੑͷΛར༻͖ͨ݅͠ॻ͖ࠐΈ DynamoDBͷߋ৽ૢ࡞
ඇਖ਼نԽ͢ΔͱಡΈࠐΈ͕ͭΒ͍ ϑϧεΩϟϯ͢Δ͔͠ͳ͍ શ෦GSIషΔͷʁ
CQRSͰ͢Α ͦ͜Ͱ
• ॻ͖ࠐΉσʔλͱಡΈࠐΉσʔλಉ͡Ͱ͋Δඞཁແ͍ • ݁Ռ߹ੑΛड͚ೖΕͯඇಉظͰಡΈࠐΈσʔλΛ࡞ΕΔ • CommandͷྃΛͬͯQuery༻σʔλΛ࡞ΕΔ • DynamoDB Streams͕Ϛον͢Δ •
CQRSΛૉʹ࣮͢Δͱ·ͣKinesis Streams͚ͩͲ • Materialized View CQRS (Command Query Responsibility Segregation)
ARCHITECTURE PATTERN AppSync͕͜Ε͔ΒॏཁͳҐஔΛΊΔͣ
DATA ATOMICITY MicroserviesతͳΠϝʔδͰ͜ͷݪ୯Ґ͕ॏཁ σʔλετΞɾςʔϒϧͷ͚ํͰͳ͍ ލ͕͍ͤͨͳΒMaterialized ViewΛ࡞Δ User Profile User Status
Name Email Read/Write Search Read/Write
͜ΜͳΠϝʔδ 1ͭͷGSIͰ༷ʑͳݕࡧ͕Ͱ͖Δ
• ෳͷ࣮ମʹލΔτϥϯβΫγϣϯ • DynamoDB୯ମͩͱ͜͜Ͳ͏ͯͭ͠Β͍ • τϥϯβΫγϣϯϨϕϧͷ࠷ߴSelializable • ͭ·ΓྻԽ • Kinesis
StreamsͰྻԽ • ίέΔ͔͠Εͳ͍ͷͰႈʹ࣮͢Δ • ݅ॻ͖ࠐΈͰطʹߦͬͨॲཧΛSkip͢Δͷႈ • ͱ͍͏͔ɺKinesis StreamsͰಉ࣌ଓΛίϯτϩʔϧ͢ΕRDSͬͨͬͯྑ͍Θ͚Ͱ… ͱ͍͑ݶք͋Δ
ADVANCED TIPS ͍͔ͭ͘͝հ
• Partition Keyͷύλʔϯ͕গͳ͍߹ • ࢄ͠ͳ͘ͳͬͯ͠·͏ • 1PartitionͷΩϟύγςΟΛӽ͑ͦ͏ͳΒγϟʔσΟϯά͢Δ • KeyҎ֎ͷࣗ໌ͳଐੑ͔ΒܭࢉՄೳͳSuffixΛ༩͢Δ •
ूܭൣғݕࡧͷඞཁ͕ͳ͚ΕHash • ඞཁ͕͋ΕMod GSIͷγϟʔσΟϯά
͜ΜͳΠϝʔδ Unicode PointΛ߹ܭͯ͠200Ͱ ׂͬͨ༨ΓΛ༻͍ͯ͠Δ
• DynamoDBͷIndexSparse • Indexଐੑ͕ଘࡏ͠ͳ͍߹ɺIndexʹͦͷσʔλؚ·ͳ͍ • ಛఆͷΞΠςϜͷΈʹଘࡏ͢ΔଐੑΛࢦఆ͢Δ͜ͱͰ Index༰ྔ͕େ͖͘ѹॖͰ͖Δ Sparse Index
SortཁΒͳ͍ GSIʹ҉ͰPrimary Keyؚ͕·ΕΔ
• Α͋͘Δ֊ߏΛ࣋ͭσʔλ • ৫ߏ • ωετͨ͠ΧςΰϥΠζ • ͳͲͳͲ • ݕࡧΛߟ͑ͳ͚Ε֊ߏΛ·Δͬͱ1ΞΠςϜʹԡ͠ࠐΉ
• Sort KeyͰ֊Λදݱ͢Δ Hierarchical data
Sort KeyͰલํҰகݕࡧ ͋ɺ͜ΕSQLΞϯνύλʔϯͰʢ͈́ ઃܭ͕ࡶͳͷ͝Ѫᇷ
“DynamoDB ދͷר” [ݕࡧ] http://marcy.hatenablog.com/entry/2018/07/31/213705 ҰԠະͳͷͰ·ͩॻ͔͘
“LET’S BECOME THE MASTER OF DYNAMODB” THANKS!! “TO BECOME THE
MASTER OF SERVERLESS”