Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Serverless??そんなことよりDBだ!! / We should know Databases before Serverless
Search
Masashi Terui
November 03, 2018
Technology
3
3.4k
Serverless??そんなことよりDBだ!! / We should know Databases before Serverless
JAWS Festa 2018の資料です
https://jft2018.jaws-ug.jp/session/729/
Masashi Terui
November 03, 2018
Tweet
Share
More Decks by Masashi Terui
See All by Masashi Terui
What is Jeffy ?? - Why did we create a new framework
marcyterui
2
790
Serverlessの今とこれから / Everything will be Serverless
marcyterui
7
3.3k
見せてやろう…!! Serverlessの本当の力を…!! / Let’s show you the real power of serverless…!!
marcyterui
6
2.4k
re:Invent 2018でServerlessの世界は何が変わったか / What has changed in the world of Serverless at re:Invent 2018??
marcyterui
4
2.7k
実践サーバレス + α / Serverless Practices
marcyterui
2
560
Serverlessを極めるためにDynamoDBデータモデリングを極めよう / Let’s become the master of DynamoDB Data Modeling to become the master of Serverless
marcyterui
12
16k
マイクロサービスを成功させるためのサーバーレスアーキテクチャ設計とNoSQLデータモデリング / Serverless Architecting and NoSQL Data Modeling for Successful Microservices
marcyterui
13
6.6k
何故私達はサーバレスアプリケーションをイベントドリブンで作るべきなのか / Why should we make serverless application as event driven?
marcyterui
8
1.2k
The theory of data modeling to become the friend of DynamoDB
marcyterui
2
210
Other Decks in Technology
See All in Technology
フルリモートワークはエンジニアの夢を叶えたか? #cm_odyssey
mamohacy
2
600
成長期に歩みを止めないための創業期の開発文化形成
mayah
6
420
運用改善、不都合な真実 / 20240722-ssmjp-kaizen
opelab
17
8k
Azure AI ことはじめ
tsubakimoto_s
0
130
MySQLのロックの種類とその競合
yoku0825
6
1.6k
Github Actions 로 Android 팀의 효율성 극대화
hadonghyun
0
160
Azure OpenAI Service Dev Day / LLMでできる!使える!生成AIエージェント
masahiro_nishimi
3
750
テストケースの自動生成に生成AIの導入を試みた話と生成AIによる今後の期待
shift_evolve
0
180
サービス開発を前に進めるために 新米リードエンジニアが 取り組んだこと / Steps Taken by a Novice Lead Engineer to Advance Service Development
nologyance
0
180
Flutter研修【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
160
[NIKKEI Tech Talk]Bias for Action!! 実践から学ぶための仕組とコミュニティ / Community for Practice and Learning
kanamasa
0
270
AIアシスタントの活用で品質の向上と開発ワークフローのスピードアップ
nagix
1
200
Featured
See All Featured
Optimising Largest Contentful Paint
csswizardry
18
2.6k
Build The Right Thing And Hit Your Dates
maggiecrowley
28
2.2k
Writing Fast Ruby
sferik
623
60k
Building Adaptive Systems
keathley
34
2k
Six Lessons from altMBA
skipperchong
24
3.2k
How GitHub (no longer) Works
holman
305
140k
Practical Orchestrator
shlominoach
185
10k
Done Done
chrislema
179
15k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
16
1.6k
Producing Creativity
orderedlist
PRO
340
39k
The Language of Interfaces
destraynor
151
23k
A Philosophy of Restraint
colly
200
16k
Transcript
SERVERLESSʁʁ ͦΜͳ͜ͱΑΓDBͩʂʂ MASASHI TERUI @ JAWS FESTA OSAKA 2018
SERVERWORKS CO.,LTD. + FREELANCER • Serverless Oji-san • Serverless Framework
Plugin Developer • Serverlessconf Tokyo 2016,2017,2018 speaker • Remote worker (in Sapporo-shi, Hokkaido) • http://marcy.hatenablog.com/entry/2018/08/04/231241 MASASHI TERUI ARCHITECT / DEVELOPER
SERVERLESSͬͯ·͔͢ʁ DynamoDBͬͯ·͔͢ʁ
• ͖ • ී௨ • ݏ͍ • LambdaͱRDSͷ૬ੑ͕ѱ͍͔Βํͳ͖͘߹͍ͬͯΔ DynamoDB͖Ͱ͔͢ʁ
ํ͕ͳ͍͔ΒDynamoDB…… ຊRDBΛ͍͍͚ͨͲ
RDBʹ͑ͯΔΜ͡Όͳ͍Ͱ͔͢ʁʁ NoSQL͕͑ͳ͍Μ͡Όͳͯ͘
RDBͷ͜ͱͪΌΜͱ͔ͬͯ·͔͢ʁʁ ͦͦ
• ίωΫγϣϯϞσϧ • Ӭଓత • ίωΫγϣϯϓʔϦϯά • ηΩϡϦςΟ • ΈࠐΈͷऑ͍ೝূ
• VPCͷΞΫηείετʢENIੜॲཧʣ LambdaͱRDSͷ૬ੑ͕Կނѱ͍ͷ͔આ໌Ͱ͖·͔͢ʁ ͱͯOLTPͰ͑ͳ͍(ඵʙेඵ)
DynamoDBͷ͜ͱ͔ͬͯ·͔͢ʁʁ ͡Ό͋
σʔλϕʔεͷ͜ͱ͔ͬͯ·͔͢ʁʁ ͱ͍͏͔
• ςʔϒϧઃܭͱ͔ΠϯσοΫεઃܭͬͯͲ͏Δͷʁ • ACIDτϥϯβΫγϣϯແ͍ͷʹେৎͳͷʁ • ݁Ռ߹ੑͬͯେৎͳͷʁ • ্ख͘ੑೳΛҾ͖ग़͢ʹͲ͏͢Εྑ͍ʁ DynamoDBͷΑ͋͘Δෆ҆
• ςʔϒϧઃܭͱ͔ΠϯσοΫεઃܭͬͯͲ͏Δͷʁ • ACIDτϥϯβΫγϣϯແ͍ͷʹେৎͳͷʁ • ݁Ռ߹ੑͬͯେৎͳͷʁ • ্ख͘ੑೳΛҾ͖ग़͢ʹͲ͏͢Εྑ͍ʁ DynamoDBͷΑ͋͘Δෆ҆ ͔Βͳ͍͔Βෆ҆ʹͳΔ
• AWS͕ఏڙ͢ΔNoSQLαʔϏε • KeyʹΑΔશҰகͱIndex୳ࡧ͕Մೳ • ϑϧϚωʔδυ • ແ੍ݶͷࣗಈ֦ுετϨʔδʢ͚ͬͨͩʣ • ΠϯελϯεͷཧෆཁɺWrite/Readͷੑೳ͚ͩΛࢦఆ͢Δ
• ͰɺΩϟύγςΟͷࣄલ༧͕ඞཁͳαʔϏεServerless͡Όͳ͍ͬͯCNCF͞Μ͕ݴͬɾɾɾ͋ɺ୭͔དྷͨΈ͍ͨ DynamoDBͷجຊ
DynamoDBͲΜͳ ”AWSαʔϏε” ͔ Ͱͳ͘
DynamoDBͲΜͳ ”σʔλϕʔε” ͔ ͬͯ·͔͢ʁ
• Consistent Hashing • Vector clocks with reconciliation during reads
• Sloppy Quorum and hinted handoff • Anti-entropy using Merkle trees • Gossip-based membership protocol and failure detection Amazon’s Dynamo ৄ͘͠Γ͍ͨͳΒʢBetterͰ͋ͬͯMustͰͳ͍ʣ https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
• ΩʔͷϋογϡͰࢄ → Partition Key • ͦͷࢄͨ͠தͰIndex͕ߏங͞Ε͍ͯΔ → Sort Key
• όοΫάϥϯυͰߋ৽ΛޙউͪͰಉظ • ݁Ռ߹ੑϕʔε • ॻ͖ࠐΈɾಡΈࠐΈͷQuorumͰ߹ੑ͕ܾ·Δ • ΩʔΛࢦఆͨ͠ಡΈࠐΈڧ߹ੑ͕બՄೳ ͑Δ͖ϙΠϯτ
Partition KeyͷϋογϡͰࢄ ͋ɺ͜ΕRDB(Hash Index)Ͱʢ͈́
• Partition Keyʹ࿈൪Λ͏ͷԿͷҙຯͳ͍͜ͱ͕͔Δ • ࿈൪ΧϯλʔΛ࡞ΔͱPartition͕ภΔ • Max + 1ͳΜͯҎͯͷ֎ •
ཧతʹҰҙͱͳΔଐੑ͕͋ΔͳΒͦΕΛ͏ • ͳ͚ΕUUIDͷΑ͏ͳিಥՄೳੑ͕ۃখͳϥϯμϜͰྑ͍ • SortͰ͖ͳ͍ͷͰSnowflakeͷΑ͏ͳॱংੑཁΒͳ͍ • User IDCognitoͰೝূ͍ͯ͠ΔͳΒCognito͕ൃߦ͢ΔIDΛ͏ • ΞΫηε੍ޚʹ͑Δ ͜ͷલఏΛ౿·͑Δͱ
• 3000RCU, 1000WCU, Index Size 10GB (2018-10-31ݱࡏ) • ͲΕ͔ҰͭͰա͢ΔͱPartitionׂ͕͞ΕΔ •
ׂ͞ΕΔͱ֬อͨ͠Ϣχοτ͕ۉʹׂΓৼΒΕΔ • εϩοτϦϯά࣌ͷҰ࣌తͳόʔετ͕͋Δ • ݶఆతͳόʔετ • ༨Λճ͢ (Adaptive Capacity) Partition͝ͱͷΩϟύγςΟ
Partition KeyͰࢄɺSort Keyʁ
B+TREE INDEX B-treeͳͷ͔B+treeͳͷ͔ެද͞Ε͍ͯͳ͍͚Ͳ Range Keyͱ͔ݺͿ͘Β͍͔ͩΒ…Ͷʁ ͋ɺ͜ΕRDBͰʢ͈́
TARGETING QUERY B+treeͷߏΛ͍ͬͯΕ Έ߹ΘͤΩʔͷΠϯσοΫε͕ແͯ͘ ͜͏Εྑ͍͜ͱ͕͔Δ
Partition KeyͰࢄͨ͠தͰSort KeyͰྻ ͜ͷΠϝʔδΛ࣋ͭ͜ͱ͕ͱͯॏཁ
• Partition KeyͰࢄͨ͠ΠϯελϯεʹIndex͕ߏங͞ΕΔ • ҰͭͷΠϯελϯεͰಉظతʹIndexߋ৽͞ΕΔ • ڧ߹ੑ͕બՄೳ • Primary Sort
KeyͷϦʔϑϊʔυʹ࣮σʔλͷΞυϨε͕ ֨ೲ͞Ε͍ͯΔΠϝʔδʢͱࢥΘΕΔʣ • LSI࣮σʔλΛ࣋ͨͳ͍ͷͰॏෳՄೳ Primary Sort Key, LSI
GSI࣮ςʔϒϧͷࣹӨ
• ू߹ʹ͓͚ΔࣹӨ • GSIͷPartition KeyΛݩʹࢄ ʢߋ৽ඇಉظͰద༻ʣ • ϦʔϑϊʔυʹଐੑͱPrimary Key ʢৄ͘͠ޙड़ʣ
• ιʔτࡁΈͷσʔλͷ෦ෳ ʢҟೝΊΔʣ ࣹӨ ࡶͳMySQLύϑΥʔϚϯενϡʔχϯά from yoku0825 https://www.slideshare.net/yoku0825/mysql-57449062/62
• KEYS_ONLY: Primary KeyͷͷΈ • INCLUDE: Primary Keyͷ + બͨ͠ଐੑ
• ALL: શͯͷଐੑ • GSIʹؚΊͨଐੑPrimary KeyΛҾ͖͢ඞཁ͕ͳ͘ͳΔ • ಡΈࠐΈͷޮԽ • ߋ৽ίετͱͷτϨʔυΦϑ • MySQLPostgreSQLͰݴ͏Covering Index / Index Only ScanͷΠϝʔδ GSI͕ࣹӨ͢ΔଐੑͷΦϓγϣϯ
ࣹӨͱ͍͏͜ͱ 1:Nͷ͕ؔ͜ͷΑ͏ʹղܾͰ͖Δ
• ϝΠϯςʔϒϧͱผͷPartitionͰࢄ͞ΕΔ • ߋ৽ඇಉظͰө • ݁Ռ߹ੑͬͯେৎͳͷʁ • RDSͩͬͯRead ReplicaಡΜͩΒ݁Ռ߹ੑͰ͢Αʁʁ •
GSIͷΫΤϦ͍͍ͩͨRead Replicaʹ͛ΔΑ͏ͳΫΤϦ GSIʹର͢ΔΫΤϦ݁Ռ߹ੑ
͋ɺ͜ΕRDBͰʢԿճʁ
ΈΛΕ࣮ʹγϯϓϧ શ෦RDBͰطʹ͋ͬͨΈͰ͢Αʁʁ
☓ DynamoDBบ͕͋Δ ◦ RDBͷบ͕͋Γ͗͢Δ →γϯϓϧ͗͢ΔDynamoDBͱͷΪϟοϓ͕ڧͯͦ͘͏ײ͡Δ͚ͩ
RDBʹચ͞ΕͯΔΜ͡Όͳ͍Ͱ͔͢ʁʁ ͱ͍͏͔ͦͷRDBͷ͜ͱ͢ΒͪΌΜͱ͔ͬͯ͋͛ͯ·͔ͨ͠ʁ
RDBͷ͜ͱ͔ͬͯ·͔ͨ͠ʁ ͱ͍͏͔ɺͦͷେ͖ͳ
DynamoDBͷσʔλϞσϦϯά ͰͦΕΛ౿·͑ͯ
ͦΕσʔλͷஔΛܾΊΔ͜ͱ ҟೝΊΔʂʂ
σʔλϞσϦϯάͷߟ͑ํͷϙΠϯτ ஔΛܾΊΔͨΊͷ
• εΩʔϚϨεͷҙຯΛߟ͑Δ • List, Map͕࣋ͯΔ͜ͱʁ • ޙ͔ΒଐੑΛγʔϜϨεʹՃͰ͖Δ͜ͱʁ • A. ͳΜͰಥͬࠐΊΔ͜ͱ
• Partition KeyͰࢄ • ςʔϒϧΛ͚ͳͯ͘ෛՙࢄ͢Δ • ۃ͢Δͱ͚Δҙຯ͕ͳ͍ɺ͚ͳ͍ํ͕ΩϟύγςΟ͕ཧ͍͢͠ • “ઃܭ͕༏ΕͨΞϓϦέʔγϣϯͰɺඞཁͳςʔϒϧ 1 ͭͷΈͰ͢” • GSIͷݸ੍ݶ(5ݸ)ͳͲ͋ΔͷͰۀυϝΠϯຖʹҰͭ͘Β͍ͷؾ࣋ͪ ςʔϒϧͷ͚ํ
• RDB • ·ͣεΩʔϚΛߟ͑Δ • ਖ਼نԽ • ͦΕʹରͯ͠Ͳ͏ΞΫηε͢Δ͔(SQL)ߟ͑Δ • DynamoDB
• εΩʔϚϨε • ඇਖ਼نԽ • ·ͣΞΫηεΛߟ͑ͯɺͦΕʹ߹ΘͤͨσʔλΛ࡞Δ • ΞϓϦέʔγϣϯͱσʔλϞσϧಉ࣌ʹઃܭ͢Δ RDBͱΞϓϩʔν͕શ͘ҟͳΔ
ACIDτϥϯβΫγϣϯ͕ແ͍ͷͲ͏͢Δͷʁ ͱ͍͑
• ݪࢠੑ (Atomicity) • Ұ؏(߹)ੑ (Consistency) • ಠཱੑ (Isolation) •
Ӭଓੑ (Durability) ͦͦACIDͬͯͳΜ͚ͩͬʁ ͜ΕΒΛ୲อ͢Δ͜ͱ͕తͰ͋ͬͯ τϥϯβΫγϣϯγεςϜͰ͋Δඞཁແ͍
• RDB • ਖ਼نԽ͍ͯ͘͠ͱςʔϒϧͷجຊతʹ૿͑Δ • ཧతʹ͔Εͨςʔϒϧ(Ϩίʔυ)ؒͷ߹ੑΛऔΔΈ͕͋Δ • ϦϨʔγϣϯγοϓ • ֎෦Ωʔ੍
• ACIDτϥϯβΫγϣϯ • DynamoDB • ඇਖ਼نԽͯ͠1ͭͷ࣮ମʹ͍ͭͯ1ͭͷΞΠςϜʹऩΊΔ • 1ΞΠςϜͷߋ৽ΞτϛοΫ • ͖݅ॻ͖ࠐΈΛָͬͨ؍తϩοΫ ACIDͷ୲อ
• PutItem • ΞΠςϜΛ·Δ͝ͱߋ৽͢Δɺແ͔ͬͨΒ࡞Δ • Get -> ॻ͖͑ -> Putͩͱ߹ੑ่͕ΕΔʢՄೳੑ͕͋Δʣ
• UpdateItem • ΞΠςϜΛ෦ߋ৽ɺແ͔ͬͨΒ࡞Δ • List, Mapͷ෦ՃɺআͳͲՄೳ • ଐੑͷΛར༻ͨ͠ߋ৽Մೳʢ͋Δଐੑʹ+1͢Δ) • ଐੑͷΛར༻͖ͨ݅͠ॻ͖ࠐΈ DynamoDBͷߋ৽ૢ࡞
ඇਖ਼نԽ͢ΔͱಡΈࠐΈ͕ͭΒ͍ ϑϧεΩϟϯ͢Δ͔͠ͳ͍ શ෦GSIషΔͷʁ
CQRSͰ͢Α ͦ͜Ͱ
• ॻ͖ࠐΉσʔλͱಡΈࠐΉσʔλಉ͡Ͱ͋Δඞཁແ͍ • ݁Ռ߹ੑΛड͚ೖΕͯඇಉظͰಡΈࠐΈσʔλΛ࡞ΕΔ • CommandͷྃΛͬͯQuery༻σʔλΛ࡞ΕΔ • DynamoDB Streams͕Ϛον͢Δ •
CQRSΛૉʹ࣮͢Δͱ·ͣKinesis Streams͚ͩͲ • Materialized View CQRS (Command Query Responsibility Segregation)
ARCHITECTURE PATTERN AppSync͕͜Ε͔ΒॏཁͳҐஔΛΊΔͣ
DATA ATOMICITY MicroserviesతͳΠϝʔδͰ͜ͷݪ୯Ґ͕ॏཁ σʔλετΞɾςʔϒϧͷ͚ํͰͳ͍ ލ͕͍ͤͨͳΒMaterialized ViewΛ࡞Δ User Profile User Status
Name Email Read/Write Search Read/Write
͜ΜͳΠϝʔδ 1ͭͷGSIͰ༷ʑͳݕࡧ͕Ͱ͖Δ
• ෳͷ࣮ମʹލΔτϥϯβΫγϣϯ • DynamoDB୯ମͩͱ͜͜Ͳ͏ͯͭ͠Β͍ • τϥϯβΫγϣϯϨϕϧͷ࠷ߴSelializable • ͭ·ΓྻԽ • Kinesis
StreamsͰྻԽ • ίέΔ͔͠Εͳ͍ͷͰႈʹ࣮͢Δ • ݅ॻ͖ࠐΈͰطʹߦͬͨॲཧΛSkip͢Δͷႈ ͱ͍͑ݶք͋Δ
• Kinesis StreamsͰಉ࣌ଓΛίϯτϩʔϧ͢Ε ผʹRDSͬͯԿͷͳ͍Θ͚Ͱ • ಉ࣌ଓ → ShardͱಉͰݻఆ • LambdaجຊWarm
Start • ͭ·ΓɺENIͷԆͱ΄΅ແԑ • σϓϩΠShardม͑ͨλΠϛϯάͰى͖͏Δ͚ͲͦͦඇಉظͳͷͰʢ͈́ ͱ͍͏͔
ADVANCED TIPS ͕࣌ؒ͋Ε͍͔ͭ͘͝հ
• DynamoDBͷIndexSparse • Indexଐੑ͕ଘࡏ͠ͳ͍߹ɺIndexʹͦͷσʔλؚ·ͳ͍ • ಛఆͷΞΠςϜͷΈʹଘࡏ͢ΔଐੑΛࢦఆ͢Δ͜ͱͰ Index༰ྔ͕େ͖͘ѹॖͰ͖Δ Sparse Index
SortཁΒͳ͍ GSIʹ҉ͰPrimary Keyؚ͕·ΕΔ
• Α͋͘Δ֊ߏΛ࣋ͭσʔλ • ৫ߏ • ωετͨ͠ΧςΰϥΠζ • ͳͲͳͲ • ݕࡧΛߟ͑ͳ͚Ε֊ߏΛ·Δͬͱ1ΞΠςϜʹԡ͠ࠐΉ
• Sort KeyͰ֊Λදݱ͢Δ Hierarchical data
Sort KeyͰલํҰகݕࡧ ͋ɺ͜ΕSQLΞϯνύλʔϯͰʢ͈́ ઃܭ͕ࡶͳͷ͝Ѫᇷ
• Itemʹ”Version”ͷΑ͏ͳଐੑΛ࣋ͨͤΔ • ଐੑͷΛར༻͖ͨ݅͠ॻ͖ࠐΈ • ConditionExpression=“Version = 1” • UpdateExpression=“ADD
Version 1” • Version = 1Ͱͳ͚ΕΤϥʔ͕ฦΔ • Version = 1ͳΒ2ʹߋ৽͞ΕΔ όʔδϣχϯάͱָ؍తϩοΫ
• తʹݴͬͯόΠϒϧ • ͜ΕΛಡΜͰશͯཧղͰ͖ͨΒҰਓલʂʂ • https://docs.aws.amazon.com/ja_jp/amazondynamodb/latest/ developerguide/best-practices.html DynamoDB Best Practices
“DynamoDB ދͷר” [ݕࡧ] http://marcy.hatenablog.com/entry/2018/07/31/213705 ҰԠະͳͷͰ·ͩॻ͔͘
“LET’S BECOME THE FRIEND OF RDB” THANKS!! “LET’S BECOME THE
FRIEND OF DYNAMODB” “LET’S BECOME THE FRIEND OF SERVERLESS”