Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
AWSで作る、サーバーレスデータ分析基盤構築 / jawsug-niigata-11
Search
kasacchiful
January 15, 2022
Programming
1
400
AWSで作る、サーバーレスデータ分析基盤構築 / jawsug-niigata-11
JAWS-UG新潟#11で発表した資料です。
kasacchiful
January 15, 2022
Tweet
Share
More Decks by kasacchiful
See All by kasacchiful
Amazon Q Developer for CLI を使って PHP Conference 新潟 2025 参加者向けにグルメサイトを構築した話 / 20250620niigata-5min-tech
kasacchiful
0
55
ワイがおすすめする新潟の食 / 20250530phpconf-niigata-eve
kasacchiful
0
310
生成AIでメタデータを生成してみた / 20250525generate-metadata-using-generative-ai
kasacchiful
0
55
Strands Agents SDK で AIエージェント作成 を試してみた / 20250525strands-agents
kasacchiful
0
170
いろんな世界を見てみよう / 20250508ninno_tech_fest
kasacchiful
0
32
Amazon Q Developer for CLIのある生活 / 20250427ai_craft_hacks_niigata1
kasacchiful
1
83
AWSのコンテナサービス / jawsug-akita-aws-container-services
kasacchiful
0
71
データ基盤でのコンテナ活用事例 / jawsug-akita-data-platform-with-container
kasacchiful
0
77
データ基盤でのコンテナ活用事例 / jawsug-niigata21-data-platform-with-container
kasacchiful
0
120
Other Decks in Programming
See All in Programming
GitHub Copilot and GitHub Codespaces Hands-on
ymd65536
2
150
A full stack side project webapp all in Kotlin (KotlinConf 2025)
dankim
0
120
ニーリーにおけるプロダクトエンジニア
nealle
0
830
AIともっと楽するE2Eテスト
myohei
6
2.6k
PHPで始める振る舞い駆動開発(Behaviour-Driven Development)
ohmori_yusuke
2
390
効率的な開発手段として VRTを活用する
ishkawa
0
140
Flutterで備える!Accessibility Nutrition Labels完全ガイド
yuukiw00w
0
160
#kanrk08 / 公開版 PicoRubyとマイコンでの自作トレーニング計測装置を用いたワークアウトの理想と現実
bash0c7
1
770
初学者でも今すぐできる、Claude Codeの生産性を10倍上げるTips
s4yuba
16
11k
git worktree × Claude Code × MCP ~生成AI時代の並列開発フロー~
hisuzuya
1
570
設計やレビューに悩んでいるPHPerに贈る、クリーンなオブジェクト設計の指針たち
panda_program
6
2.1k
WebViewの現在地 - SwiftUI時代のWebKit - / The Current State Of WebView
marcy731
0
120
Featured
See All Featured
Building a Modern Day E-commerce SEO Strategy
aleyda
42
7.4k
The Invisible Side of Design
smashingmag
301
51k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Testing 201, or: Great Expectations
jmmastey
43
7.6k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
46
9.6k
Typedesign – Prime Four
hannesfritz
42
2.7k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
Music & Morning Musume
bryan
46
6.6k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
How to Think Like a Performance Engineer
csswizardry
25
1.7k
How to train your dragon (web standard)
notwaldorf
95
6.1k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
2.9k
Transcript
AWSͰ࡞ΔɺαʔόʔϨε σʔλੳج൫ߏங JAWS-UG৽ׁ#11 2022-01-15 @kasacchiful
Classmethod, Inc. Solutions Architect / Software Develper Favorite: Community: •
JAWS-UG Niigata • Python ML in Niigata • JaSST Niigata • ASTER • SWANII • etc. Hiroshi Kasahara @kasacchiful @kasacchiful 2
αʔόʔϨεͷੳج൫
σʔλੳʹ͓͚Δ֤छAWSαʔϏε
σʔλͷՃʗੳʹ AWS Lambda Մೳ
ෳࡶɾେنͳΒ AWS Step Functions Λ׆༻
αʔόʔϨεύλʔϯ IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO
Ϣʔεέʔεผʹύλʔϯ͕͋Δ IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO
ύλʔϯͷৄࡉBlack BeltͷࢿྉΛࢀߟʹ IUUQTEBXTTUBUJDDPNXFCJOBSTKQQEGTFSWJDFT@"84@#MBDL#FU@4FSWFSMFTT@6TFDBTF@1BUUFSOTQEG :PV5VCFͰͷղઆಈըIUUQTZPVUVCF)*M8ESC@Z.
S3ʹೖΕͯ͠·͑ɺͳΜͱ͔ͳΔ
αʔόʔϨεͰσʔλ࿈ܞ͢Δࡍʹ ϋϚͬͨͱ͜Ζ
Step FunctionsͷεςʔτϚγϯͰLambdaͷ ϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλΛՃ
Step FunctionsͷεςʔτϚγϯͰLambdaͷ ϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλΛՃ σʔλൃੜݩ͔ΒɺσʔλΛऔ ಘͯ͠4ʹอଘ ֤ϑΝΠϧຖʹɺ࠷ݶͷσʔ λՃΛͯ͠ɺ4ʹอଘ 2VJDL4JHIU #* ༻ʹ
ෳϑΝΠϧͷσʔλΛ·ͱΊ ͯదʹܗ͢Δ
͍Ζ͍ΖϋϚͬͨͱ͜Ζ 4ͭհ
1. ಛఆͷσʔλϑΝΠϧଟ͗͢
Έ: ͋ΔಛఆͷσʔλϑΝΠϧ͚ͩҟৗʹଟ͍ • 5ؒͷσʔλ͕1ϑΝΠϧʹ͋Δ • தϛϦඵ୯ҐͷϨίʔυ • ಛఆͷॲཧ͚͕͔͔ͩ࣌ؒΔ
• ݅ଟ͍σʔλɺBIʹग़ྗ͠ͳ͍߲ͩͬͨ • ࣍ॲཧ͔ΒΓͯ͠ɺຖ࣌ॲཧʹมߋ • ࣍ॲཧͷϘτϧωοΫΛআ͍ͨ ରॲ๏: ͋ΔಛఆͷσʔλϑΝΠϧ͚ͩɺຖ࣌ॲ ཧʹมߋ
2. AthenaͷΫΥʔλ
• σʔλҠߦ࣌ʹɺ࣍ॲཧͷ࠷ޙͷLambdaͰΤϥʔʹͳΔ • લஈͰॲཧͨ͠ෳσʔλΛAthenaͬͯSQLΫΤϦͰऔಘ͢Δͱ͜ΖͰ ্ݶʹҾ͔͔ͬΔ • Lambdaؔ1ͭʹ͖ͭɺɹstart-query-executionɹAPIΛ5ճίʔϧ • Ұ࣌తʹόʔετͰ্ݶ80·Ͱ૿͑Δ͚ͲɺσʔλҠߦ࣌ʹ20Ͱ಄ଧͪ •
্ݶ؇ਃ͢Ε্ݶ͋͛ΒΕΔ Έ: AthenaͷΫΤϦಉ࣮࣌ߦͷΫΥʔλʹ Ҿ͔͔ͬΔ
IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM
ରॲ๏: Step Functions ͷMapεςʔτͷ࠷େಉ ࣮࣌ߦΛઃఆ • Mapεςʔτ (ྻ͢ͱɺಉ࣮࣌ߦͰྻཁૉΛॲཧ͢ΔΠϝʔδ) ͷ࠷େಉ࣮࣌ߦΛઃఆ͠ɺAthenaͷ start-query-execution
APIίʔ ϧΛ࠷େ20·Ͱʹ͓͑͞Δ
Mapεςʔτʹ͍ͭͯɺҎԼͷهࣄΛࢀߟʹ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTUFQGVODUJPOTVQEBUFNBQTUBUF IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHBNB[POTUBUFTMBOHVBHFNBQTUBUFIUNM
3. Step FunctionsͷΫΥʔλ
Έ: Step FunctionsͷΠϕϯτཤྺ͕ΫΥʔ λʹҾ͔͔ͬΔ • ͋Δಛఆͷ͚ͩɺຖ࣌ॲཧͷϑΝΠϧ͕ҟৗʹଟ͍ • 1࣌ؒܦͬͯҟৗऴྃɻStep FunctionsͷΠϕϯτཤྺͷ্ݶ౸ୡ (25,000Πϕϯτ)
• ্ݶ؇ෆՄͷ߲ { "error": "States.Runtime", "cause": "The execution reached the maximum number of history events (25000)." }
IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM
ରॲ๏: Step Functions ͷεςʔτϚγϯΛೖΕ ࢠʹ • εςʔτϚγϯΛೖΕࢠʹ͢Δ͜ͱͰɺΠϕϯτཤྺ্ݶʹҾ͔͔ͬ Βͳ͍Α͏ʹͨ͠ • Lambdaͷಉ࣮࣌ߦ͕͔ͳΓ૿͑ΔͷͰɺҎԼͷରԠΛՃ
✓ Lambdaͷಉ࣮࣌ߦͷ্ݶ؇ਃ ✓ Step FunctionsͷMapεςʔτͷ࠷େಉ࣮࣌ߦΛઃఆ
มߋલ มߋޙ
มߋલ มߋޙ
4. Lambdaͷεέʔϧ͕͍͔ͭͳ͍
Έ: 1ճ͚ͩLambdaͷRateLimitΤϥʔʹૺ۰ • ಉ࣮࣌ߦͷΤϥʔͷΑ͏͚ͩͲ… • ͢Ͱʹಉ࣮࣌ߦͷ্ݶΛҾ্͖͍͛ͯΔͷͷɺ֤ؔͷϞχλϦ ϯάݟΔݶΓɺಉ࣮࣌ߦʹ౸ୡ͍ͯ͠ͳ͍ { "error": "Lambda.TooManyRequestsException",
"cause": "Rate Exceeded. (Service: Lambda, Status Code: 429, Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Extended Request ID: null)" }
IUUQTEPDTBXTBNB[PODPNKB@KQMBNCEBMBUFTUEHJOWPDBUJPOTDBMJOHIUNM
ରॲ๏: LambdaؔͷRetryઃఆΛݟ͠ • Step Functions ͷ Mapεςʔτͷ࠷େಉ࣮࣌ߦΛݟ͠ • Step Functions
Ͱఆٛ͢Δ Lambda ͷ Retry ઃఆΛݟ͠
Retry ͷִؒʹ͍ͭͯҎԼͷهࣄ͕ৄ͍͠ $ node -e '((i,m,b)=>{for(let w=i,c=0;c<m;c++){console.log(w+=(c==0?0:b**c))}})(2,7,1.85)' 2 3.85 7.272500000000001
13.604125000000002 25.317631250000005 46.987617812500005 87.07709295312502 IUUQTEFWDMBTTNFUIPEKQBSUJDMFTXBJU@UJNF@BOE@QBSBNT@JO@TUFQ@GVODUJPO@SFUSZ
Lambda ͷ Provisioned Concurrency ઃఆࠓճ ࣮ࢪͯ͠ͳ͍ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTMBNCEBQSPWJTJPOFEDPODVSSFODZDPMETUBSU
σʔλͷՃʹ AWS Glueͱ͍͏αʔϏε͋ΔΑʁ
σʔλͷՃͳΒGlue͕͋Δ GlueΘͣʹɺΘ͟Θ͟Step Functions + LambdaͰΉඞཁ͋Δͷ͔ʁ • Step Functions + Lambdaͷ߹ɺΑ͘ΘΕΔ։ൃϑϨʔϜϫʔΫ͕͑ΔͷͰɺෳਓ
Ͱͷ։ൃ͕͍͢͠ɻ ✓ ࠓճ Serverless Framework ͬͨɻ • σʔλϑΝΠϧ͕ଟͯ͘ɺσʔλ1݅͋ͨΓͷ༰ྔ͕ͦ͜·Ͱେ͖͘ͳ͚Εɺ࣍ ୈͰLambdaͰॲཧ͕Ͱ͖Δɻ • LambdaͰΓΕͳ͍σʔλ༰ྔ࣮ߦ࣌ؒΛѻ͏߹ɺGlueͬͨํ͕͍͍ɻ ✓ ࠷େϝϞϦׂ: 10240MBɺ࠷େ࣮ߦ࣌ؒ: 15ɺ /tmp σΟϨΫτϦαΠζ: 512MB
͓·͚
͓·͚: AWS Data Wrangler͕ศར IUUQTHJUIVCDPNBXTMBCTBXTEBUBXSBOHMFS
͓·͚: AWS Data Wrangler͕ศར PandasͷػೳΛAWSʹ֦ு͢ΔɺΦʔϓϯιʔεͷPythonϥΠϒϥϦ • PandasσʔλϑϨʔϜͱAWSͷσʔλؔ࿈ͷαʔϏεͱΛ͏·͘ଓͯ͘͠Ε Δ ✓ Redshift
/ Glue / Athena / EMR ͳͲ • ௨ৗͷETLλεΫʹඞཁͳ͕ؔἧ͍ͬͯΔ
ҙ: ϑΝΠϧαΠζ͕େ͖ͯ͘ɺͦͷ··ͩ ͱLambdaʹΒͳ͍ • LambdaͷσϓϩΠύοέʔδඇѹॖ࣌ʹ250MBҎԼʹ͢Δඞཁ͕͋Δ ✓ AWS Data WranglerΛී௨ʹpipΠϯετʔϧ͢Δͱɺ250MB͑Δ •
GitHubͷReleaseϖʔδʹ͋ΔɺLambda Layer༻ͷzipϑΝΠϧΛར༻͠Α͏
·ͱΊ • αʔόʔϨεαʔϏεΛۦͯ͠ɺσʔλੳج൫ΛߏஙՄೳ • αʔόʔϨεͷΑ͋͘ΔΞʔΩςΫνϟύλʔϯΛ͏·͍͘͜ͳ͠ ·͠ΐ͏
͓͠·͍