Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
AWSで作る、サーバーレスデータ分析基盤構築 / jawsug-niigata-11
Search
kasacchiful
January 15, 2022
Programming
1
390
AWSで作る、サーバーレスデータ分析基盤構築 / jawsug-niigata-11
JAWS-UG新潟#11で発表した資料です。
kasacchiful
January 15, 2022
Tweet
Share
More Decks by kasacchiful
See All by kasacchiful
いろんな世界を見てみよう / 20250508ninno_tech_fest
kasacchiful
0
22
Amazon Q Developer for CLIのある生活 / 20250427ai_craft_hacks_niigata1
kasacchiful
1
65
AWSのコンテナサービス / jawsug-akita-aws-container-services
kasacchiful
0
65
データ基盤でのコンテナ活用事例 / jawsug-akita-data-platform-with-container
kasacchiful
0
66
データ基盤でのコンテナ活用事例 / jawsug-niigata21-data-platform-with-container
kasacchiful
0
100
AWSのコンテナサービス / jawsug_niigata21_aws_container_services
kasacchiful
1
88
Amazon S3 TablesとAmazon S3 Metadataを触ってみた / 20250201-jawsug-tochigi-s3tables-s3metadata
kasacchiful
0
310
Amazon S3 TablesとAmazon S3 Metadataを動かしてみた / 20250125-niigata-5min-tech-lt
kasacchiful
0
33
dbt coreとFargateでデータ変換 / 20240928-jawsug-toyama-hokuriku-shinkansen
kasacchiful
1
130
Other Decks in Programming
See All in Programming
Language Server と喋ろう – TSKaigi 2025
pizzacat83
2
220
プロダクトエンジニアのしごと 〜 受託 × 高難度を乗り越えるOptium開発 〜
algoartis
0
250
ドメイン駆動設計とXPで支える子どもの未来 / Domain-Driven Design and XP Supporting Children's Future
nrslib
0
340
バイラテラルアップサンプリング
fadis
3
650
Design Pressure
hynek
0
190
Live Coding: Migrating an Application to Signals
manfredsteyer
PRO
0
130
Embracing Ruby magic
vinistock
2
300
Носок на сок
bo0om
0
1.5k
クラス設計の手順
akikogoto
0
140
生成AI時代のフルスタック開発
kenn
8
1k
オープンソースコントリビュート入門
_katsuma
0
150
ソフトウェア品質特性、意識してますか?AIの真の力を引き出す活用事例 / ai-and-software-quality
minodriven
17
5.3k
Featured
See All Featured
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Building Better People: How to give real-time feedback that sticks.
wjessup
368
19k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
440
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
105
19k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Stop Working from a Prison Cell
hatefulcrawdad
268
20k
Building a Modern Day E-commerce SEO Strategy
aleyda
40
7.3k
How to Ace a Technical Interview
jacobian
276
23k
Being A Developer After 40
akosma
91
590k
How STYLIGHT went responsive
nonsquared
100
5.5k
Statistics for Hackers
jakevdp
799
220k
Transcript
AWSͰ࡞ΔɺαʔόʔϨε σʔλੳج൫ߏங JAWS-UG৽ׁ#11 2022-01-15 @kasacchiful
Classmethod, Inc. Solutions Architect / Software Develper Favorite: Community: •
JAWS-UG Niigata • Python ML in Niigata • JaSST Niigata • ASTER • SWANII • etc. Hiroshi Kasahara @kasacchiful @kasacchiful 2
αʔόʔϨεͷੳج൫
σʔλੳʹ͓͚Δ֤छAWSαʔϏε
σʔλͷՃʗੳʹ AWS Lambda Մೳ
ෳࡶɾେنͳΒ AWS Step Functions Λ׆༻
αʔόʔϨεύλʔϯ IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO
Ϣʔεέʔεผʹύλʔϯ͕͋Δ IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO
ύλʔϯͷৄࡉBlack BeltͷࢿྉΛࢀߟʹ IUUQTEBXTTUBUJDDPNXFCJOBSTKQQEGTFSWJDFT@"84@#MBDL#FU@4FSWFSMFTT@6TFDBTF@1BUUFSOTQEG :PV5VCFͰͷղઆಈըIUUQTZPVUVCF)*M8ESC@Z.
S3ʹೖΕͯ͠·͑ɺͳΜͱ͔ͳΔ
αʔόʔϨεͰσʔλ࿈ܞ͢Δࡍʹ ϋϚͬͨͱ͜Ζ
Step FunctionsͷεςʔτϚγϯͰLambdaͷ ϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλΛՃ
Step FunctionsͷεςʔτϚγϯͰLambdaͷ ϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλΛՃ σʔλൃੜݩ͔ΒɺσʔλΛऔ ಘͯ͠4ʹอଘ ֤ϑΝΠϧຖʹɺ࠷ݶͷσʔ λՃΛͯ͠ɺ4ʹอଘ 2VJDL4JHIU #* ༻ʹ
ෳϑΝΠϧͷσʔλΛ·ͱΊ ͯదʹܗ͢Δ
͍Ζ͍ΖϋϚͬͨͱ͜Ζ 4ͭհ
1. ಛఆͷσʔλϑΝΠϧଟ͗͢
Έ: ͋ΔಛఆͷσʔλϑΝΠϧ͚ͩҟৗʹଟ͍ • 5ؒͷσʔλ͕1ϑΝΠϧʹ͋Δ • தϛϦඵ୯ҐͷϨίʔυ • ಛఆͷॲཧ͚͕͔͔ͩ࣌ؒΔ
• ݅ଟ͍σʔλɺBIʹग़ྗ͠ͳ͍߲ͩͬͨ • ࣍ॲཧ͔ΒΓͯ͠ɺຖ࣌ॲཧʹมߋ • ࣍ॲཧͷϘτϧωοΫΛআ͍ͨ ରॲ๏: ͋ΔಛఆͷσʔλϑΝΠϧ͚ͩɺຖ࣌ॲ ཧʹมߋ
2. AthenaͷΫΥʔλ
• σʔλҠߦ࣌ʹɺ࣍ॲཧͷ࠷ޙͷLambdaͰΤϥʔʹͳΔ • લஈͰॲཧͨ͠ෳσʔλΛAthenaͬͯSQLΫΤϦͰऔಘ͢Δͱ͜ΖͰ ্ݶʹҾ͔͔ͬΔ • Lambdaؔ1ͭʹ͖ͭɺɹstart-query-executionɹAPIΛ5ճίʔϧ • Ұ࣌తʹόʔετͰ্ݶ80·Ͱ૿͑Δ͚ͲɺσʔλҠߦ࣌ʹ20Ͱ಄ଧͪ •
্ݶ؇ਃ͢Ε্ݶ͋͛ΒΕΔ Έ: AthenaͷΫΤϦಉ࣮࣌ߦͷΫΥʔλʹ Ҿ͔͔ͬΔ
IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM
ରॲ๏: Step Functions ͷMapεςʔτͷ࠷େಉ ࣮࣌ߦΛઃఆ • Mapεςʔτ (ྻ͢ͱɺಉ࣮࣌ߦͰྻཁૉΛॲཧ͢ΔΠϝʔδ) ͷ࠷େಉ࣮࣌ߦΛઃఆ͠ɺAthenaͷ start-query-execution
APIίʔ ϧΛ࠷େ20·Ͱʹ͓͑͞Δ
Mapεςʔτʹ͍ͭͯɺҎԼͷهࣄΛࢀߟʹ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTUFQGVODUJPOTVQEBUFNBQTUBUF IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHBNB[POTUBUFTMBOHVBHFNBQTUBUFIUNM
3. Step FunctionsͷΫΥʔλ
Έ: Step FunctionsͷΠϕϯτཤྺ͕ΫΥʔ λʹҾ͔͔ͬΔ • ͋Δಛఆͷ͚ͩɺຖ࣌ॲཧͷϑΝΠϧ͕ҟৗʹଟ͍ • 1࣌ؒܦͬͯҟৗऴྃɻStep FunctionsͷΠϕϯτཤྺͷ্ݶ౸ୡ (25,000Πϕϯτ)
• ্ݶ؇ෆՄͷ߲ { "error": "States.Runtime", "cause": "The execution reached the maximum number of history events (25000)." }
IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM
ରॲ๏: Step Functions ͷεςʔτϚγϯΛೖΕ ࢠʹ • εςʔτϚγϯΛೖΕࢠʹ͢Δ͜ͱͰɺΠϕϯτཤྺ্ݶʹҾ͔͔ͬ Βͳ͍Α͏ʹͨ͠ • Lambdaͷಉ࣮࣌ߦ͕͔ͳΓ૿͑ΔͷͰɺҎԼͷରԠΛՃ
✓ Lambdaͷಉ࣮࣌ߦͷ্ݶ؇ਃ ✓ Step FunctionsͷMapεςʔτͷ࠷େಉ࣮࣌ߦΛઃఆ
มߋલ มߋޙ
มߋલ มߋޙ
4. Lambdaͷεέʔϧ͕͍͔ͭͳ͍
Έ: 1ճ͚ͩLambdaͷRateLimitΤϥʔʹૺ۰ • ಉ࣮࣌ߦͷΤϥʔͷΑ͏͚ͩͲ… • ͢Ͱʹಉ࣮࣌ߦͷ্ݶΛҾ্͖͍͛ͯΔͷͷɺ֤ؔͷϞχλϦ ϯάݟΔݶΓɺಉ࣮࣌ߦʹ౸ୡ͍ͯ͠ͳ͍ { "error": "Lambda.TooManyRequestsException",
"cause": "Rate Exceeded. (Service: Lambda, Status Code: 429, Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Extended Request ID: null)" }
IUUQTEPDTBXTBNB[PODPNKB@KQMBNCEBMBUFTUEHJOWPDBUJPOTDBMJOHIUNM
ରॲ๏: LambdaؔͷRetryઃఆΛݟ͠ • Step Functions ͷ Mapεςʔτͷ࠷େಉ࣮࣌ߦΛݟ͠ • Step Functions
Ͱఆٛ͢Δ Lambda ͷ Retry ઃఆΛݟ͠
Retry ͷִؒʹ͍ͭͯҎԼͷهࣄ͕ৄ͍͠ $ node -e '((i,m,b)=>{for(let w=i,c=0;c<m;c++){console.log(w+=(c==0?0:b**c))}})(2,7,1.85)' 2 3.85 7.272500000000001
13.604125000000002 25.317631250000005 46.987617812500005 87.07709295312502 IUUQTEFWDMBTTNFUIPEKQBSUJDMFTXBJU@UJNF@BOE@QBSBNT@JO@TUFQ@GVODUJPO@SFUSZ
Lambda ͷ Provisioned Concurrency ઃఆࠓճ ࣮ࢪͯ͠ͳ͍ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTMBNCEBQSPWJTJPOFEDPODVSSFODZDPMETUBSU
σʔλͷՃʹ AWS Glueͱ͍͏αʔϏε͋ΔΑʁ
σʔλͷՃͳΒGlue͕͋Δ GlueΘͣʹɺΘ͟Θ͟Step Functions + LambdaͰΉඞཁ͋Δͷ͔ʁ • Step Functions + Lambdaͷ߹ɺΑ͘ΘΕΔ։ൃϑϨʔϜϫʔΫ͕͑ΔͷͰɺෳਓ
Ͱͷ։ൃ͕͍͢͠ɻ ✓ ࠓճ Serverless Framework ͬͨɻ • σʔλϑΝΠϧ͕ଟͯ͘ɺσʔλ1݅͋ͨΓͷ༰ྔ͕ͦ͜·Ͱେ͖͘ͳ͚Εɺ࣍ ୈͰLambdaͰॲཧ͕Ͱ͖Δɻ • LambdaͰΓΕͳ͍σʔλ༰ྔ࣮ߦ࣌ؒΛѻ͏߹ɺGlueͬͨํ͕͍͍ɻ ✓ ࠷େϝϞϦׂ: 10240MBɺ࠷େ࣮ߦ࣌ؒ: 15ɺ /tmp σΟϨΫτϦαΠζ: 512MB
͓·͚
͓·͚: AWS Data Wrangler͕ศར IUUQTHJUIVCDPNBXTMBCTBXTEBUBXSBOHMFS
͓·͚: AWS Data Wrangler͕ศར PandasͷػೳΛAWSʹ֦ு͢ΔɺΦʔϓϯιʔεͷPythonϥΠϒϥϦ • PandasσʔλϑϨʔϜͱAWSͷσʔλؔ࿈ͷαʔϏεͱΛ͏·͘ଓͯ͘͠Ε Δ ✓ Redshift
/ Glue / Athena / EMR ͳͲ • ௨ৗͷETLλεΫʹඞཁͳ͕ؔἧ͍ͬͯΔ
ҙ: ϑΝΠϧαΠζ͕େ͖ͯ͘ɺͦͷ··ͩ ͱLambdaʹΒͳ͍ • LambdaͷσϓϩΠύοέʔδඇѹॖ࣌ʹ250MBҎԼʹ͢Δඞཁ͕͋Δ ✓ AWS Data WranglerΛී௨ʹpipΠϯετʔϧ͢Δͱɺ250MB͑Δ •
GitHubͷReleaseϖʔδʹ͋ΔɺLambda Layer༻ͷzipϑΝΠϧΛར༻͠Α͏
·ͱΊ • αʔόʔϨεαʔϏεΛۦͯ͠ɺσʔλੳج൫ΛߏஙՄೳ • αʔόʔϨεͷΑ͋͘ΔΞʔΩςΫνϟύλʔϯΛ͏·͍͘͜ͳ͠ ·͠ΐ͏
͓͠·͍