Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
サーバーレスでデータ連携する際にハマったところ / ninno-techfest-20211022-lt
Search
kasacchiful
October 22, 2021
Programming
0
510
サーバーレスでデータ連携する際にハマったところ / ninno-techfest-20211022-lt
2021/10/22 (金) NINNO Tech Fest #2 の LT で発表した資料
kasacchiful
October 22, 2021
Tweet
Share
More Decks by kasacchiful
See All by kasacchiful
AWS Application Composerで始める、 サーバーレスなデータ基盤構築 / 20240406-jawsug-hokuriku-shinkansen
kasacchiful
1
290
AWSの各種サービス紹介と活用方法 − AI・ML活用デモを交えて − / 20231208aws-aiml-seminar
kasacchiful
0
400
Amazon Rekognition デモ / 20231208-aws_seminar-01-rekognition-demo
kasacchiful
0
400
Amazon Lookout for Vision デモ / 20231208-aws_seminar-02-lookout-vision-demo
kasacchiful
0
400
Python機械学習勉強会in新潟のロゴが無いので、生成AIで作ってみましょう / osc2023niigata
kasacchiful
0
320
Amazon Bedrock概要と生成AIの基礎 / 20231118-jawsug-niigata-15
kasacchiful
0
66
生成AIと自然言語処理の基礎 / 20231111-pyml-niigata-18.pdf
kasacchiful
0
150
最近やってる、サーバーレスでデータ分析基盤を構築している話 / 20230916-nds63
kasacchiful
0
77
Glue for Rayを使ってみよう #devio2023 / devio2023-glue-for-ray
kasacchiful
0
730
Other Decks in Programming
See All in Programming
Let's learn code review
riofujimon
2
620
初心者のためのRubyKaigi入門/RubyKaigi Introduction
a_matsuda
10
1.7k
Implementing Design Systems in Swift
seyfoyun
2
500
Next.js App Router
quramy
12
2.1k
Snowflakeで眠ったデータを起こそう!
estie
0
160
Build Apps for iOS, Android & Desktop in 100% Kotlin With Compose Multiplatform (mDevCamp 2024)
zsmb
0
480
Go製Webアプリケーションのエラーとの向き合い方大全、あるいはやっぱりスタックトレース欲しいやん / Kyoto.go #50
utgwkk
6
1.9k
Behind VS Code Extensions for JavaScript / TypeScript Linnting and Formatting
unvalley
6
1.3k
AWS CDKコントリビュートTIPS / aws-cdk-contribution-tips
gotok365
4
540
CREってこういうこと? 体験入社 - 提案資料 - / what-is-cre-trial-employment
shinden
1
570
PostmanでAPIの動作確認が楽になった話
h455h1
0
190
新宿ダンジョンを可視化してみた
satoshi7190
3
420
Featured
See All Featured
A Philosophy of Restraint
colly
197
16k
Music & Morning Musume
bryan
41
5.6k
Thoughts on Productivity
jonyablonski
60
3.9k
KATA
mclloyd
16
12k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
228
16k
The World Runs on Bad Software
bkeepers
PRO
61
6.7k
Robots, Beer and Maslow
schacon
PRO
155
7.9k
Testing 201, or: Great Expectations
jmmastey
30
6.4k
Making the Leap to Tech Lead
cromwellryan
125
8.5k
Fontdeck: Realign not Redesign
paulrobertlloyd
76
4.9k
Optimising Largest Contentful Paint
csswizardry
13
2.4k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
18
7k
Transcript
αʔόʔϨεͰσʔλ࿈ܞ͢Δࡍʹ ϋϚͬͨͱ͜Ζ ּݪ /*//05FDI'FTU
ຊ͓͢͠Δ͜ͱ wαʔόʔϨεͰσʔλ࿈ܞج൫Λߏஙͨ͠ࡍɺϋϚͬͨͱ͜Ζ͕৭ʑ ͋ͬͨͷͰɺ؆୯ʹ͓͠͠·͢ɻ w"84Ͱߏங͍ͯ͠·͢ɻ"84͋·ΓΘ͔Βͳ͍ํɺ༰ࠐΈೖͬͯ ΔͷͰ͍͔͠͠Ε·ͤΜɻ w-5ͳͷʹϖʔδ࡞ͬͨͷͰɺۦ͚ͰਐΈ·͢ɻ
ࣗݾհ w ּݪ w ΫϥεϝιουגࣜձࣾσʔλΞφϦςΟΫεࣄۀຊ෦ w ৽ׁݝ৽ׁࢢࡏॅ w ͖ͳ"84αʔϏε4ɺ-BNCEBɺ4BHF.BLFS
w +"846(৽ׁࢧ෦ʗ1ZUIPOػցֶशษڧձJO৽ׁʗ +B445৽ׁʗ48"/**ʗFUD @kasacchiful @kasacchiful
։࢝ w IUUQTQZNMOJJHBUBDPOOQBTTDPNFWFOU 1ZUIPOػցֶशษڧձJO৽ׁ
w IUUQTKBXTVHOJJHBUBDPOOQBTTDPN +"846(/JJHBUB
+"841"/,3"5*0/ ʙ ͷ࣌ؒΠϕϯτ w IUUQTKBXTQBOLSBUJPOKBXTVHKQ
ຊ αʔόʔϨεͷσʔλج൫
ྫ͑͜Μͳͭ σʔλੳʹ͓͚Δ֤छ"84αʔϏε
ྫ͑͜Μͳͭ 4ʹσʔλೖΕͯɺ-BNCEBͰՃɺ݁ՌΛ4ʹ͓͘ɻ
ྫ͑͜Μͳͭ 4ʹՃࡁσʔλ͓͍ͨΒɺ͋ͱੳʹͲ͏ͧɻ
ࠓճ͜Μͳײ͡ͷͭ 4UFQ'VODUJPOTͷεςʔτϚγϯͰ-BNCEBͷϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλ ΛՃ w ճεέδϡʔϧ࣮ߦ
4UFQ'VODUJPOTͷεςʔτϚγϯͰ-BNCEBͷϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλ ΛՃ w ճεέδϡʔϧ࣮ߦ ࠓճ͜Μͳײ͡ͷͭ σʔλൃੜݩ͔ΒɺσʔλΛऔ ಘͯ͠4ʹอଘ ֤ϑΝΠϧຖʹɺ࠷ݶͷσʔ λՃΛͯ͠ɺ4ʹอଘ
2VJDL4JHIU #* ༻ʹ ෳϑΝΠϧͷσʔλΛ·ͱΊ ͯదʹܗ͢Δ
͍Ζ͍ΖϋϚͬͨ ϋϚͬͨΛ4ͭհ
ϋϚͬͨͦͷ ಛఆͷσʔλϑΝΠϧଟ͗͢
Έ͋ΔಛఆͷσʔλϑΝΠϧ͚ͩҟৗʹଟ͍ w ؒͷσʔλ͕ϑΝΠϧʹ͋Δ w தϛϦඵ୯ҐͷϨίʔυ w ಛఆͷॲཧ͚͕͔͔ͩ࣌ؒΔ ಛఆͷσʔλϑΝΠϧଟ͗͢
ಛఆͷσʔλϑΝΠϧଟ͗͢ ରॲ๏͋ΔಛఆͷσʔλϑΝΠϧ͚ͩɺຖ࣌ॲཧʹมߋ w ݅ଟ͍σʔλɺ#*ʹग़ྗ͠ͳ͍߲ͩͬͨ w ࣍ॲཧ͔ΒΓͯ͠ɺຖ࣌ॲཧʹมߋ w ࣍ॲཧͷϘτϧωοΫΛআ͍ͨ
ϋϚͬͨͦͷ Athena ͷΫΥʔλ
Έ"UIFOBͷΫΤϦಉ࣮࣌ߦͷΫΥʔλʹҾ͔͔ͬΔ w σʔλҠߦ࣌ʹɺ࣍ॲཧͷ࠷ޙͷ-BNCEBͰΤϥʔʹͳΔ w લஈͰॲཧͨ͠ෳσʔλΛ"UIFOBͬͯ42-ΫΤϦͰऔಘ͢Δͱ͜ΖͰ্ݶʹҾ͔͔ͬΔ w -BNCEBؔͭʹ͖ͭɺɹstart-query-executionɹ"1*Λճίʔϧ w Ұ࣌తʹόʔετͰ্ݶ·Ͱ૿͑Δ͚ͲɺσʔλҠߦ࣌ʹͰ಄ଧͪ w
্ݶ؇ਃ͢Ε্ݶ͋͛ΒΕΔ "UIFOBͷΫΥʔλ
"UIFOBͷΫΥʔλ IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM
ରॲ๏4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦΛઃఆ w .BQεςʔτ ྻ͢ͱɺಉ࣮࣌ߦͰྻཁૉΛॲཧ͢ΔΠϝʔδ ͷ࠷େಉ࣮࣌ߦΛઃ ఆ͠ɺ"UIFOBͷTUBSURVFSZFYFDVUJPO"1*ίʔϧΛ࠷େ·Ͱʹ͓͑͞Δ "UIFOBͷΫΥʔλ
.BQεςʔτʹ͍ͭͯɺҎԼͷهࣄΛࢀߟʹ "UIFOBͷΫΥʔλ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTUFQGVODUJPOTVQEBUFNBQTUBUF IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHBNB[POTUBUFTMBOHVBHFNBQTUBUFIUNM
ϋϚͬͨͦͷ Step Functions ͷΫΥʔλ
Έ4UFQ'VODUJPOTͷΠϕϯτཤྺ͕ΫΥʔλʹҾ͔͔ͬΔ w ͋Δಛఆͷ͚ͩɺຖ࣌ॲཧͷϑΝΠϧ͕ҟৗʹଟ͍ w ࣌ؒܦͬͯҟৗऴྃɻ4UFQ'VODUJPOTͷΠϕϯτཤྺͷ্ݶ౸ୡ Πϕϯτ w ্ݶ؇ෆՄͷ߲
4UFQ'VODUJPOTͷΫΥʔλ { "error": "States.Runtime", "cause": "The execution reached the maximum number of history events (25000)." }
4UFQ'VODUJPOTͷΫΥʔλ IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM
ରॲ๏4UFQ'VODUJPOTͷεςʔτϚγϯΛೖΕࢠʹ w εςʔτϚγϯΛೖΕࢠʹ͢Δ͜ͱͰɺΠϕϯτཤྺ্ݶʹҾ͔͔ͬΒͳ͍Α͏ʹͨ͠ w -BNCEBͷಉ࣮࣌ߦ͕͔ͳΓ૿͑ΔͷͰɺҎԼͷରԠΛՃ 㾎 -BNCEBͷಉ࣮࣌ߦͷ্ݶ؇ਃ 㾎 4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦΛઃఆ 4UFQ'VODUJPOTͷΫΥʔλ
4UFQ'VODUJPOTͷΫΥʔλ มߋલ มߋޙ
4UFQ'VODUJPOTͷΫΥʔλ มߋલ มߋޙ
ϋϚͬͨͦͷ Lambdaͷεέʔϧ͕͍͔ͭͳ͍
Έճ͚ͩ-BNCEBͷ3BUF-JNJUΤϥʔʹૺ۰ w ಉ࣮࣌ߦͷΤϥʔͷΑ͏͚ͩͲʜ w ͢Ͱʹಉ࣮࣌ߦͷ্ݶΛҾ্͖͍͛ͯΔͷͷɺ֤ؔͷϞχλϦϯάݟΔݶΓɺಉ࣮࣌ ߦʹ౸ୡ͍ͯ͠ͳ͍ -BNCEBͷεέʔϧ͕͍͔ͭͳ͍ { "error":
"Lambda.TooManyRequestsException", "cause": "Rate Exceeded. (Service: Lambda, Status Code: 429, Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Extended Request ID: null)" }
-BNCEBͷεέʔϧ͕͍͔ͭͳ͍ IUUQTEPDTBXTBNB[PODPNKB@KQMBNCEBMBUFTUEHJOWPDBUJPOTDBMJOHIUNM
ରॲ๏-BNCEBؔͷ3FUSZઃఆΛݟ͠ w 4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦΛݟ͠ w 4UFQ'VODUJPOTͰఆٛ͢Δ-BNCEBͷ3FUSZઃఆΛݟ͠ -BNCEBͷεέʔϧ͕͍͔ͭͳ͍
ରॲ๏-BNCEBؔͷ3FUSZઃఆΛݟ͠ w 3FUSZͷִؒʹ͍ͭͯҎԼͷهࣄ͕ৄ͍͠ -BNCEBͷεέʔϧ͕͍͔ͭͳ͍ $ node -e '((i,m,b)=>{for(let w=i,c=0;c<m;c++){console.log(w+=(c==0?0:b**c))}})(2,7,1.85)'
2 3.85 7.272500000000001 13.604125000000002 25.317631250000005 46.987617812500005 87.07709295312502 IUUQTEFWDMBTTNFUIPEKQBSUJDMFTXBJU@UJNF@BOE@QBSBNT@JO@TUFQ@GVODUJPO@SFUSZ
ରॲ๏-BNCEBؔͷ3FUSZ݅Λݟ͠ w -BNCEBͷ1SPWJTJPOFE$PODVSSFODZઃఆࠓճ࣮ࢪͯ͠ͳ͍ -BNCEBͷεέʔϧ͕͍͔ͭͳ͍ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTMBNCEBQSPWJTJPOFEDPODVSSFODZDPMETUBSU
ิ σʔλͷՃʹ AWS Glueͱ͍͏αʔϏε͋ΔΑʁ
(MVFΘͣʹɺΘ͟Θ͟4UFQ'VODUJPOT -BNCEBͰΉඞཁ͋Δͷ͔ʁ w 4UFQ'VODUJPOT -BNCEBͷ߹ɺΑ͘ΘΕΔ։ൃϑϨʔϜϫʔΫ͕͑ΔͷͰɺෳਓ Ͱͷ։ൃ͕͍͢͠ɻ 㾎 ࠓճ4FSWFSMFTT'SBNFXPSLͬͨɻ w σʔλϑΝΠϧ͕ଟͯ͘ɺσʔλ݅͋ͨΓͷ༰ྔ͕ͦ͜·Ͱେ͖͘ͳ͚Εɺ࣍ୈ
Ͱ-BNCEBͰॲཧ͕Ͱ͖Δɻ w -BNCEBͰΓΕͳ͍σʔλ༰ྔΛѻ͏߹ɺ(MVFͬͨํ͕͍͍ɻ 㾎 ࠷େϝϞϦׂ.#ɺ࠷େ࣮ߦ࣌ؒɺUNQσΟϨΫτϦαΠζ.# σʔλͷՃͳΒ(MVF͕͋Δ
·ͱΊ • αʔόʔϨεαʔϏεΛۦͯ͠ɺσʔλੳج൫Λ ߏஙՄೳ • αʔόʔϨεͷΑ͋͘ΔΞʔΩςΫνϟύλʔϯΛ͏ ·͍͘͜ͳ͠·͠ΐ͏
None