Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
サーバーレスでデータ連携する際にハマったところ / ninno-techfest-20211022-lt
Search
kasacchiful
October 22, 2021
Programming
0
530
サーバーレスでデータ連携する際にハマったところ / ninno-techfest-20211022-lt
2021/10/22 (金) NINNO Tech Fest #2 の LT で発表した資料
kasacchiful
October 22, 2021
Tweet
Share
More Decks by kasacchiful
See All by kasacchiful
AWS IoT 1-clickがサービス終了するので、SORACOMに移行した話 / 20240518-jawsug-niigata-iotlt-niigata
kasacchiful
0
100
AWS Application Composerで始める、 サーバーレスなデータ基盤構築 / 20240406-jawsug-hokuriku-shinkansen
kasacchiful
1
400
AWSの各種サービス紹介と活用方法 − AI・ML活用デモを交えて − / 20231208aws-aiml-seminar
kasacchiful
0
440
Amazon Rekognition デモ / 20231208-aws_seminar-01-rekognition-demo
kasacchiful
0
450
Amazon Lookout for Vision デモ / 20231208-aws_seminar-02-lookout-vision-demo
kasacchiful
0
440
Python機械学習勉強会in新潟のロゴが無いので、生成AIで作ってみましょう / osc2023niigata
kasacchiful
0
380
Amazon Bedrock概要と生成AIの基礎 / 20231118-jawsug-niigata-15
kasacchiful
0
93
生成AIと自然言語処理の基礎 / 20231111-pyml-niigata-18.pdf
kasacchiful
0
180
最近やってる、サーバーレスでデータ分析基盤を構築している話 / 20230916-nds63
kasacchiful
0
88
Other Decks in Programming
See All in Programming
さきがけから振り返るアーキテクチャ刷新 / Reflecting on the Architectural Renewal from the Vanguard
nrslib
2
770
APIのない大学ログインWebサービスをWKWebViewとJavaScriptでアプリ化した話
akidon0000
1
330
【Go言語】golangci-lintの使い方
tomo1227
0
270
AWS CDKにおける「再利用性」を考える / aws-cdk-reusability
gotok365
6
1.3k
Product Management LT会_クアンド新家
shinshin
0
210
Microservices rules (July 2024) : what good looks like
cer
PRO
0
1.6k
ぼっちを避けて楽しむためのアノテコノテ / Various Tips and Tricks to Avoid Loneliness and Have Fun
nrslib
3
1.7k
Folding Cheat Sheet #7
philipschwarz
PRO
0
150
Prompt FlowによるLLMアプリケーション開発
yuto2000
1
1k
Cloudflare Workers x AWS Lambdaの組み合わせユースケース / Cloudflare Workers x AWS Lambda Combination Use Case
seike460
PRO
2
310
Harnessing Large Language Models for Training-free Video Anomaly Detection
tereka114
1
1.3k
AWSでゲームサーバーを運用! Amazon GameLiftのお話
iriikeita
0
200
Featured
See All Featured
Bash Introduction
62gerente
607
210k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
13
430
Web development in the modern age
philhawksworth
203
10k
KATA
mclloyd
20
13k
Fontdeck: Realign not Redesign
paulrobertlloyd
79
5.1k
WebSockets: Embracing the real-time Web
robhawkes
59
7.2k
Pencils Down: Stop Designing & Start Developing
hursman
118
11k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
248
20k
Imperfection Machines: The Place of Print at Facebook
scottboms
262
13k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
16
1.6k
A Modern Web Designer's Workflow
chriscoyier
689
190k
Designing for Performance
lara
604
67k
Transcript
αʔόʔϨεͰσʔλ࿈ܞ͢Δࡍʹ ϋϚͬͨͱ͜Ζ ּݪ /*//05FDI'FTU
ຊ͓͢͠Δ͜ͱ wαʔόʔϨεͰσʔλ࿈ܞج൫Λߏஙͨ͠ࡍɺϋϚͬͨͱ͜Ζ͕৭ʑ ͋ͬͨͷͰɺ؆୯ʹ͓͠͠·͢ɻ w"84Ͱߏங͍ͯ͠·͢ɻ"84͋·ΓΘ͔Βͳ͍ํɺ༰ࠐΈೖͬͯ ΔͷͰ͍͔͠͠Ε·ͤΜɻ w-5ͳͷʹϖʔδ࡞ͬͨͷͰɺۦ͚ͰਐΈ·͢ɻ
ࣗݾհ w ּݪ w ΫϥεϝιουגࣜձࣾσʔλΞφϦςΟΫεࣄۀຊ෦ w ৽ׁݝ৽ׁࢢࡏॅ w ͖ͳ"84αʔϏε4ɺ-BNCEBɺ4BHF.BLFS
w +"846(৽ׁࢧ෦ʗ1ZUIPOػցֶशษڧձJO৽ׁʗ +B445৽ׁʗ48"/**ʗFUD @kasacchiful @kasacchiful
։࢝ w IUUQTQZNMOJJHBUBDPOOQBTTDPNFWFOU 1ZUIPOػցֶशษڧձJO৽ׁ
w IUUQTKBXTVHOJJHBUBDPOOQBTTDPN +"846(/JJHBUB
+"841"/,3"5*0/ ʙ ͷ࣌ؒΠϕϯτ w IUUQTKBXTQBOLSBUJPOKBXTVHKQ
ຊ αʔόʔϨεͷσʔλج൫
ྫ͑͜Μͳͭ σʔλੳʹ͓͚Δ֤छ"84αʔϏε
ྫ͑͜Μͳͭ 4ʹσʔλೖΕͯɺ-BNCEBͰՃɺ݁ՌΛ4ʹ͓͘ɻ
ྫ͑͜Μͳͭ 4ʹՃࡁσʔλ͓͍ͨΒɺ͋ͱੳʹͲ͏ͧɻ
ࠓճ͜Μͳײ͡ͷͭ 4UFQ'VODUJPOTͷεςʔτϚγϯͰ-BNCEBͷϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλ ΛՃ w ճεέδϡʔϧ࣮ߦ
4UFQ'VODUJPOTͷεςʔτϚγϯͰ-BNCEBͷϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλ ΛՃ w ճεέδϡʔϧ࣮ߦ ࠓճ͜Μͳײ͡ͷͭ σʔλൃੜݩ͔ΒɺσʔλΛऔ ಘͯ͠4ʹอଘ ֤ϑΝΠϧຖʹɺ࠷ݶͷσʔ λՃΛͯ͠ɺ4ʹอଘ
2VJDL4JHIU #* ༻ʹ ෳϑΝΠϧͷσʔλΛ·ͱΊ ͯదʹܗ͢Δ
͍Ζ͍ΖϋϚͬͨ ϋϚͬͨΛ4ͭհ
ϋϚͬͨͦͷ ಛఆͷσʔλϑΝΠϧଟ͗͢
Έ͋ΔಛఆͷσʔλϑΝΠϧ͚ͩҟৗʹଟ͍ w ؒͷσʔλ͕ϑΝΠϧʹ͋Δ w தϛϦඵ୯ҐͷϨίʔυ w ಛఆͷॲཧ͚͕͔͔ͩ࣌ؒΔ ಛఆͷσʔλϑΝΠϧଟ͗͢
ಛఆͷσʔλϑΝΠϧଟ͗͢ ରॲ๏͋ΔಛఆͷσʔλϑΝΠϧ͚ͩɺຖ࣌ॲཧʹมߋ w ݅ଟ͍σʔλɺ#*ʹग़ྗ͠ͳ͍߲ͩͬͨ w ࣍ॲཧ͔ΒΓͯ͠ɺຖ࣌ॲཧʹมߋ w ࣍ॲཧͷϘτϧωοΫΛআ͍ͨ
ϋϚͬͨͦͷ Athena ͷΫΥʔλ
Έ"UIFOBͷΫΤϦಉ࣮࣌ߦͷΫΥʔλʹҾ͔͔ͬΔ w σʔλҠߦ࣌ʹɺ࣍ॲཧͷ࠷ޙͷ-BNCEBͰΤϥʔʹͳΔ w લஈͰॲཧͨ͠ෳσʔλΛ"UIFOBͬͯ42-ΫΤϦͰऔಘ͢Δͱ͜ΖͰ্ݶʹҾ͔͔ͬΔ w -BNCEBؔͭʹ͖ͭɺɹstart-query-executionɹ"1*Λճίʔϧ w Ұ࣌తʹόʔετͰ্ݶ·Ͱ૿͑Δ͚ͲɺσʔλҠߦ࣌ʹͰ಄ଧͪ w
্ݶ؇ਃ͢Ε্ݶ͋͛ΒΕΔ "UIFOBͷΫΥʔλ
"UIFOBͷΫΥʔλ IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM
ରॲ๏4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦΛઃఆ w .BQεςʔτ ྻ͢ͱɺಉ࣮࣌ߦͰྻཁૉΛॲཧ͢ΔΠϝʔδ ͷ࠷େಉ࣮࣌ߦΛઃ ఆ͠ɺ"UIFOBͷTUBSURVFSZFYFDVUJPO"1*ίʔϧΛ࠷େ·Ͱʹ͓͑͞Δ "UIFOBͷΫΥʔλ
.BQεςʔτʹ͍ͭͯɺҎԼͷهࣄΛࢀߟʹ "UIFOBͷΫΥʔλ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTUFQGVODUJPOTVQEBUFNBQTUBUF IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHBNB[POTUBUFTMBOHVBHFNBQTUBUFIUNM
ϋϚͬͨͦͷ Step Functions ͷΫΥʔλ
Έ4UFQ'VODUJPOTͷΠϕϯτཤྺ͕ΫΥʔλʹҾ͔͔ͬΔ w ͋Δಛఆͷ͚ͩɺຖ࣌ॲཧͷϑΝΠϧ͕ҟৗʹଟ͍ w ࣌ؒܦͬͯҟৗऴྃɻ4UFQ'VODUJPOTͷΠϕϯτཤྺͷ্ݶ౸ୡ Πϕϯτ w ্ݶ؇ෆՄͷ߲
4UFQ'VODUJPOTͷΫΥʔλ { "error": "States.Runtime", "cause": "The execution reached the maximum number of history events (25000)." }
4UFQ'VODUJPOTͷΫΥʔλ IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM
ରॲ๏4UFQ'VODUJPOTͷεςʔτϚγϯΛೖΕࢠʹ w εςʔτϚγϯΛೖΕࢠʹ͢Δ͜ͱͰɺΠϕϯτཤྺ্ݶʹҾ͔͔ͬΒͳ͍Α͏ʹͨ͠ w -BNCEBͷಉ࣮࣌ߦ͕͔ͳΓ૿͑ΔͷͰɺҎԼͷରԠΛՃ 㾎 -BNCEBͷಉ࣮࣌ߦͷ্ݶ؇ਃ 㾎 4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦΛઃఆ 4UFQ'VODUJPOTͷΫΥʔλ
4UFQ'VODUJPOTͷΫΥʔλ มߋલ มߋޙ
4UFQ'VODUJPOTͷΫΥʔλ มߋલ มߋޙ
ϋϚͬͨͦͷ Lambdaͷεέʔϧ͕͍͔ͭͳ͍
Έճ͚ͩ-BNCEBͷ3BUF-JNJUΤϥʔʹૺ۰ w ಉ࣮࣌ߦͷΤϥʔͷΑ͏͚ͩͲʜ w ͢Ͱʹಉ࣮࣌ߦͷ্ݶΛҾ্͖͍͛ͯΔͷͷɺ֤ؔͷϞχλϦϯάݟΔݶΓɺಉ࣮࣌ ߦʹ౸ୡ͍ͯ͠ͳ͍ -BNCEBͷεέʔϧ͕͍͔ͭͳ͍ { "error":
"Lambda.TooManyRequestsException", "cause": "Rate Exceeded. (Service: Lambda, Status Code: 429, Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Extended Request ID: null)" }
-BNCEBͷεέʔϧ͕͍͔ͭͳ͍ IUUQTEPDTBXTBNB[PODPNKB@KQMBNCEBMBUFTUEHJOWPDBUJPOTDBMJOHIUNM
ରॲ๏-BNCEBؔͷ3FUSZઃఆΛݟ͠ w 4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦΛݟ͠ w 4UFQ'VODUJPOTͰఆٛ͢Δ-BNCEBͷ3FUSZઃఆΛݟ͠ -BNCEBͷεέʔϧ͕͍͔ͭͳ͍
ରॲ๏-BNCEBؔͷ3FUSZઃఆΛݟ͠ w 3FUSZͷִؒʹ͍ͭͯҎԼͷهࣄ͕ৄ͍͠ -BNCEBͷεέʔϧ͕͍͔ͭͳ͍ $ node -e '((i,m,b)=>{for(let w=i,c=0;c<m;c++){console.log(w+=(c==0?0:b**c))}})(2,7,1.85)'
2 3.85 7.272500000000001 13.604125000000002 25.317631250000005 46.987617812500005 87.07709295312502 IUUQTEFWDMBTTNFUIPEKQBSUJDMFTXBJU@UJNF@BOE@QBSBNT@JO@TUFQ@GVODUJPO@SFUSZ
ରॲ๏-BNCEBؔͷ3FUSZ݅Λݟ͠ w -BNCEBͷ1SPWJTJPOFE$PODVSSFODZઃఆࠓճ࣮ࢪͯ͠ͳ͍ -BNCEBͷεέʔϧ͕͍͔ͭͳ͍ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTMBNCEBQSPWJTJPOFEDPODVSSFODZDPMETUBSU
ิ σʔλͷՃʹ AWS Glueͱ͍͏αʔϏε͋ΔΑʁ
(MVFΘͣʹɺΘ͟Θ͟4UFQ'VODUJPOT -BNCEBͰΉඞཁ͋Δͷ͔ʁ w 4UFQ'VODUJPOT -BNCEBͷ߹ɺΑ͘ΘΕΔ։ൃϑϨʔϜϫʔΫ͕͑ΔͷͰɺෳਓ Ͱͷ։ൃ͕͍͢͠ɻ 㾎 ࠓճ4FSWFSMFTT'SBNFXPSLͬͨɻ w σʔλϑΝΠϧ͕ଟͯ͘ɺσʔλ݅͋ͨΓͷ༰ྔ͕ͦ͜·Ͱେ͖͘ͳ͚Εɺ࣍ୈ
Ͱ-BNCEBͰॲཧ͕Ͱ͖Δɻ w -BNCEBͰΓΕͳ͍σʔλ༰ྔΛѻ͏߹ɺ(MVFͬͨํ͕͍͍ɻ 㾎 ࠷େϝϞϦׂ.#ɺ࠷େ࣮ߦ࣌ؒɺUNQσΟϨΫτϦαΠζ.# σʔλͷՃͳΒ(MVF͕͋Δ
·ͱΊ • αʔόʔϨεαʔϏεΛۦͯ͠ɺσʔλੳج൫Λ ߏஙՄೳ • αʔόʔϨεͷΑ͋͘ΔΞʔΩςΫνϟύλʔϯΛ͏ ·͍͘͜ͳ͠·͠ΐ͏
None