AWSで作る、サーバーレスデータ分析基盤構築 / jawsug-niigata-11

Slide 1

Slide 1 text

AWSͰ࡞ΔɺαʔόʔϨε σʔλ෼ੳج൫ߏங JAWS-UG৽ׁ#11 2022-01-15 @kasacchiful

Slide 2

Slide 2 text

Classmethod, Inc. Solutions Architect / Software Develper Favorite: Community: • JAWS-UG Niigata • Python ML in Niigata • JaSST Niigata • ASTER • SWANII • etc. Hiroshi Kasahara @kasacchiful @kasacchiful 2

Slide 3

Slide 3 text

αʔόʔϨεͷ෼ੳج൫

Slide 4

Slide 4 text

σʔλ෼ੳʹ͓͚Δ֤छAWSαʔϏε

Slide 5

Slide 5 text

σʔλͷՃ޻ʗ෼ੳʹ AWS Lambda ΋Մೳ

Slide 6

Slide 6 text

ෳࡶɾେن໛ͳΒ AWS Step Functions Λ׆༻

Slide 7

Slide 7 text

αʔόʔϨεύλʔϯ IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO

Slide 8

Slide 8 text

Ϣʔεέʔεผʹύλʔϯ͕͋Δ IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO

Slide 9

Slide 9 text

ύλʔϯͷৄࡉ͸Black BeltͷࢿྉΛࢀߟʹ IUUQTEBXTTUBUJDDPNXFCJOBSTKQQEGTFSWJDFT@"84@#MBDL#FU@4FSWFSMFTT@6TFDBTF@1BUUFSOTQEG :PV5VCFͰͷղઆಈըIUUQTZPVUVCF)*M8ESC@Z.

Slide 10

Slide 10 text

S3ʹೖΕͯ͠·͑͹ɺͳΜͱ͔ͳΔ

Slide 11

Slide 11 text

αʔόʔϨεͰσʔλ࿈ܞ͢Δࡍʹ ϋϚͬͨͱ͜Ζ

Slide 12

Slide 12 text

Step FunctionsͷεςʔτϚγϯͰLambdaͷ ϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλΛՃ޻

Slide 13

Slide 13 text

Step FunctionsͷεςʔτϚγϯͰLambdaͷ ϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλΛՃ޻ σʔλൃੜݩ͔ΒɺσʔλΛऔ ಘͯ͠4ʹอଘ ֤ϑΝΠϧຖʹɺ࠷௿ݶͷσʔ λՃ޻Λͯ͠ɺ4ʹอଘ 2VJDL4JHIU #* ༻ʹ ෳ਺ϑΝΠϧͷσʔλΛ·ͱΊ ͯద੾ʹ੔ܗ͢Δ

Slide 14

Slide 14 text

͍Ζ͍ΖϋϚͬͨͱ͜Ζ 4ͭ঺հ

Slide 15

Slide 15 text

1. ಛఆͷσʔλϑΝΠϧଟ͗͢

Slide 16

Slide 16 text

೰Έ: ͋ΔಛఆͷσʔλϑΝΠϧ͚ͩҟৗʹଟ͍ • 5෼ؒͷσʔλ͕1ϑΝΠϧʹ͋Δ • த਎͸ϛϦඵ୯ҐͷϨίʔυ • ಛఆͷॲཧ͚͕͔͔ͩ࣌ؒΔ

Slide 17

Slide 17 text

• ݅਺ଟ͍σʔλ͸ɺBIʹग़ྗ͠ͳ͍߲໨ͩͬͨ • ೔࣍ॲཧ͔Β੾Γ཭ͯ͠ɺຖ࣌ॲཧʹมߋ • ೔࣍ॲཧͷϘτϧωοΫΛআ͍ͨ ରॲ๏: ͋ΔಛఆͷσʔλϑΝΠϧ͚ͩɺຖ࣌ॲ ཧʹมߋ

Slide 18

Slide 18 text

2. AthenaͷΫΥʔλ

Slide 19

Slide 19 text

• σʔλҠߦ࣌ʹɺ೔࣍ॲཧͷ࠷ޙͷLambdaͰΤϥʔʹͳΔ • લஈͰॲཧͨ͠ෳ਺σʔλΛAthena࢖ͬͯSQLΫΤϦͰऔಘ͢Δͱ͜ΖͰ ্ݶʹҾ͔͔ͬΔ • Lambdaؔ਺1ͭʹ͖ͭɺɹstart-query-executionɹAPIΛ5ճίʔϧ • Ұ࣌తʹόʔετͰ্ݶ80·Ͱ૿͑Δ͚ͲɺσʔλҠߦ࣌ʹ͸20Ͱ಄ଧͪ • ্ݶ؇࿨ਃ੥͢Ε͹্ݶ͋͛ΒΕΔ ೰Έ: AthenaͷΫΤϦಉ࣮࣌ߦ਺ͷΫΥʔλʹ Ҿ͔͔ͬΔ

Slide 20

Slide 20 text

IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

Slide 21

Slide 21 text

ରॲ๏: Step Functions ͷMapεςʔτͷ࠷େಉ ࣮࣌ߦ਺Λઃఆ • Mapεςʔτ (഑ྻ౉͢ͱɺಉ࣮࣌ߦͰ഑ྻཁૉΛॲཧ͢ΔΠϝʔδ) ͷ࠷େಉ࣮࣌ߦ਺Λઃఆ͠ɺAthenaͷ start-query-execution APIίʔ ϧΛ࠷େ20·Ͱʹ͓͑͞Δ

Slide 22

Slide 22 text

Mapεςʔτʹ͍ͭͯ͸ɺҎԼͷهࣄΛࢀߟʹ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTUFQGVODUJPOTVQEBUFNBQTUBUF IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHBNB[POTUBUFTMBOHVBHFNBQTUBUFIUNM

Slide 23

Slide 23 text

3. Step FunctionsͷΫΥʔλ

Slide 24

Slide 24 text

೰Έ: Step FunctionsͷΠϕϯτཤྺ਺͕ΫΥʔ λʹҾ͔͔ͬΔ • ͋Δಛఆͷ೔͚ͩɺຖ࣌ॲཧͷϑΝΠϧ਺͕ҟৗʹଟ͍ • 1࣌ؒܦͬͯҟৗऴྃɻStep FunctionsͷΠϕϯτཤྺ਺ͷ্ݶ౸ୡ (25,000Πϕϯτ) • ্ݶ؇࿨ෆՄͷ߲໨ { "error": "States.Runtime", "cause": "The execution reached the maximum number of history events (25000)." }

Slide 25

Slide 25 text

IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

Slide 26

Slide 26 text

ରॲ๏: Step Functions ͷεςʔτϚγϯΛೖΕ ࢠʹ • εςʔτϚγϯΛೖΕࢠʹ͢Δ͜ͱͰɺΠϕϯτཤྺ্ݶʹҾ͔͔ͬ Βͳ͍Α͏ʹͨ͠ • Lambdaͷಉ࣮࣌ߦ਺͕͔ͳΓ૿͑ΔͷͰɺҎԼͷରԠΛ௥Ճ ✓ Lambdaͷಉ࣮࣌ߦ਺ͷ্ݶ؇࿨ਃ੥ ✓ Step FunctionsͷMapεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ

Slide 27

Slide 27 text

มߋલ มߋޙ

Slide 28

Slide 28 text

มߋલ มߋޙ

Slide 29

Slide 29 text

4. Lambdaͷεέʔϧ͕௥͍͔ͭͳ͍

Slide 30

Slide 30 text

೰Έ: 1ճ͚ͩLambdaͷRateLimitΤϥʔʹૺ۰ • ಉ࣮࣌ߦ਺ͷΤϥʔͷΑ͏͚ͩͲ… • ͢Ͱʹಉ࣮࣌ߦ਺ͷ্ݶΛҾ্͖͍͛ͯΔ΋ͷͷɺ֤ؔ਺ͷϞχλϦ ϯάݟΔݶΓɺಉ࣮࣌ߦ਺ʹ౸ୡ͍ͯ͠ͳ͍ { "error": "Lambda.TooManyRequestsException", "cause": "Rate Exceeded. (Service: Lambda, Status Code: 429, Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Extended Request ID: null)" }

Slide 31

Slide 31 text

IUUQTEPDTBXTBNB[PODPNKB@KQMBNCEBMBUFTUEHJOWPDBUJPOTDBMJOHIUNM

Slide 32

Slide 32 text

ରॲ๏: Lambdaؔ਺ͷRetryઃఆΛݟ௚͠ • Step Functions ͷ Mapεςʔτͷ࠷େಉ࣮࣌ߦ਺Λݟ௚͠ • Step Functions Ͱఆٛ͢Δ Lambda ͷ Retry ઃఆΛݟ௚͠

Slide 33

Slide 33 text

Retry ͷִؒʹ͍ͭͯ͸ҎԼͷهࣄ͕ৄ͍͠ $ node -e '((i,m,b)=>{for(let w=i,c=0;c

Slide 34

Slide 34 text

Lambda ͷ Provisioned Concurrency ઃఆ͸ࠓճ ࣮ࢪͯ͠ͳ͍ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTMBNCEBQSPWJTJPOFEDPODVSSFODZDPMETUBSU

Slide 35

Slide 35 text

σʔλͷՃ޻ʹ͸ AWS Glueͱ͍͏αʔϏε͋ΔΑʁ

Slide 36

Slide 36 text

σʔλͷՃ޻ͳΒGlue͕͋Δ Glue࢖ΘͣʹɺΘ͟Θ͟Step Functions + LambdaͰ૊Ήඞཁ͋Δͷ͔ʁ • Step Functions + Lambdaͷ৔߹ɺΑ͘࢖ΘΕΔ։ൃϑϨʔϜϫʔΫ͕࢖͑ΔͷͰɺෳ਺ਓ Ͱͷ։ൃ͕͠΍͍͢ɻ ✓ ࠓճ͸ Serverless Framework ࢖ͬͨɻ • σʔλϑΝΠϧ਺͕ଟͯ͘΋ɺσʔλ1݅͋ͨΓͷ༰ྔ͕ͦ͜·Ͱେ͖͘ͳ͚Ε͹ɺ޻෉࣍ ୈͰLambdaͰॲཧ͕Ͱ͖Δɻ • LambdaͰ͸৐Γ੾Εͳ͍σʔλ༰ྔ΍࣮ߦ࣌ؒΛѻ͏৔߹͸ɺGlue࢖ͬͨํ͕͍͍ɻ ✓ ࠷େϝϞϦׂ౰: 10240MBɺ࠷େ࣮ߦ࣌ؒ: 15෼ɺ /tmp σΟϨΫτϦαΠζ: 512MB

Slide 37

Slide 37 text

͓·͚

Slide 38

Slide 38 text

͓·͚: AWS Data Wrangler͕ศར IUUQTHJUIVCDPNBXTMBCTBXTEBUBXSBOHMFS

Slide 39

Slide 39 text

͓·͚: AWS Data Wrangler͕ศར PandasͷػೳΛAWSʹ֦ு͢ΔɺΦʔϓϯιʔεͷPythonϥΠϒϥϦ • PandasσʔλϑϨʔϜͱAWSͷσʔλؔ࿈ͷαʔϏεͱΛ͏·͘઀ଓͯ͘͠Ε Δ ✓ Redshift / Glue / Athena / EMR ͳͲ • ௨ৗͷETLλεΫʹඞཁͳؔ਺͕ἧ͍ͬͯΔ

Slide 40

Slide 40 text

஫ҙ఺: ϑΝΠϧαΠζ͕େ͖ͯ͘ɺͦͷ··ͩ ͱLambdaʹ৐Βͳ͍ • LambdaͷσϓϩΠύοέʔδ͸ඇѹॖ࣌ʹ250MBҎԼʹ͢Δඞཁ͕͋Δ ✓ AWS Data WranglerΛී௨ʹpipΠϯετʔϧ͢Δͱɺ250MB௒͑Δ • GitHubͷReleaseϖʔδʹ͋ΔɺLambda Layer༻ͷzipϑΝΠϧΛར༻͠Α͏

Slide 41

Slide 41 text

·ͱΊ • αʔόʔϨεαʔϏεΛۦ࢖ͯ͠ɺσʔλ෼ੳج൫ΛߏஙՄೳ • αʔόʔϨεͷΑ͋͘ΔΞʔΩςΫνϟύλʔϯΛ͏·͘࢖͍͜ͳ͠ ·͠ΐ͏

Slide 42

Slide 42 text

͓͠·͍