AWS Step Functions を用いた非同期学習処理の例

by Hacarus Inc.

Slide 1

Slide 1 text

AWS Step FunctionsΛ༻͍ͨ ඇಉظػցֶशॲཧͷྫ גࣜձࣾ)"$"364಺໺޺ढ़ɹ!+"846(ژ౎

Slide 2

Slide 2 text

✓ ಺໺ ޺ढ़ (͏ͪͷ ͻΖͱ͠) ✓ 2017೥ ཧֶ (ത࢜) - ஍ٿ࣓ؾݍͷ෺ཧաఔͷݚڀ ✓ 2021೥ HACARUS ʹத్ೖࣾ - ΞϓϦέʔγϣϯΤϯδχΞ - AWS Λ༻͍ͨ SaaS ͷ։ൃ౳ ✓ ݸਓతڵຯ - ػցֶशͷࣗવՊֶ΁ͷԠ༻ - WebγεςϜ։ൃ 2 ࣗݾ঺հ -20℃ͷத࣓ྗܭΛຒΊ͍ͯΔ @Χϓεέʔγϯά (Χφμ)

Slide 3

Slide 3 text

3 ࠓճͷετʔϦʔ 1. AWS ্Ͱػցֶशॲཧͷ API Λ։ൃ։࢝ɻػցֶश෦෼Λඇಉظͷ෼ࢄॲ ཧʹɻJob ؅ཧํ๏Ͱ೰Ή 2. AWS SA ͷํʹ AWS StepFunctions ΛקΊΒΕͨ 3. AWS Step Functions Λಋೖͯ͠ Job ؅ཧͷ໰୊ΛҰؾʹղܾɻ࣮૷ίετ ΋ӡ༻අ༻΋཈͑ΒΕΔߏ੒ʹ

Slide 4

Slide 4 text

✓ ը૾தͷ෺ମ(ͷγάφϧ)Λݕ஌͢Δػցֶशॲཧͷٕज़͕͋ͬͨ ✓ BtoB ޲͚ʹ্هػցֶशॲཧͷAPIΛ։ൃ ✓ API ͷཁ݅ - Ϣʔβʔ͔Βͷը૾͸Ұ౓ʹେྔʹ౤ೖ͞ΕΔ͜ͱΛ૝ఆ - ػցֶश͸ඇಉظ͔ͭ෼ࢄॲཧΛߦ͍ɺϢʔβʔମݧΛམͱ͞ͳ͍ • Job ͷड͚෇͚ͱ Job ݁ՌऔಘΛผAPIʹ͢Δ 4 ΠϯτϩμΫγϣϯ ~ ෺ମݕ஌ॲཧͷAPI։ൃ ~

Slide 5

Slide 5 text

5 API ͷཁ݅ͷ֓؍ ᶃ ᶄ 1. ը૾ͷ౤ೖΛड͚ͯ Job Λ࡞੒ɺ͙͢ʹ Job ID Λൃߦɻ෺ମݕ஌ͷػցֶशॲཧ ͸ը૾͝ͱʹඇಉظ෼ࢄॲཧͱ࣮ͯ͠ߦ͞ΕΔɻ(1 Job - ෳ਺ Process ͷؔ܎) 2. Job ID ͷ݁ՌΛϦΫΤετɻJob ͕ऴ͍ྃͯͨ͠Β݁ՌΛฦ٫

Slide 6

Slide 6 text

6 ࠷ॳʹߟ͍͑ͯͨ AWS ্ͷߏ੒ (ൈਮ) ✓ SQSͰΩϡʔΠϯάͯ͠ඇಉظͰ Lambda ͕ػցֶशॲཧΛ࣮ߦ ✓ Lambda ͷػցֶशॲཧऴྃ࣌ʹ Process ͷεςʔλεΛ fi nish ʹ ✓ ෼ࢄ͞Εͨ Process ͷεςʔλε͕ fi nish ʹͳͬͨΒ Job ͷεςʔλε ΋ fi nish ʹ͍ͨ͠

Slide 7

Slide 7 text

✓ ෼ࢄ͞Εͨ Process ͷεςʔλε͕͢΂ͯ fi nish ʹͳͬͨΒ Job ͷεςʔλε΋ fi nish ʹ͍ͨ͠ - ํ๏1. RDS ΁ͷϙʔϦϯάॲཧͰ Job ʹඥͮ͘͢΂ͯͷ Process ͷεςʔλεΛ؂ ࢹ͍ͯͯ͠ɺ͢΂ͯ fi nish ʹͳͬͨΒ Job ͷεςʔλε΋ fi nish ʹ͢Δ (લϖʔδਤ) - ํ๏2. Job ʹඥͮ͘ ML Process ͕1ͭऴΘΔͨͼʹɺRDS ʹͦͷ Job ͢΂ͯͷ Process ͷεςʔλε͕ fi nish ʹͳ͍ͬͯΔ͔Λ֬ೝͯ͠ɺ͢΂ͯ fi nish Ͱ͋Ε͹ Job ͷεςʔλε΋ fi nish ʹ 7 ߟ͍͑ͯͨ Job ؅ཧͷํ๏

Slide 8

Slide 8 text

8 AWS SA ͱͷํͱͷ΍ΓͱΓ Step Functions Λ࢖͑͹͍͍Α Job ؅ཧ࣮૷ΊΜͲͤ͐͘ʙ ͳΔ΄Ͳɺ΍ͬͯΈ·͢ʂʂ (Step Functions …🤔? ) AWS SA: H ͞Μ

Slide 9

Slide 9 text

✓ AWS Step Functions ͱ͸ - ෼ࢄΞϓϦέʔγϣϯ΍ػցֶशύΠϓϥΠϯͳͲΛϩʔίʔυͰߏஙͰ ͖ΔαʔϏεɻϢʔεέʔεଟ͠ - ো֐ɾ࠶ࢼߦɾฒྻԽ౳Λ Step Functions ͕؅ཧ͢ΔͨΊɺϏδωεϩδ οΫʹूதͰ͖Δ ✓ ϫʔΫϑϩʔͷߏங - εςʔτΛͭͳ͗߹ΘͤͯϫʔΫϑϩʔΛߏங - Work fl ow Studio Ͱࢹ֮తʹߏஙՄೳ 9 AWS Step Functions ʹ͍ͭͯ https://aws.amazon.com/jp/step-functions/?step-functions.sort-by=item.additionalFields.postDateTime&step-functions.sort-order=desc

Slide 10

Slide 10 text

✓ Step Functions εςʔτͷೖग़ྗ - ֤εςʔτͷೖग़ྗΛ͏·͘ௐ੔͢Δ͜ͱͰॲཧϑϩʔΛͭͳ͛ΒΕΔ - InputPathɾResultPathɾOutputPath ͷཧղ͕ඞਢɻ Class Method ͞Μ ͷهࣄ͕ඇৗʹࢀߟʹͳͬͨ 10 Step Functions ͰͷϫʔΫϑϩʔߏஙํ๏ ٙࣅMLॲཧ ٙࣅޙॲཧ ✓ ࣗ෼ͷ Step Functions ֶशํ๏ - MapɾLambda Λ࢖͏ํ਑ΛઌʹܾΊΔ - ೖग़ྗͷΈ߹Θͤͨٙࣅ ML ॲཧɾٙࣅ ޙॲཧͷ Lambda Λ४උͯ͠ೖग़ྗͷௐ ੔Λ࣮ࢪ

Slide 11

Slide 11 text

11 Step Functions Λ࢖ͬͨ AWS ্ͷߏ੒ (ൈਮ) ✓ Step Functions ͷ Map ػೳΛ࢖ͬͯػց ֶशॲཧΛ෼ࢄ ✓ Map ͷॲཧޙʹಉظΛࣗಈతʹͱͬͯ͘ ΕΔ ✓ Map ऴྃޙʹ Job ͷεςʔλεΛ fi nish ʹॻ͖׵͑

Slide 12

Slide 12 text

12 ࣮ࡍͷߏ੒

Slide 13

Slide 13 text

✓ Job ؅ཧͷ࣮૷ͷखؒΛ࡟ݮ ✓ RDS ͷ؂ࢹΛ͠ͳͯ͘Α͘ͳΓɺBackend Λৗ࣌ىಈ͢Δඞཁ͕ͳ͘ͳͬͨ - Backend ॲཧΛ Lambda Ͱ࣮ߦͰ͖ΔΑ͏ʹͳΓӡ༻අ༻Λ࡟ݮ ✓ ػցֶशॲཧͷΤϥʔൃੜ࣌ͷ҆৺ײ - ϦτϥΠॲཧ౳Λద੾ͳҐஔʹ͙͢ʹೖΕΒΕΔ 13 Step Functions Λ࢖ͬͨ݁Ռ

Slide 14

Slide 14 text

✓ ͕࣌ؒͳ͍தͰͷ AWS ͷαʔϏεΛ༻͍ͨ։ൃͷ՝୊ - ͱΓ͋͑ͣ஌͍ͬͯΔαʔϏεΛ࢖ͬͯ͠·͏ ✓ ֶश͕࣌ؒଟগ͋ͬͯ΋AWS ͷαʔϏε͸ଟ͘ɺͲΕΛֶश͢Δ͔͕೉͍͠ - ࠓճ͸ɺ AWS SA ͷํʹ Step Functions ΛΦεεϝͯ͠΋Β͑ͨͱ͍͏ ҆৺ײ͕͋ͬͨ - ୭͔ʹഎதΛԡͯ͠΋Β͑Δ؀ڥ͸େࣄ → ࣾ಺ʹ AWS ʹৄ͍͠ਓ͕͍ͳ ͍৔߹͸ AWS SA ʹ૬ஊͰ͖ΔΑ͏ʹͳ͍ͬͯΔͱΑ͍ 14 AWS αʔϏεΛ༻͍ͨ։ൃʹ͍ͭͯײͨ͜͡ͱ

Slide 15

Slide 15 text

✓ ඇಉظ෼ࢄॲཧͷJob؅ཧͷղܾࡦͱͯ͠ Step Functions Λར༻ - ΄΅࣮૷͢Δ͜ͱͳ͘ Job؅ཧ Λ࣮ݱ - શମతͳӡ༻අ༻ͷ཈੍΋Մೳ ✓ Step Functions ͸͍͍ͧ - ؆୯ʹεςʔτΛ૊Έ߹ΘͤͯෳࡶͳϫʔΫϑϩʔΛߏங - ·ͩॳ৺ऀͳͷͰ࢖͍͜ͳ͍͖͍ͯͨ͠ ✓ AWS ͷΞʔΩςΫνϟͰ໎͍ͬͯΔਓ͸ SA ʹ૬ஊͯ͠ΈΑ͏ 15 ·ͱΊ

Slide 16

Slide 16 text

✓ TECH in ژ౎ ͱ͸ - Technology Enhancement Community by HACARUS ͷུɻٕज़޷͖͕ ϥϑʹू·ֶͬͯ΂Δ৔Λͭ͘Γɺژ౎͔Β໘ന͍΋ͷΛੜΈग़͍ͨ͠ - جຊΦϯϥΠϯ(zoom)ɻ ௚઀ൃදऀͱ΍ΓͱΓ͢Δ࿮ͱௌ͚ͩ͘ͷ࿮͋Γ - ͦͷ͏ͪΦϑϥΠϯ։࠵΋͍ͨ͠ ✓ TECH in ژ౎ Λٹ͍͍ͨ - 3ճ΍͚ͬͨͲࢀՃऀ͕૿͑ͳ͍ʂ 16 HACARUS ࣾ֎ษڧձ: TECH in ژ౎