ECS+S3でスクレイピングを試してみた

by sug1t0m0

Slide 1

Slide 1 text

ECS+S3Ͱ εΫϨΠϐϯάΛࢼͯ͠Έͨ @sug1t0m0_agrict

Slide 2

Slide 2 text

ࣗݾ঺հ ΤϯδχΞྺ - ϑϩϯτΤϯυ : 1೥໨ झຯ - ͓ञ αοΧʔ ྉཧ ͓ञ ਿࢁ ஐ඙ @sug1t0m0_agrict

Slide 3

Slide 3 text

ૣ଎ɺຊ୊ʹ

Slide 4

Slide 4 text

Ϟνϕʔγϣϯ - PythonͰ๭αΠτΛεΫϨΠϐϯά͢ΔΞϓϦέʔγϣϯΛ࡞੒ - Python͸ϩʔΧϧ؀ڥΛԚછ͢Δ(Πϝʔδ)ͳͷͰ։ൃ؀ڥʹDockerΛબ୒ - Ϋϩʔϥͱͯ͠ϔουϨεϒϥ΢βΛར༻͔ͨͬͨ͠ͷͰɺίϯςφؒ௨৴ ͷ؆୯ԽͷͨΊʹαʔϏεΛ·ͱΊΔ͜ͱʹ ↓ - 2ίϯςφͰ1αʔϏε - ఆظ࣮ߦͱ͔΋ࢼͯ͠Έ͍ͨ ↓ ECS(ىಈλΠϓ͸Fargate)ʹσϓϩΠ (ͨ͠Β͍͍Μ͡Όͳ͍͔ͱࢥͬͨ)

Slide 5

Slide 5 text

ཧ૝ͷܗ

Slide 6

Slide 6 text

ݱ࣮ͷ࿩

Slide 7

Slide 7 text

΍ͬͨ͜ͱ - εΫϨΠϐϯάΞϓϦέʔγϣϯΛDockernize - Ϋϥελ͔Β্هͷDockerίϯςφΛىಈ͢ΔλεΫΛ࣮ߦ - ϩάΛCloudWatchʹग़ྗ - εΫϨΠϐϯάͷग़ྗ݁ՌΛS3ʹอଘ -λεΫͷఆظ࣮ߦ

Slide 8

Slide 8 text

ݱঢ়

Slide 9

Slide 9 text

ϋϚͬͨͱ͜Ζ

Slide 10

Slide 10 text

λεΫ࣮ߦϩʔϧͷϙϦγʔΛΞλονͰ͖ͳ͍ - νϡʔτϦΞϧ: Amazon ECS CLI Λ࢖༻ͯ͠ Fargate λεΫͷΫϥελʔΛ࡞੒͢ΔΛࢀߟʹ࡞ۀΛਐΊͯ ͍͕ͨҎԼͷίϚϯυ͕͏·࣮͘ߦ͞Εͳ͔ͬͨ aws iam --region us-west-2 attach-role-policy - -role-name ecsTaskExecutionRole --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

Slide 11

Slide 11 text

ίϯςφʹ؀ڥม਺Λ౉͢ - Ϋϥελઃఆը໘͔ΒλεΫఆٛΛબ୒͠ ͯλεΫΛબ୒͢Δͱ͖ʹʮίϯςφͷ্ॻ ͖ʯ͔Β؀ڥม਺ΛઃఆՄೳ

Slide 12

Slide 12 text

ϓϩηε͕ࢮΜͰ΋κϯϏΈ͍ͨʹ ίϯςφ্ཱ͕͕ͪΔ - αʔϏεΛఆٛ͢Δ͜ͱʹΑΓɺΦʔτε έʔϧ΍ίϯςφ͕͚ͨ͜৔߹ʹ࠶ىಈ͞Ε Δతͳ͜ͱ͕ॻ͍͕ͯ͋ͬͨ͜Ε͕ݪҼ?

Slide 13

Slide 13 text

ఆظ࣮ߦ͸Մೳͩͬͨ - λεΫͷఆظ࣮ߦ͸Ϋϥελͷઃఆը໘͔ ΒઃఆՄೳ - cronه๏΋࢖༻Մೳ

Slide 14

Slide 14 text

੒Ռ෺

Slide 15

Slide 15 text

·ͱΊ

Slide 16

Slide 16 text

·ͱΊ - S3ͷΞΫηεΩʔΛ؀ڥม਺ͱͯ͠ίϯςφʹ ౉͕ͨ͠, ΋ͬͱ͍͍ํ๏͕͋Δ͸ͣ - 1λεΫͰىಈ͢Δίϯςφ਺Λ૿΍ͯ͠ίϯς φؒ௨৴Λࢼͯ͠Έ͍ͨ - ecs-cli up ίϚϯυͰαϒωοτ͕2ͭఆٛ͞Ε ͨVPC͕ੜ੒͞ΕΔͷ͸Կނͳͷ͔? (azʹґଘ͢ΔΒ͍͠ɺ࣭ٙԠ౴ͷ࣌ؒʹղܾ)