ECS+S3でスクレイピングを試してみた

Transcript

1. 1.

   ECS+S3Ͱ εΫϨΠϐϯάΛࢼͯ͠Έͨ @sug1t0m0_agrict

2. 2.

   ࣗݾ঺հ ΤϯδχΞྺ - ϑϩϯτΤϯυ : 1೥໨ झຯ - ͓ञ αοΧʔ

   ྉཧ ͓ञ ਿࢁ ஐ඙ @sug1t0m0_agrict
3. 3.

   ૣ଎ɺຊ୊ʹ

4. 4.

   Ϟνϕʔγϣϯ - PythonͰ๭αΠτΛεΫϨΠϐϯά͢ΔΞϓϦέʔγϣϯΛ࡞੒ - Python͸ϩʔΧϧ؀ڥΛԚછ͢Δ(Πϝʔδ)ͳͷͰ։ൃ؀ڥʹDockerΛબ୒ - Ϋϩʔϥͱͯ͠ϔουϨεϒϥ΢βΛར༻͔ͨͬͨ͠ͷͰɺίϯςφؒ௨৴ ͷ؆୯ԽͷͨΊʹαʔϏεΛ·ͱΊΔ͜ͱʹ ↓ -

   2ίϯςφͰ1αʔϏε - ఆظ࣮ߦͱ͔΋ࢼͯ͠Έ͍ͨ ↓ ECS(ىಈλΠϓ͸Fargate)ʹσϓϩΠ (ͨ͠Β͍͍Μ͡Όͳ͍͔ͱࢥͬͨ)
5. 5.

   ཧ૝ͷܗ

6. 6.

   ݱ࣮ͷ࿩

7. 7.

   ΍ͬͨ͜ͱ - εΫϨΠϐϯάΞϓϦέʔγϣϯΛDockernize - Ϋϥελ͔Β্هͷDockerίϯςφΛىಈ͢ΔλεΫΛ࣮ߦ - ϩάΛCloudWatchʹग़ྗ - εΫϨΠϐϯάͷग़ྗ݁ՌΛS3ʹอଘ -λεΫͷఆظ࣮ߦ

8. 8.

   ݱঢ়

9. 9.

   ϋϚͬͨͱ͜Ζ

10. 10.

    λεΫ࣮ߦϩʔϧͷϙϦγʔΛΞλονͰ͖ͳ͍ - νϡʔτϦΞϧ: Amazon ECS CLI Λ࢖༻ͯ͠ Fargate λεΫͷΫϥελʔΛ࡞੒͢ΔΛࢀߟʹ࡞ۀΛਐΊͯ ͍͕ͨҎԼͷίϚϯυ͕͏·࣮͘ߦ͞Εͳ͔ͬͨ

    aws iam --region us-west-2 attach-role-policy - -role-name ecsTaskExecutionRole --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
11. 11.

    ίϯςφʹ؀ڥม਺Λ౉͢ - Ϋϥελઃఆը໘͔ΒλεΫఆٛΛબ୒͠ ͯλεΫΛબ୒͢Δͱ͖ʹʮίϯςφͷ্ॻ ͖ʯ͔Β؀ڥม਺ΛઃఆՄೳ

12. 12.

    ϓϩηε͕ࢮΜͰ΋κϯϏΈ͍ͨʹ ίϯςφ্ཱ͕͕ͪΔ - αʔϏεΛఆٛ͢Δ͜ͱʹΑΓɺΦʔτε έʔϧ΍ίϯςφ͕͚ͨ͜৔߹ʹ࠶ىಈ͞Ε Δతͳ͜ͱ͕ॻ͍͕ͯ͋ͬͨ͜Ε͕ݪҼ?

13. 13.

    ఆظ࣮ߦ͸Մೳͩͬͨ - λεΫͷఆظ࣮ߦ͸Ϋϥελͷઃఆը໘͔ ΒઃఆՄೳ - cronه๏΋࢖༻Մೳ

14. 14.

    ੒Ռ෺

15. 15.

    ·ͱΊ

16. 16.

    ·ͱΊ - S3ͷΞΫηεΩʔΛ؀ڥม਺ͱͯ͠ίϯςφʹ ౉͕ͨ͠, ΋ͬͱ͍͍ํ๏͕͋Δ͸ͣ - 1λεΫͰىಈ͢Δίϯςφ਺Λ૿΍ͯ͠ίϯς φؒ௨৴Λࢼͯ͠Έ͍ͨ - ecs-cli

    up ίϚϯυͰαϒωοτ͕2ͭఆٛ͞Ε ͨVPC͕ੜ੒͞ΕΔͷ͸Կނͳͷ͔? (azʹґଘ͢ΔΒ͍͠ɺ࣭ٙԠ౴ͷ࣌ؒʹղܾ)