ECS+S3でスクレイピングを試してみた

Transcript

1. ECS+S3Ͱ εΫϨΠϐϯάΛࢼͯ͠Έͨ @sug1t0m0_agrict
2. ࣗݾ঺հ ΤϯδχΞྺ - ϑϩϯτΤϯυ : 1೥໨ झຯ - ͓ञ αοΧʔ ྉཧ ͓ञ ਿࢁ ஐ඙ @sug1t0m0_agrict
3. ૣ଎ɺຊ୊ʹ
4. Ϟνϕʔγϣϯ - PythonͰ๭αΠτΛεΫϨΠϐϯά͢ΔΞϓϦέʔγϣϯΛ࡞੒ - Python͸ϩʔΧϧ؀ڥΛԚછ͢Δ(Πϝʔδ)ͳͷͰ։ൃ؀ڥʹDockerΛબ୒ - Ϋϩʔϥͱͯ͠ϔουϨεϒϥ΢βΛར༻͔ͨͬͨ͠ͷͰɺίϯςφؒ௨৴ ͷ؆୯ԽͷͨΊʹαʔϏεΛ·ͱΊΔ͜ͱʹ ↓ - 2ίϯςφͰ1αʔϏε - ఆظ࣮ߦͱ͔΋ࢼͯ͠Έ͍ͨ ↓ ECS(ىಈλΠϓ͸Fargate)ʹσϓϩΠ (ͨ͠Β͍͍Μ͡Όͳ͍͔ͱࢥͬͨ)
5. ཧ૝ͷܗ
6. ݱ࣮ͷ࿩
7. ΍ͬͨ͜ͱ - εΫϨΠϐϯάΞϓϦέʔγϣϯΛDockernize - Ϋϥελ͔Β্هͷDockerίϯςφΛىಈ͢ΔλεΫΛ࣮ߦ - ϩάΛCloudWatchʹग़ྗ - εΫϨΠϐϯάͷग़ྗ݁ՌΛS3ʹอଘ -λεΫͷఆظ࣮ߦ
8. ݱঢ়
9. ϋϚͬͨͱ͜Ζ
10. λεΫ࣮ߦϩʔϧͷϙϦγʔΛΞλονͰ͖ͳ͍ - νϡʔτϦΞϧ: Amazon ECS CLI Λ࢖༻ͯ͠ Fargate λεΫͷΫϥελʔΛ࡞੒͢ΔΛࢀߟʹ࡞ۀΛਐΊͯ ͍͕ͨҎԼͷίϚϯυ͕͏·࣮͘ߦ͞Εͳ͔ͬͨ aws iam --region us-west-2 attach-role-policy - -role-name ecsTaskExecutionRole --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
11. ίϯςφʹ؀ڥม਺Λ౉͢ - Ϋϥελઃఆը໘͔ΒλεΫఆٛΛબ୒͠ ͯλεΫΛબ୒͢Δͱ͖ʹʮίϯςφͷ্ॻ ͖ʯ͔Β؀ڥม਺ΛઃఆՄೳ
12. ϓϩηε͕ࢮΜͰ΋κϯϏΈ͍ͨʹ ίϯςφ্ཱ͕͕ͪΔ - αʔϏεΛఆٛ͢Δ͜ͱʹΑΓɺΦʔτε έʔϧ΍ίϯςφ͕͚ͨ͜৔߹ʹ࠶ىಈ͞Ε Δతͳ͜ͱ͕ॻ͍͕ͯ͋ͬͨ͜Ε͕ݪҼ?
13. ఆظ࣮ߦ͸Մೳͩͬͨ - λεΫͷఆظ࣮ߦ͸Ϋϥελͷઃఆը໘͔ ΒઃఆՄೳ - cronه๏΋࢖༻Մೳ
14. ੒Ռ෺
15. ·ͱΊ
16. ·ͱΊ - S3ͷΞΫηεΩʔΛ؀ڥม਺ͱͯ͠ίϯςφʹ ౉͕ͨ͠, ΋ͬͱ͍͍ํ๏͕͋Δ͸ͣ - 1λεΫͰىಈ͢Δίϯςφ਺Λ૿΍ͯ͠ίϯς φؒ௨৴Λࢼͯ͠Έ͍ͨ - ecs-cli up ίϚϯυͰαϒωοτ͕2ͭఆٛ͞Ε ͨVPC͕ੜ੒͞ΕΔͷ͸Կނͳͷ͔? (azʹґଘ͢ΔΒ͍͠ɺ࣭ٙԠ౴ͷ࣌ؒʹղܾ)