$30 off During Our Annual Pro Sale. View Details »

ECS+S3でスクレイピングを試してみた

 ECS+S3でスクレイピングを試してみた

sug1t0m0

July 05, 2019
Tweet

More Decks by sug1t0m0

Other Decks in Programming

Transcript

  1. ECS+S3Ͱ εΫϨΠϐϯάΛࢼͯ͠Έͨ @sug1t0m0_agrict

  2. ࣗݾ঺հ ΤϯδχΞྺ - ϑϩϯτΤϯυ : 1೥໨ झຯ - ͓ञ αοΧʔ

    ྉཧ ͓ञ ਿࢁ ஐ඙ @sug1t0m0_agrict
  3. ૣ଎ɺຊ୊ʹ

  4. Ϟνϕʔγϣϯ - PythonͰ๭αΠτΛεΫϨΠϐϯά͢ΔΞϓϦέʔγϣϯΛ࡞੒ - Python͸ϩʔΧϧ؀ڥΛԚછ͢Δ(Πϝʔδ)ͳͷͰ։ൃ؀ڥʹDockerΛબ୒ - Ϋϩʔϥͱͯ͠ϔουϨεϒϥ΢βΛར༻͔ͨͬͨ͠ͷͰɺίϯςφؒ௨৴ ͷ؆୯ԽͷͨΊʹαʔϏεΛ·ͱΊΔ͜ͱʹ ↓ -

    2ίϯςφͰ1αʔϏε - ఆظ࣮ߦͱ͔΋ࢼͯ͠Έ͍ͨ ↓ ECS(ىಈλΠϓ͸Fargate)ʹσϓϩΠ (ͨ͠Β͍͍Μ͡Όͳ͍͔ͱࢥͬͨ)
  5. ཧ૝ͷܗ

  6. ݱ࣮ͷ࿩

  7. ΍ͬͨ͜ͱ - εΫϨΠϐϯάΞϓϦέʔγϣϯΛDockernize - Ϋϥελ͔Β্هͷDockerίϯςφΛىಈ͢ΔλεΫΛ࣮ߦ - ϩάΛCloudWatchʹग़ྗ - εΫϨΠϐϯάͷग़ྗ݁ՌΛS3ʹอଘ -λεΫͷఆظ࣮ߦ

  8. ݱঢ়

  9. ϋϚͬͨͱ͜Ζ

  10. λεΫ࣮ߦϩʔϧͷϙϦγʔΛΞλονͰ͖ͳ͍ - νϡʔτϦΞϧ: Amazon ECS CLI Λ࢖༻ͯ͠ Fargate λεΫͷΫϥελʔΛ࡞੒͢ΔΛࢀߟʹ࡞ۀΛਐΊͯ ͍͕ͨҎԼͷίϚϯυ͕͏·࣮͘ߦ͞Εͳ͔ͬͨ

    aws iam --region us-west-2 attach-role-policy - -role-name ecsTaskExecutionRole --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
  11. ίϯςφʹ؀ڥม਺Λ౉͢ - Ϋϥελઃఆը໘͔ΒλεΫఆٛΛબ୒͠ ͯλεΫΛબ୒͢Δͱ͖ʹʮίϯςφͷ্ॻ ͖ʯ͔Β؀ڥม਺ΛઃఆՄೳ

  12. ϓϩηε͕ࢮΜͰ΋κϯϏΈ͍ͨʹ ίϯςφ্ཱ͕͕ͪΔ - αʔϏεΛఆٛ͢Δ͜ͱʹΑΓɺΦʔτε έʔϧ΍ίϯςφ͕͚ͨ͜৔߹ʹ࠶ىಈ͞Ε Δతͳ͜ͱ͕ॻ͍͕ͯ͋ͬͨ͜Ε͕ݪҼ?

  13. ఆظ࣮ߦ͸Մೳͩͬͨ - λεΫͷఆظ࣮ߦ͸Ϋϥελͷઃఆը໘͔ ΒઃఆՄೳ - cronه๏΋࢖༻Մೳ

  14. ੒Ռ෺

  15. ·ͱΊ

  16. ·ͱΊ - S3ͷΞΫηεΩʔΛ؀ڥม਺ͱͯ͠ίϯςφʹ ౉͕ͨ͠, ΋ͬͱ͍͍ํ๏͕͋Δ͸ͣ - 1λεΫͰىಈ͢Δίϯςφ਺Λ૿΍ͯ͠ίϯς φؒ௨৴Λࢼͯ͠Έ͍ͨ - ecs-cli

    up ίϚϯυͰαϒωοτ͕2ͭఆٛ͞Ε ͨVPC͕ੜ੒͞ΕΔͷ͸Կނͳͷ͔? (azʹґଘ͢ΔΒ͍͠ɺ࣭ٙԠ౴ͷ࣌ؒʹղܾ)