Upgrade to Pro — share decks privately, control downloads, hide ads and more …

クローラーをテストしよう!

Chanmoro
February 26, 2019

 クローラーをテストしよう!

クローラーの品質を維持するためにテストを整備した話
tag: crawler, python, scrapy, TDD

Chanmoro

February 26, 2019
Tweet

More Decks by Chanmoro

Other Decks in Technology

Transcript

  1. ࣗݾ঺հ \ lOBNFz< ྆֯࿨ي ΋ΖͣΈ͔͖ͣ   !$IBO@NPSP > 

    DPNQBOZTDPVUZ*OD   QPTJUJPOΤϯδχΞˑ%+୲౰  DBSFFS< 4*FSͰ8FCͷडୗ։ൃ ೥   "*ϕϯνϟʔͰνϟοτϘοτ։ൃ ೥   ΑΓTDPVUZೖࣾ > ^
  2. scouty ͕าΜͰ͖ͨΫϩʔϥʔ։ൃͷྺ࢙ (͜Ε·Ͱ) w ࣾ௕ !ITIJNBEB@ ͕(SFBUͳΞΠσΟΞΛࢥ͍ͭ͘ w ͍͔ͭ͘ͷΫϩʔϥʔΛεϐʔυ༏ઌͰॻ͖·͘Δ ˠͷϑΣʔζ

     w ͱʹ͔͘଎͘ॻ͍ͯܗʹ͢ΔɺҰ൪ಘҙͳ΍ΓํͰॻ͘ w ΤϯδχΞ΋਺໊ՃΘΓΫϩʔϧઌͷ௥Ճ΍ػೳ௥ՃΛ͠·͘Δ ˠͷϑΣʔζ  w ͋Δఔ౓ͷ࢓૊ΈΛ࡞Γͭͭεϐʔυ༏ઌͰ૿ڧ͍ͤͯ͘͞
  3. scouty ͕าΜͰ͖ͨΫϩʔϥʔ։ൃͷྺ࢙ (͍·) w ࣾ௕ !ITIJNBEB@ ͕(SFBUͳΞΠσΟΞΛࢥ͍ͭ͘ w ͍͔ͭ͘ͷΫϩʔϥʔΛεϐʔυ༏ઌͰॻ͖·͘Δ ˠͷϑΣʔζ

     w ͱʹ͔͘଎͘ॻ͍ͯܗʹ͢ΔɺҰ൪ಘҙͳ΍ΓํͰॻ͘ w ΤϯδχΞ΋਺໊ՃΘΓΫϩʔϧઌͷ௥Ճ΍ػೳ௥ՃΛ͠·͘Δ ˠͷϑΣʔζ  w ͋Δఔ౓ͷ࢓૊ΈΛ࡞Γͭͭεϐʔυ༏ઌͰ૿ڧ͍ͤͯ͘͞ w த௕ظͷઓུΛཱͯͯαʔϏεՁ஋Λ૿Ճͤ͞ଓ͚Δ ˠ ͷϑΣʔζ  w ٕज़తෛ࠴͕໨ཱͭΑ͏ʹͳΓ։ൃͷ଎౓͕མͪ࢝ΊΔ w ߟྀ͕ඞཁͳલఏ৚͕݅૿͑Δ ৑௕ʹݟ͑ΔॲཧΛม͑ͨΒόάͬͨʂͱ͔  w ͜Ε·Ͱͷܦݧ͔Βಠࣗͷϊ΢ϋ΢͕஝ੵ͞Ε֦ுͷํ਑͕ఆ·͍ͬͯ͘ w ͜ͷ෦෼͸ཁΒͳ͍ɺ͜͏͍͏ػೳ͕͋Ε͹։ൃ͕ര଎ʹͳΔɺͱ͔
  4. Ϋϩʔϥʔͷςετʹ͍ͭͯ Amazon
 DynamoDB Amazon
 RDS Amazon 
 ElastiCache Amazon
 SQS

    Ϋϩʔϧઌ ֎෦αʔϏε Scrapy spider middleware pipeline Ϋϩʔϧ ϦΫΤετ Request Output Input Response ΫϩʔϥʔΛʮ*OQVUΛड͚औΓɺ֎෦αʔϏεͱ௨৴͠ɺ%#ʹ0VUQVU͢Δॲཧʯͱଊ͑Δ
  5. Ϋϩʔϥʔͷςετʹ͍ͭͯ Amazon
 DynamoDB Amazon
 RDS Amazon 
 ElastiCache Amazon
 SQS

    Ϋϩʔϧઌ ֎෦αʔϏε Scrapy spider middleware pipeline Ϋϩʔϧ ϦΫΤετ Request Output Input ςετର৅ Output͕૝ఆ௨Γ͔Λςετ͍ͨ͠ ֎෦αʔϏεͷϨεϙϯεΛݻఆ͢Δ Response *OQVUͱ֎෦αʔϏεͷϨεϙϯεΛݻఆͨ͠ঢ়ଶͰɺΫϩʔϥʔͷ0VUQVUΛςετ͍ͨ͠ ΫϩʔϥʔͷInputΛݻఆ
  6. Ϋϩʔϥʔͷςετʹ͍ͭͯ Amazon
 DynamoDB Amazon
 RDS Amazon 
 ElastiCache Amazon
 SQS

    Ϋϩʔϧઌ ֎෦αʔϏε Scrapy spider middleware pipeline Ϋϩʔϧ ϦΫΤετ Request Output Input ςετର৅ ΫϩʔϥʔInput/Outputʹ͍ͭͯ ςετίʔυΛॻ͘ Response ֎෦αʔϏεΛελϒʹஔ͖׵͑ɺΫϩʔϥʔͷ*OQVU0VUQVUʹର͢ΔςετίʔυΛॻ͘ ֎෦αʔϏε ελϒ (middleware) ֎෦αʔϏεͷϨεϙϯεΛγϛϡϨʔτ ͢ΔελϒΛࠩ͠ࠐΉ ֤ DB ͸ localstack Λ࢖ͬͯγϛϡϨʔτ
  7. scouty Ϋϩʔϥʔͷ CI/CD ύΠϓϥΠϯ Amazon ECS Docker compose ࣗಈςετ •

    ίʔυϑΥʔϚοτνΣοΫ • unittest ࣮ߦ • Integration test ࣮ߦ • ΫϩʔϧઌαʔϏεͷߏ଄มԽݕ஌ $ git push git push ΛτϦΨʔʹ ࣗಈςετΛ࣮ߦ ࣗಈίʔυϨϏϡʔ $ ecs-goploy HJUQVTIͷ౓ʹࣗಈςετ͕࣮ߦ͞Ε͓ͯΓɺΫϩʔϥʔΛյͨ࣌͠ʹ͙͢ʹؾ͚ͮΔ ΫϩʔϥʔʹݶΒͣTDPVUZͷશͯͷαʔϏεʹಉ༷ͷ$*$%ύΠϓϥΠϯ͕੔උ͞Ε͍ͯΔ
  8. ਓʹؔ͢Δ͋ΒΏΔ৘ใΛ஝ੵ͠ɺ࠷ߴͷબ୒ࢶΛఏڙ͢Δɻ
 γεςϜʹΑͬͯϛεϚον͕ແ͍ੈքΛ࣮ݱ͢Δɻ scoutyͷ໨ࢦ͢ະདྷɾϏδϣϯ 1ਓ1ݸɺ
 scoutyΞΧ΢ϯτΛ
 ͍࣋ͬͯΔ ࠃ಺
 176Ґ ࠃ಺
 176Ґ

    ࣗࣾͷਓࣄ ଞࣾͷਓࣄ ໘ ஊ ධ Ձ ࣾ ಺ ධ Ձ ೥ ऩ ৘ ใ ͕̋̋ ޷͖ U X ৬ɾαʔϏεͷఏڙऀ ΋ͬͱ೥ऩͷߴ͍࢓ࣄ ΋ͬͱ΍Γ͕͍ͷ͋Δ࢓ࣄ ࣗ෼ʹ͔͠Ͱ͖ͳ͍࢓ࣄ ຊਓ ࣗݾਃࠂ
  9. ᅂ޷ੑ৘ใ ʮ͕̋̋޷͖ʯ ೳྗ৘ใ ʮࠃ಺̋̋Ґʯ ৬຿৘ใ ʮ೥ऩ̋̋ສԁʯ ʮೖ͔ࣾΒ̋೥ʯ ߦಈ৘ใ ʮϝοηʔδฦ৴཰̋ˋʯ ʮυλΩϟϯ཰̋ˋʯ

    ༩৴৘ใ ʮ೥ऩ̋̋ԁʯ ʮ࠴຿ͳ͠ʯ ઐ໳ܥSNS ݸਓܥSNS ࣾ಺ਓࣄ
 ͔Βͷ৘ใ ϏδωεSNS ಉ྅͔Βͷ
 ϨϑΝϨϯε ΠϕϯτSNS 1ਓ1ݸscoutyΞΧ΢ϯτΛ࣋ͪɺ
 ৘ใ͕ϥΠϑϩάͷΑ͏ʹ஝ੵ͢Δ ຊਓ Ӿཡ ৘ใฤू ৬ɾαʔϏε
 ͷఏڙऀ ݸਓʹύʔιφϥΠζͨ͠
 αʔϏεͷఏڙ Ӿཡ ৽نࣄۀ