クローラーをテストしよう！

by Chanmoro

Slide 1

Slide 1 text

2019/02/26 https://scouty.co.jp ΫϩʔϥʔΛςετ͠Α͏ʂ Kazuki Morozumi

Slide 2

Slide 2 text

ࣗݾ঺հ \ lOBNFz< ྆֯࿨ي ΋ΖͣΈ͔͖ͣ !$IBO@NPSP > DPNQBOZTDPVUZ*OD QPTJUJPOΤϯδχΞˑ%+୲౰ DBSFFS< 4*FSͰ8FCͷडୗ։ൃ ೥ "*ϕϯνϟʔͰνϟοτϘοτ։ൃ ೥ ΑΓTDPVUZೖࣾ > ^

Slide 3

Slide 3 text

ࠓ೔࿩͢͜ͱ w TDPVUZͷΫϩʔϥʔʹ͍ͭͯ w Ϋϩʔϥʔͷςετʹ͍ͭͯ w ࠓޙͷల๬

Slide 4

Slide 4 text

ʮscouty ͷΫϩʔϥʔʹ͍ͭͯʯ

Slide 5

Slide 5 text

ͦ΋ͦ΋ scouty ͬͯԿʁ

Slide 6

Slide 6 text

2018೥8݄27೔ʹ scouty ਖ਼ࣜ൛ ΛϦϦʔε

Slide 7

Slide 7 text

scouty ʹ͍ͭͯ w ༷ʑͳ৘ใιʔεΛΫϩʔϧ͠ݸਓͷΞ ΢τϓοτΛՄࢹԽ w ֤৘ใιʔεΛఆظతʹΫϩʔϧ͠৘ใ ͷ઱౓Λҡ࣋ w ৘ใιʔε͸4/4 ϒϩάαʔϏε ݸ ਓͰϗεςΟϯά͍ͯ͠ΔϖʔδͳͲ༷ʑ

Slide 8

Slide 8 text

Ͳ͏͍͏Ϋϩʔϥʔ͕ඞཁʁ

Slide 9

Slide 9 text

scouty ΫϩʔϥʔʹٻΊΒΕΔཁ݅ w ؒҧͬͨσʔλͷऔಘ͸ઈର/( w ͭͷαΠτ΍"1*ΛεΫϨΠϐϯά͢Δॲཧࣗମ͸؆୯ w ΫϩʔϧઌαʔϏεͷߏ଄มԽɺߋ৽͠ଓ͚ͯ΋σʔλ͕յΕͳ͍ઃܭ w Ұఆͷظؒ಺ʹ࠷৽৘ใ͕औಘ͞ΕΔඞཁ͕͋Δ w ΫϩʔϧͷεέδϡʔϦϯά w ਫฏεέʔϧ͠΍͍͢ઃܭɺΞʔΩςΫνϟʔ

Slide 10

Slide 10 text

Slide 11

Slide 11 text

શମͷΞʔΩςΫνϟʔ

Slide 12

Slide 12 text

scouty ΫϩʔϥʔͷγεςϜߏ੒

Slide 13

Slide 13 text

scouty ΫϩʔϥʔͷγεςϜߏ੒ Ϋϩʔϥʔ

Slide 14

Slide 14 text

scouty ΫϩʔϥʔͷγεςϜߏ੒ Ϋϩʔϥʔ •ΫϩʔϧͷϦΫΤετ • ΫϩʔϧࡁΈσʔλͷอଘ • Ϋϩʔϧ׬ྃΠϕϯτ௨஌

Slide 15

Slide 15 text

scouty ͕าΜͰ͖ͨΫϩʔϥʔ։ൃͷྺ࢙ (͜Ε·Ͱ) w ࣾ௕ !ITIJNBEB@ ͕(SFBUͳΞΠσΟΞΛࢥ͍ͭ͘ w ͍͔ͭ͘ͷΫϩʔϥʔΛεϐʔυ༏ઌͰॻ͖·͘Δ ˠͷϑΣʔζ w ͱʹ͔͘଎͘ॻ͍ͯܗʹ͢ΔɺҰ൪ಘҙͳ΍ΓํͰॻ͘ w ΤϯδχΞ΋਺໊ՃΘΓΫϩʔϧઌͷ௥Ճ΍ػೳ௥ՃΛ͠·͘Δ ˠͷϑΣʔζ w ͋Δఔ౓ͷ࢓૊ΈΛ࡞Γͭͭεϐʔυ༏ઌͰ૿ڧ͍ͤͯ͘͞

Slide 16

Slide 16 text

scouty ͕าΜͰ͖ͨΫϩʔϥʔ։ൃͷྺ࢙ (͍·) w ࣾ௕ !ITIJNBEB@ ͕(SFBUͳΞΠσΟΞΛࢥ͍ͭ͘ w ͍͔ͭ͘ͷΫϩʔϥʔΛεϐʔυ༏ઌͰॻ͖·͘Δ ˠͷϑΣʔζ w ͱʹ͔͘଎͘ॻ͍ͯܗʹ͢ΔɺҰ൪ಘҙͳ΍ΓํͰॻ͘ w ΤϯδχΞ΋਺໊ՃΘΓΫϩʔϧઌͷ௥Ճ΍ػೳ௥ՃΛ͠·͘Δ ˠͷϑΣʔζ w ͋Δఔ౓ͷ࢓૊ΈΛ࡞Γͭͭεϐʔυ༏ઌͰ૿ڧ͍ͤͯ͘͞ w த௕ظͷઓུΛཱͯͯαʔϏεՁ஋Λ૿Ճͤ͞ଓ͚Δ ˠ ͷϑΣʔζ w ٕज़తෛ࠴͕໨ཱͭΑ͏ʹͳΓ։ൃͷ଎౓͕མͪ࢝ΊΔ w ߟྀ͕ඞཁͳલఏ৚͕݅૿͑Δ ৑௕ʹݟ͑ΔॲཧΛม͑ͨΒόάͬͨʂͱ͔ w ͜Ε·Ͱͷܦݧ͔Βಠࣗͷϊ΢ϋ΢͕஝ੵ͞Ε֦ுͷํ਑͕ఆ·͍ͬͯ͘ w ͜ͷ෦෼͸ཁΒͳ͍ɺ͜͏͍͏ػೳ͕͋Ε͹։ൃ͕ര଎ʹͳΔɺͱ͔

Slide 17

Slide 17 text

ςετΛॻ͍ͯ҆৺͍ͨ͠ʂ

Slide 18

Slide 18 text

ʮΫϩʔϥʔͷςετʹ͍ͭͯʯ

Slide 19

Slide 19 text

Ϋϩʔϥʔͷςετʹ͍ͭͯ Amazon  DynamoDB Amazon  RDS Amazon   ElastiCache Amazon  SQS Ϋϩʔϧઌ ֎෦αʔϏε Scrapy spider middleware pipeline Ϋϩʔϧ ϦΫΤετ Request Output Input Response ΫϩʔϥʔΛʮ*OQVUΛड͚औΓɺ֎෦αʔϏεͱ௨৴͠ɺ%#ʹ0VUQVU͢Δॲཧʯͱଊ͑Δ

Slide 20

Slide 20 text

Ϋϩʔϥʔͷςετʹ͍ͭͯ Amazon  DynamoDB Amazon  RDS Amazon   ElastiCache Amazon  SQS Ϋϩʔϧઌ ֎෦αʔϏε Scrapy spider middleware pipeline Ϋϩʔϧ ϦΫΤετ Request Output Input ςετର৅ Output͕૝ఆ௨Γ͔Λςετ͍ͨ͠ ֎෦αʔϏεͷϨεϙϯεΛݻఆ͢Δ Response *OQVUͱ֎෦αʔϏεͷϨεϙϯεΛݻఆͨ͠ঢ়ଶͰɺΫϩʔϥʔͷ0VUQVUΛςετ͍ͨ͠ ΫϩʔϥʔͷInputΛݻఆ

Slide 21

Slide 21 text

Ϋϩʔϥʔͷςετʹ͍ͭͯ Amazon  DynamoDB Amazon  RDS Amazon   ElastiCache Amazon  SQS Ϋϩʔϧઌ ֎෦αʔϏε Scrapy spider middleware pipeline Ϋϩʔϧ ϦΫΤετ Request Output Input ςετର৅ ΫϩʔϥʔInput/Outputʹ͍ͭͯ ςετίʔυΛॻ͘ Response ֎෦αʔϏεΛελϒʹஔ͖׵͑ɺΫϩʔϥʔͷ*OQVU0VUQVUʹର͢ΔςετίʔυΛॻ͘ ֎෦αʔϏε ελϒ (middleware) ֎෦αʔϏεͷϨεϙϯεΛγϛϡϨʔτ ͢ΔελϒΛࠩ͠ࠐΉ ֤ DB ͸ localstack Λ࢖ͬͯγϛϡϨʔτ

Slide 22

Slide 22 text

scrapy ʹ͸͜ͷΑ͏ͳػೳ͕ͳ͍ͷͰ ࣗલͰςετͷ࢓૊ΈΛ։ൃ

Slide 23

Slide 23 text

ςετίʔυͷαϯϓϧ ҎԼΛݻఆͯ͠ΫϩʔϥʔΛ࣮ߦ • Ϋϩʔϥʔ΁ͷҾ਺ • ελϒͷϨεϙϯεσʔλ DB ΁ग़ྗ͞Εͨ஋͕૝ఆ௨Γ͔Λ ςετ͢Δ

Slide 24

Slide 24 text

֎෦αʔϏεͷελϒͷαϯϓϧίʔυ ֤ςετέʔε͔Βࢦఆͨ͠σʔλͷ ύεʹஔ͖׵͑ΒΕΔ

Slide 25

Slide 25 text

Ϋϩʔϥʔͷςετ͕͋Δͱ Կ͕͏Ε͍͠ͷʁ

Slide 26

Slide 26 text

TDD ͕Ͱ͖Δʂ

Slide 27

Slide 27 text

scouty Ϋϩʔϥʔͷ CI/CD ύΠϓϥΠϯ Amazon ECS Docker compose ࣗಈςετ • ίʔυϑΥʔϚοτνΣοΫ • unittest ࣮ߦ • Integration test ࣮ߦ • ΫϩʔϧઌαʔϏεͷߏ଄มԽݕ஌ $ git push git push ΛτϦΨʔʹ ࣗಈςετΛ࣮ߦ ࣗಈίʔυϨϏϡʔ $ ecs-goploy HJUQVTIͷ౓ʹࣗಈςετ͕࣮ߦ͞Ε͓ͯΓɺΫϩʔϥʔΛյͨ࣌͠ʹ͙͢ʹؾ͚ͮΔ ΫϩʔϥʔʹݶΒͣTDPVUZͷશͯͷαʔϏεʹಉ༷ͷ$*$%ύΠϓϥΠϯ͕੔උ͞Ε͍ͯΔ

Slide 28

Slide 28 text

scouty Ϋϩʔϥʔͷ ChatOps σϓϩΠ࡞ۀ͸CPUʹΑΓࣗಈԽ͞Ε͍ͯΔͷͰ͍ͭͰ΋୭Ͱ΋࣮ߦͰ͖Δ

Slide 29

Slide 29 text

scouty Ϋϩʔϥʔͷ؂ࢹ Ϋϩʔϥʔͷ֤TUBUTΛऩू͠ಈ࡞ঢ়گΛμογϡϘʔυͰৗʹ֬ೝͰ͖ΔΑ͏ʹ͍ͯ͠Δ

Slide 30

Slide 30 text

ʮࠓޙͷల๬ʯ

Slide 31

Slide 31 text

ʮੈͷதͷϛεϚονΛແ͘͢ʯ

Slide 32

Slide 32 text

ਓʹؔ͢Δ͋ΒΏΔ৘ใΛ஝ੵ͠ɺ࠷ߴͷબ୒ࢶΛఏڙ͢Δɻ  γεςϜʹΑͬͯϛεϚον͕ແ͍ੈքΛ࣮ݱ͢Δɻ scoutyͷ໨ࢦ͢ະདྷɾϏδϣϯ 1ਓ1ݸɺ  scoutyΞΧ΢ϯτΛ  ͍࣋ͬͯΔ ࠃ಺  176Ґ ࠃ಺  176Ґ ࣗࣾͷਓࣄ ଞࣾͷਓࣄ ໘ ஊ ධ Ձ ࣾ ಺ ධ Ձ ೥ ऩ ৘ ใ ͕̋̋ ޷͖ U X ৬ɾαʔϏεͷఏڙऀ ΋ͬͱ೥ऩͷߴ͍࢓ࣄ ΋ͬͱ΍Γ͕͍ͷ͋Δ࢓ࣄ ࣗ෼ʹ͔͠Ͱ͖ͳ͍࢓ࣄ ຊਓ ࣗݾਃࠂ

Slide 33

Slide 33 text

ᅂ޷ੑ৘ใ ʮ͕̋̋޷͖ʯ ೳྗ৘ใ ʮࠃ಺̋̋Ґʯ ৬຿৘ใ ʮ೥ऩ̋̋ສԁʯ ʮೖ͔ࣾΒ̋೥ʯ ߦಈ৘ใ ʮϝοηʔδฦ৴཰̋ˋʯ ʮυλΩϟϯ཰̋ˋʯ ༩৴৘ใ ʮ೥ऩ̋̋ԁʯ ʮ࠴຿ͳ͠ʯ ઐ໳ܥSNS ݸਓܥSNS ࣾ಺ਓࣄ  ͔Βͷ৘ใ ϏδωεSNS ಉ྅͔Βͷ  ϨϑΝϨϯε ΠϕϯτSNS 1ਓ1ݸscoutyΞΧ΢ϯτΛ࣋ͪɺ  ৘ใ͕ϥΠϑϩάͷΑ͏ʹ஝ੵ͢Δ ຊਓ Ӿཡ ৘ใฤू ৬ɾαʔϏε  ͷఏڙऀ ݸਓʹύʔιφϥΠζͨ͠  αʔϏεͷఏڙ Ӿཡ ৽نࣄۀ

Slide 34

Slide 34 text

Ϋϩʔϥʔ։ൃऀϩʔϧͷࠓޙͷల๬ w Ϋϩʔϥʔͷ௥Ճɾӡ༻Λۃݶ·Ͱ؆қԽɾࣗಈԽ w σʔλΛར༻ͨ͠ԾઆݕূΛߴ଎Խ