Pro Yearly is on sale from $80 to $50! »

1日あたり数百万商品をクロールする 大規模クローラーの裏側 / How IQON crawler works

1日あたり数百万商品をクロールする 大規模クローラーの裏側 / How IQON crawler works

D0f23208dff6d54a2a4305e65af569c8?s=128

Takehiro Shiozaki

August 01, 2017
Tweet

Transcript

  1. © 2017 VASILY,Inc. ೔͋ͨΓ਺ඦສ঎඼ΛΫϩʔϧ͢Δ େن໛Ϋϩʔϥʔͷཪଆ 4QFFF$BGF.FFUVQ Ԙ㟒݈߂

  2. © 2017 VASILY,Inc. ࣗݾ঺հ ⾣ Ԙ㟒݈߂ ⾣ 7"4*-:৽ଔೖࣾ ⾣ όοΫΤϯυΤϯδχΞ

    ⾣ 3VCZ ⾣ (PPHMF#JH2VFSZ ⾣ "QBDIF4PMS ⾣ &NCVML ⾣ %JHEBH ▶ $ crontab -l
 0 0 7 8 * /bin/increment_age  ฐٕࣾज़ސ໰.BU[ࢯ
  3. © 2017 VASILY,Inc. ໊ࣾגࣜձࣾ7"4*-: ϰΝγϦʔ 7"4*-: *OD ઃཱ೥݄ ॴࡏ஍౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ' ैۀһ໊

    ࢿຊۚԯ
 ୅දऔక໾ۚࢁ༟थ
 औక໾ࠓଜխ޾ઍ༿େี
 ओཁגओ άϩʔϏεϕϯνϟʔΩϟϐλϧ ҏ౻஧ςΫϊϩδʔϕϯνϟʔζ (.0ϕϯνϟʔύʔτφʔζ ,%%*גࣜձࣾ גࣜձࣾߨஊࣾ
  4. © 2017 VASILY,Inc. Ҏ্ͷϑΝογϣϯ&$αΠτ͔Βͷ΂ສ఺Λ௒͑Δ঎඼Λܝࡌ ݄ؒສਓҎ্͕ར༻͢Δ೔ຊ࠷େڃͷϑΝογϣϯαΠτ

  5. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ
  6. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ
  7. © 2017 VASILY,Inc. *20/ͷΫϩʔϥʔ  ⾣ ఏܞαΠτ਺Ҏ্ ⾣ ৗ࣌ߪങՄೳ఺਺ɿສ఺Ҏ্

  8. © 2017 VASILY,Inc. *20/ͷΫϩʔϥʔ  ⾣ ໿߲໨ͷ঎඼৘ใΛऔಘ ঎඼ࣸਅ Ձ֨ ࡏݿ਺

    ϒϥϯυ ঎඼໊
  9. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ
  10. © 2017 VASILY,Inc. ෼ࢄΫϩʔϦϯά ⾣ ର৅ϖʔδ͕๲େͳͷͰ෼ࢄॲཧ ⾣ ෼ࢄΫϩʔϧ༻ͷϑϨʔϜϫʔΫ͸ͳ͍ ⾣ 3VCZΛ༻͍ϑϧεΫϥονͰ࣮૷

     4DSBQZEPFTO`UQSPWJEFBOZCVJMUJOGBDJMJUZGPSSVOOJOH DSBXMTJOBEJTUSJCVUF NVMUJTFSWFS NBOOFS IUUQTEPDTDSBQZPSHFOMBUFTUUPQJDTQSBDUJDFTIUNMEJTUSJCVUFEDSBXMT
  11. © 2017 VASILY,Inc. UBTLRVFVFΛհͨ͠෼ࢄɾඇಉظॲཧ ⾣ 424ΛUBTLRVFVFͱͯ͠࢖༻ ⾣ UBTLΛ࣮ߦ͢ΔϓϩηεʢXPSLFSʣಉ࢜͸ૄ݁߹ ⾣ ඇಉظॲཧϥΠϒϥϦͱͯ͠ɺ4IPSZVLFOΛ࢖༻

     XPSLFS XPSLFS FORVFVF EFRVFVF 424
  12. © 2017 VASILY,Inc. 4IPSZVLFO  ⾣ 3VCZͰॻ͔ΕͨඇಉظॲཧϑϨʔϜϫʔΫ ⾣ ෳ਺Ωϡʔͷ؅ཧػೳ ⾣

    ϚϧνεϨου class HelloWorker include Shoryuken::Worker shoryuken_options queue: 'hello', auto_delete: true def perform(sqs_msg, name) puts "Hello, #{name}" end end HelloWorker.perform_async('joe') BTZODDBMM
  13. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB
  14. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB
  15. © 2017 VASILY,Inc. $MPVE8BUDI-BNCEB ⾣ ఆظతʹ$MPVE8BUDI&WFOU͕ൃՐ͠ɺ-BNCEBΛىಈ ⾣ -BNCEB͕424ʹλεΫΛ౤ೖ͢Δ͜ͱͰɺΫϩʔϧ։࢝ $MPVE8BUDI 4/4

    -BNCEB 424 JOWPLF FORVFVF 
  16. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB
  17. © 2017 VASILY,Inc. TUBSUDSBXMFSXPSLFS ⾣ 3%4͔Βͦͷ೔ͷΫϩʔϧର৅αΠτ৘ใΛऔಘ ⾣ ͦΕΒͷαΠτΛΫϩʔϧ͢ΔͨΊͷλεΫΛ౤ೖ TUBSUDSBXMFS 424

    3%4 Ϋϩʔϧର৅αΠτ৘ใऔಘ FORVFVF º&$αΠτͷ਺
  18. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB
  19. © 2017 VASILY,Inc. MJTUQBHFFORVFVFXPSLFS ⾣ &$αΠτͷϖʔδૹΓ෦෼ΛεΫϨΠϐϯά ⾣ શϖʔδͷ63-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ MJTUQBHFFORVFVF 424

    ϖʔδૹΓ෦෼ղੳ FORVFVF ºϖʔδͷ਺ &$TJUF IUUQTFYBNQMFDPNJUFNT QBHF IUUQTFYBNQMFDPNJUFNT QBHF IUUQTFYBNQMFDPNJUFNT QBHF
  20. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB
  21. © 2017 VASILY,Inc. JUFNQBHFFORVFVFXPSLFS ⾣ ঎඼Ϧετϖʔδ͔Β঎඼ৄࡉϖʔδ΁ͷ63-Λղੳ ⾣ શϖʔδͷ63-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ JUFNQBHFFORVFVF 424

    Ϧετϖʔδˠৄࡉϖʔδͷ63-Λऔಘ FORVFVF ºϖʔδͷ਺ &$TJUF IUUQTFYBNQMFDPNJUFNT IUUQTFYBNQMFDPNJUFNT IUUQTFYBNQMFDPNJUFNT
  22. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB
  23. © 2017 VASILY,Inc. JUFNQBHFEPXOMPBEXPSLFS ⾣ ঎඼ৄࡉϖʔδ͔Β)5.-Λμ΢ϯϩʔυ ⾣ μ΢ϯϩʔυִؒͷௐ੔ͷͨΊʹ3FEJTʹ෼ࢄϩοΫʢޙड़ʣΛ࣮ݱ ⾣ )5.-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ

    JUFNQBHFFORVFVF 424 ৄࡉϖʔδͷ)5.-Λऔಘ FORVFVF &$TJUF <!DOCTYPE> <HTML><HEAD><TITLE>トップス... ϩοΫऔಘ
  24. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB
  25. © 2017 VASILY,Inc. JUFNQBHFQBSTFXPSLFS ⾣ 91BUIɾਖ਼نදݱΛ࢖͍)5.-Λύʔε ⾣ ύʔεઃఆʢ91BUIɾਖ਼نදݱ͸XFCπʔϧͰೖߘ ⾣ ύʔε݁ՌΛ%#ʹॻ͖ࠐΉλεΫΛ౤ೖ

    JUFNQBHFQBSTF 424 ύʔεઃఆΛऔಘ FORVFVF { "title": "トップス", "price": 9800, 3%4 ύʔεઃఆΛೖߘ 
  26. © 2017 VASILY,Inc. ͜ΕҎ߱ͷॲཧ ⾣ ࣌ؒͷ౎߹Ͱࠓճ͸ׂѪ ⾣ Ϋϩʔϧ݁ՌΛ%#ʹॻ͖ࠐΈ ⾣ ը૾ॲཧ

    ⾣ ಁաॲཧ ⾣ ୅ද৭நग़ ⾣ ঎඼ΧςΰϦʔࣗಈ෼ྨ 
  27. © 2017 VASILY,Inc. μ΢ϯϩʔυִؒ  ⾣ 3FEJTͰ෼ࢄϩοΫΛ࣮ݱ͠ɺμ΢ϯϩʔυִؒΛௐ੔ IUUQTSFEJTJPUPQJDTEJTUMPDL EPXOMPBEXPSLFS" EPXOMPBEXPSLFS#

    HFU@MPDLTVDDFTT MPDLFE HFU@MPDLGBJM EPXOMPBE HFU@MPDLGBJM FYQJSF HFU@MPDLTVDDFTT
  28. © 2017 VASILY,Inc. จࣈίʔυ໰୊  ⾣ NFUBDIBSTFU͸࢖༻͍ͯ͠ͳ͍ ⾣ ,DPOWʢOLGϥούʔʣͷจࣈίʔυࣗಈਪଌػೳΛར༻ ▶

    ::Kconv.toutf8(str)͚ͩͰ0,
  29. © 2017 VASILY,Inc. 41" 4JOHMF1BHF"QQMJDBUJPO ͷରԠ  ⾣ 41"͸)5.-ʹ͸΄ͱΜͲͷ৘ใ͕ͳ͍ ⾣

    1IBOUPN+4Λ࢖ͬͨ1SPYZΛհ͢Δ ⾣ PO-PBEΠϕϯτൃՐޙͷ৘ใΛऔಘ EPXOMPBEXPSLFS &$TJUF EPXOMPBE
  30. © 2017 VASILY,Inc. 424ͷαΠζ੍ݶ  ⾣ 424ʹ͸,#ҎԼͷςΩετσʔλ͔֨͠ೲͰ͖ͳ͍ ⾣ Ұ෦ͷϖʔδͷ)5.-͸͜ΕΛ௒ա ⾣

    )UNM$PNQSFTTPS ;MJC #BTFͰ)5.-Λѹॖ ⾣ ʙఔ౓ʹѹॖ Base64.encode64(
 Zlib::Deflate.deflate(
 HtmlCompressor::Compressor.new.compress(html)
 )
 )
  31. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ
  32. © 2017 VASILY,Inc. Πϯϑϥߏ੒ਤ  &$TQPUqFFUJOTUBODFT XPSLFSTJODPOUBJOFS %FQMPZ$POUBJOFS 424 -BNCEB

    $MPVE8BUDI 8BUDI.FUSJDT "VUP4MBDF FORVFVFEFRVFVF
  33. © 2017 VASILY,Inc. "QBDIF.FTPT.BSBUIPO  ⾣ "QBDIF.FTPT ⾣ "EJTUSJCVUFETZTUFNTLFSOFM ⾣

    ෳ਺ϚγϯΛͭͷܭࢉػϓʔϧͱͯ͠ந৅Խ ⾣ .BSBUIPO ⾣ .FTPT্Ͱಈ࡞͢ΔίϯςφΦʔέετϨʔγϣϯπʔϧ ⾣ .FTPTͷλεΫΛσʔϞϯԽ
  34. © 2017 VASILY,Inc. EPXOMPBEXPSLFSͷՔಇ཰  ίϯςφ਺ ॲཧ଎౓ ⾣ EPXOMPBEXPSLFSʹ͸ϩοΫ͕͋ΔͨΊɺ޻෉͕ඞཁ XJUI޻෉

    XJUIPVU޻෉
  35. © 2017 VASILY,Inc. EPXOMPBEΩϡʔͷଟॏԽ  ⾣ EPXOMPBEΩϡʔɾ෼ࢄϩοΫ͸&$αΠτຖʹಠཱ EPXOMPBEXPSLFS EPXOMPBEXPSLFS ϥϯμϜʹEFRVFVF

    αΠτʹରԠͨ͠ ෼ࢄϩοΫΛऔಘ
  36. © 2017 VASILY,Inc. ΦʔτεέʔϧͷͨΊͷϝτϦΫε  424 -BNCEB $MPVE8BUDI &WFOU ⾣

    EPXOMPBEΩϡʔ͸ະॲཧͷλεΫͷ਺Λ؂ࢹͯ͠΋ҙຯ͕ͳ͍ ⾣ ͦͷ୅ΘΓʹɺະॲཧλεΫͷΩϡʔͷݸ਺Λ؂ࢹ ⾣ $MPVE8BUDIͰ͸؂ࢹͰ͖ͳ͍ ⾣ -BNCEBͰ424ͷ"1*Λୟ͘ JOWPLF HFUUIFOVNCFSPG OPOFNQUZEPXOMPBERVFVFT DIBOHFUIFOVNCFSPGDPOUBJOFST
  37. © 2017 VASILY,Inc. 'PSNPSFJOGPSNBUJPO  ⾣ %PDLFS"QBDIF.FTPT.BSBUIPOʹΑΔഒ଎͍*20/Ϋϩʔϥʔͷߏங ⾣ IUUQUFDIWBTJMZKQFOUSZJRPODSBXMFSCZEPDLFSBOENFTPTBOENBSBUIPO ⾣

    "QBDIF.FTPT.BSBUIPOΛຊ൪Ͱӡ༻͢ΔͨΊͷͭͷ5JQT ⾣ IUUQUFDIWBTJMZKQFOUSZBQBDIFNFTPTBOENBSBUIPOUJQT ⾣ 1SPEVDUJPOEFQMPZNFOUPGUIF%PDLFSDPOUBJOFSXJUI.BSBUIPO ⾣ IUUQTTQFBLFSEFDLDPNLPUBUTVQSPEVDUJPOEFQMPZNFOUPGUIFEPDLFSDPOUBJOFSXJUINBSBUIPO ⾣ "QBDIF.FTPTXJUI"NB[PO&$4QPU'MFFU ⾣ IUUQTTQFBLFSEFDLDPNLPUBUTVBQBDIFNFTPTXJUIBNB[POFDTQPUqFFU
  38. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ
  39. © 2017 VASILY,Inc. ·ͱΊ  ⾣ *20/ͷΫϩʔϥʔ͸ຖ೔਺ඦສ঎඼ΛΫϩʔϧ͍ͯ͠Δ ⾣ େن໛ͳ෼ࢄΫϩʔϥʔΛ3VCZͰϑϧεΫϥονͰ࣮૷ ⾣

    ඇಉظॲཧΛ׆༻ͨ͠ॊೈͳΞϓϦέʔγϣϯ ⾣ εέʔϥϒϧͳΠϯϑϥͷ্ͰɺεϐʔυΞοϓˍඅ༻࡟ݮ
  40. © 2017 VASILY,Inc. 8FSF)JSJOH  IUUQTWBTJMZKQSFDSVJU