Upgrade to Pro — share decks privately, control downloads, hide ads and more …

1日あたり数百万商品をクロールする 大規模クローラーの裏側 / How IQON crawler works

1日あたり数百万商品をクロールする 大規模クローラーの裏側 / How IQON crawler works

Takehiro Shiozaki

August 01, 2017
Tweet

More Decks by Takehiro Shiozaki

Other Decks in Technology

Transcript

  1. © 2017 VASILY,Inc.
    ೔͋ͨΓ਺ඦສ঎඼ΛΫϩʔϧ͢Δ
    େن໛Ϋϩʔϥʔͷཪଆ
    4QFFF$BGF.FFUVQ
    Ԙ㟒݈߂

    View Slide

  2. © 2017 VASILY,Inc.
    ࣗݾ঺հ
    ⾣ Ԙ㟒݈߂
    ⾣ 7"4*-:৽ଔೖࣾ
    ⾣ όοΫΤϯυΤϯδχΞ
    ⾣ 3VCZ
    ⾣ (PPHMF#JH2VFSZ
    ⾣ "QBDIF4PMS
    ⾣ &NCVML
    ⾣ %JHEBH
    ▶ $ crontab -l

    0 0 7 8 * /bin/increment_age

    ฐٕࣾज़ސ໰.BU[ࢯ

    View Slide

  3. © 2017 VASILY,Inc.
    ໊ࣾגࣜձࣾ7"4*-: ϰΝγϦʔ
    7"4*-: *OD
    ઃཱ೥݄
    ॴࡏ஍౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ'
    ैۀһ໊
    ࢿຊۚԯ

    ୅දऔక໾ۚࢁ༟थ

    औక໾ࠓଜխ޾ઍ༿େี

    ओཁגओ
    άϩʔϏεϕϯνϟʔΩϟϐλϧ
    ҏ౻஧ςΫϊϩδʔϕϯνϟʔζ
    (.0ϕϯνϟʔύʔτφʔζ
    ,%%*גࣜձࣾ
    גࣜձࣾߨஊࣾ

    View Slide

  4. © 2017 VASILY,Inc.
    Ҏ্ͷϑΝογϣϯ&$αΠτ͔Βͷ΂ສ఺Λ௒͑Δ঎඼Λܝࡌ
    ݄ؒສਓҎ্͕ར༻͢Δ೔ຊ࠷େڃͷϑΝογϣϯαΠτ

    View Slide

  5. © 2017 VASILY,Inc.
    ໨࣍

    ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
    ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
    ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
    ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
    ⾣ εέʔϥϒϧͳΠϯϑϥ
    ⾣ %PDLFS.FTPT.BSBUIPOʹΑΔΦʔτεέʔϧ
    ⾣ ·ͱΊ

    View Slide

  6. © 2017 VASILY,Inc.
    ໨࣍

    ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
    ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
    ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
    ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
    ⾣ εέʔϥϒϧͳΠϯϑϥ
    ⾣ %PDLFS.FTPT.BSBUIPOʹΑΔΦʔτεέʔϧ
    ⾣ ·ͱΊ

    View Slide

  7. © 2017 VASILY,Inc.
    *20/ͷΫϩʔϥʔ

    ⾣ ఏܞαΠτ਺Ҏ্
    ⾣ ৗ࣌ߪങՄೳ఺਺ɿສ఺Ҏ্

    View Slide

  8. © 2017 VASILY,Inc.
    *20/ͷΫϩʔϥʔ

    ⾣ ໿߲໨ͷ঎඼৘ใΛऔಘ
    ঎඼ࣸਅ
    Ձ֨
    ࡏݿ਺
    ϒϥϯυ
    ঎඼໊

    View Slide

  9. © 2017 VASILY,Inc.
    ໨࣍

    ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
    ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
    ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
    ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
    ⾣ εέʔϥϒϧͳΠϯϑϥ
    ⾣ %PDLFS.FTPT.BSBUIPOʹΑΔΦʔτεέʔϧ
    ⾣ ·ͱΊ

    View Slide

  10. © 2017 VASILY,Inc.
    ෼ࢄΫϩʔϦϯά
    ⾣ ର৅ϖʔδ͕๲େͳͷͰ෼ࢄॲཧ
    ⾣ ෼ࢄΫϩʔϧ༻ͷϑϨʔϜϫʔΫ͸ͳ͍
    ⾣ 3VCZΛ༻͍ϑϧεΫϥονͰ࣮૷

    4DSBQZEPFTO`UQSPWJEFBOZCVJMUJOGBDJMJUZGPSSVOOJOH
    DSBXMTJOBEJTUSJCVUF NVMUJTFSWFS
    NBOOFS
    IUUQTEPDTDSBQZPSHFOMBUFTUUPQJDTQSBDUJDFTIUNMEJTUSJCVUFEDSBXMT

    View Slide

  11. © 2017 VASILY,Inc.
    UBTLRVFVFΛհͨ͠෼ࢄɾඇಉظॲཧ
    ⾣ 424ΛUBTLRVFVFͱͯ͠࢖༻
    ⾣ UBTLΛ࣮ߦ͢ΔϓϩηεʢXPSLFSʣಉ࢜͸ૄ݁߹
    ⾣ ඇಉظॲཧϥΠϒϥϦͱͯ͠ɺ4IPSZVLFOΛ࢖༻

    XPSLFS XPSLFS
    FORVFVF EFRVFVF
    424

    View Slide

  12. © 2017 VASILY,Inc.
    4IPSZVLFO

    ⾣ 3VCZͰॻ͔ΕͨඇಉظॲཧϑϨʔϜϫʔΫ
    ⾣ ෳ਺Ωϡʔͷ؅ཧػೳ
    ⾣ ϚϧνεϨου
    class HelloWorker
    include Shoryuken::Worker
    shoryuken_options queue: 'hello', auto_delete: true
    def perform(sqs_msg, name)
    puts "Hello, #{name}"
    end
    end
    HelloWorker.perform_async('joe')
    BTZODDBMM

    View Slide

  13. © 2017 VASILY,Inc.
    શମߏ੒ਤ

    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB

    View Slide

  14. © 2017 VASILY,Inc.
    શମߏ੒ਤ

    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB

    View Slide

  15. © 2017 VASILY,Inc.
    $MPVE8BUDI-BNCEB
    ⾣ ఆظతʹ$MPVE8BUDI&WFOU͕ൃՐ͠ɺ-BNCEBΛىಈ
    ⾣ -BNCEB͕424ʹλεΫΛ౤ೖ͢Δ͜ͱͰɺΫϩʔϧ։࢝
    $MPVE8BUDI 4/4 -BNCEB 424
    JOWPLF FORVFVF

    View Slide

  16. © 2017 VASILY,Inc.
    શମߏ੒ਤ

    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB

    View Slide

  17. © 2017 VASILY,Inc.
    TUBSUDSBXMFSXPSLFS
    ⾣ 3%4͔Βͦͷ೔ͷΫϩʔϧର৅αΠτ৘ใΛऔಘ
    ⾣ ͦΕΒͷαΠτΛΫϩʔϧ͢ΔͨΊͷλεΫΛ౤ೖ
    TUBSUDSBXMFS
    424
    3%4
    Ϋϩʔϧର৅αΠτ৘ใऔಘ
    FORVFVF º&$αΠτͷ਺

    View Slide

  18. © 2017 VASILY,Inc.
    શମߏ੒ਤ

    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB

    View Slide

  19. © 2017 VASILY,Inc.
    MJTUQBHFFORVFVFXPSLFS
    ⾣ &$αΠτͷϖʔδૹΓ෦෼ΛεΫϨΠϐϯά
    ⾣ શϖʔδͷ63-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ
    MJTUQBHFFORVFVF
    424
    ϖʔδૹΓ෦෼ղੳ
    FORVFVF ºϖʔδͷ਺

    &$TJUF
    IUUQTFYBNQMFDPNJUFNT QBHF
    IUUQTFYBNQMFDPNJUFNT QBHF
    IUUQTFYBNQMFDPNJUFNT QBHF

    View Slide

  20. © 2017 VASILY,Inc.
    શମߏ੒ਤ

    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB

    View Slide

  21. © 2017 VASILY,Inc.
    JUFNQBHFFORVFVFXPSLFS
    ⾣ ঎඼Ϧετϖʔδ͔Β঎඼ৄࡉϖʔδ΁ͷ63-Λղੳ
    ⾣ શϖʔδͷ63-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ
    JUFNQBHFFORVFVF
    424
    Ϧετϖʔδˠৄࡉϖʔδͷ63-Λऔಘ
    FORVFVF ºϖʔδͷ਺

    &$TJUF
    IUUQTFYBNQMFDPNJUFNT
    IUUQTFYBNQMFDPNJUFNT
    IUUQTFYBNQMFDPNJUFNT

    View Slide

  22. © 2017 VASILY,Inc.
    શମߏ੒ਤ

    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB

    View Slide

  23. © 2017 VASILY,Inc.
    JUFNQBHFEPXOMPBEXPSLFS
    ⾣ ঎඼ৄࡉϖʔδ͔Β)5.-Λμ΢ϯϩʔυ
    ⾣ μ΢ϯϩʔυִؒͷௐ੔ͷͨΊʹ3FEJTʹ෼ࢄϩοΫʢޙड़ʣΛ࣮ݱ
    ⾣ )5.-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ
    JUFNQBHFFORVFVF
    424
    ৄࡉϖʔδͷ)5.-Λऔಘ
    FORVFVF
    &$TJUF

    トップス...
    ϩοΫऔಘ

    View Slide

  24. © 2017 VASILY,Inc.
    શମߏ੒ਤ

    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB

    View Slide

  25. © 2017 VASILY,Inc.
    JUFNQBHFQBSTFXPSLFS
    ⾣ 91BUIɾਖ਼نදݱΛ࢖͍)5.-Λύʔε
    ⾣ ύʔεઃఆʢ91BUIɾਖ਼نදݱ͸XFCπʔϧͰೖߘ
    ⾣ ύʔε݁ՌΛ%#ʹॻ͖ࠐΉλεΫΛ౤ೖ
    JUFNQBHFQBSTF
    424
    ύʔεઃఆΛऔಘ
    FORVFVF
    {
    "title": "トップス",
    "price": 9800,
    3%4
    ύʔεઃఆΛೖߘ

    View Slide

  26. © 2017 VASILY,Inc.
    ͜ΕҎ߱ͷॲཧ
    ⾣ ࣌ؒͷ౎߹Ͱࠓճ͸ׂѪ
    ⾣ Ϋϩʔϧ݁ՌΛ%#ʹॻ͖ࠐΈ
    ⾣ ը૾ॲཧ
    ⾣ ಁաॲཧ
    ⾣ ୅ද৭நग़
    ⾣ ঎඼ΧςΰϦʔࣗಈ෼ྨ

    View Slide

  27. © 2017 VASILY,Inc.
    μ΢ϯϩʔυִؒ

    ⾣ 3FEJTͰ෼ࢄϩοΫΛ࣮ݱ͠ɺμ΢ϯϩʔυִؒΛௐ੔
    IUUQTSFEJTJPUPQJDTEJTUMPDL
    EPXOMPBEXPSLFS" EPXOMPBEXPSLFS#
    [email protected]
    MPDLFE
    [email protected]
    EPXOMPBE
    [email protected]
    FYQJSF
    [email protected]

    View Slide

  28. © 2017 VASILY,Inc.
    จࣈίʔυ໰୊

    ⾣ NFUBDIBSTFU͸࢖༻͍ͯ͠ͳ͍
    ⾣ ,DPOWʢOLGϥούʔʣͷจࣈίʔυࣗಈਪଌػೳΛར༻
    ▶ ::Kconv.toutf8(str)͚ͩͰ0,

    View Slide

  29. © 2017 VASILY,Inc.
    41" 4JOHMF1BHF"QQMJDBUJPO
    ͷରԠ

    ⾣ 41"͸)5.-ʹ͸΄ͱΜͲͷ৘ใ͕ͳ͍
    ⾣ 1IBOUPN+4Λ࢖ͬͨ1SPYZΛհ͢Δ
    ⾣ PO-PBEΠϕϯτൃՐޙͷ৘ใΛऔಘ
    EPXOMPBEXPSLFS
    &$TJUF
    EPXOMPBE

    View Slide

  30. © 2017 VASILY,Inc.
    424ͷαΠζ੍ݶ

    ⾣ 424ʹ͸,#ҎԼͷςΩετσʔλ͔֨͠ೲͰ͖ͳ͍
    ⾣ Ұ෦ͷϖʔδͷ)5.-͸͜ΕΛ௒ա
    ⾣ )UNM$PNQSFTTPS ;MJC #BTFͰ)5.-Λѹॖ
    ⾣ ʙఔ౓ʹѹॖ
    Base64.encode64(

    Zlib::Deflate.deflate(

    HtmlCompressor::Compressor.new.compress(html)

    )

    )

    View Slide

  31. © 2017 VASILY,Inc.
    ໨࣍

    ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
    ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
    ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
    ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
    ⾣ εέʔϥϒϧͳΠϯϑϥ
    ⾣ %PDLFS.FTPT.BSBUIPOʹΑΔΦʔτεέʔϧ
    ⾣ ·ͱΊ

    View Slide

  32. © 2017 VASILY,Inc.
    Πϯϑϥߏ੒ਤ

    &$TQPUqFFUJOTUBODFT
    XPSLFSTJODPOUBJOFS
    %FQMPZ$POUBJOFS
    424
    -BNCEB
    $MPVE8BUDI
    8BUDI.FUSJDT
    "VUP4MBDF
    FORVFVFEFRVFVF

    View Slide

  33. © 2017 VASILY,Inc.
    "QBDIF.FTPT.BSBUIPO

    ⾣ "QBDIF.FTPT
    ⾣ "EJTUSJCVUFETZTUFNTLFSOFM
    ⾣ ෳ਺ϚγϯΛͭͷܭࢉػϓʔϧͱͯ͠ந৅Խ
    ⾣ .BSBUIPO
    ⾣ .FTPT্Ͱಈ࡞͢ΔίϯςφΦʔέετϨʔγϣϯπʔϧ
    ⾣ .FTPTͷλεΫΛσʔϞϯԽ

    View Slide

  34. © 2017 VASILY,Inc.
    EPXOMPBEXPSLFSͷՔಇ཰

    ίϯςφ਺
    ॲཧ଎౓

    ⾣ EPXOMPBEXPSLFSʹ͸ϩοΫ͕͋ΔͨΊɺ޻෉͕ඞཁ
    XJUI޻෉
    XJUIPVU޻෉

    View Slide

  35. © 2017 VASILY,Inc.
    EPXOMPBEΩϡʔͷଟॏԽ

    ⾣ EPXOMPBEΩϡʔɾ෼ࢄϩοΫ͸&$αΠτຖʹಠཱ
    EPXOMPBEXPSLFS
    EPXOMPBEXPSLFS
    ϥϯμϜʹEFRVFVF
    αΠτʹରԠͨ͠
    ෼ࢄϩοΫΛऔಘ

    View Slide

  36. © 2017 VASILY,Inc.
    ΦʔτεέʔϧͷͨΊͷϝτϦΫε

    424
    -BNCEB
    $MPVE8BUDI
    &WFOU
    ⾣ EPXOMPBEΩϡʔ͸ະॲཧͷλεΫͷ਺Λ؂ࢹͯ͠΋ҙຯ͕ͳ͍
    ⾣ ͦͷ୅ΘΓʹɺະॲཧλεΫͷΩϡʔͷݸ਺Λ؂ࢹ
    ⾣ $MPVE8BUDIͰ͸؂ࢹͰ͖ͳ͍
    ⾣ -BNCEBͰ424ͷ"1*Λୟ͘
    JOWPLF
    HFUUIFOVNCFSPG
    OPOFNQUZEPXOMPBERVFVFT
    DIBOHFUIFOVNCFSPGDPOUBJOFST

    View Slide

  37. © 2017 VASILY,Inc.
    'PSNPSFJOGPSNBUJPO

    ⾣ %PDLFS"QBDIF.FTPT.BSBUIPOʹΑΔഒ଎͍*20/Ϋϩʔϥʔͷߏங
    ⾣ IUUQUFDIWBTJMZKQFOUSZJRPODSBXMFSCZEPDLFSBOENFTPTBOENBSBUIPO
    ⾣ "QBDIF.FTPT.BSBUIPOΛຊ൪Ͱӡ༻͢ΔͨΊͷͭͷ5JQT
    ⾣ IUUQUFDIWBTJMZKQFOUSZBQBDIFNFTPTBOENBSBUIPOUJQT
    ⾣ 1SPEVDUJPOEFQMPZNFOUPGUIF%PDLFSDPOUBJOFSXJUI.BSBUIPO
    ⾣ IUUQTTQFBLFSEFDLDPNLPUBUTVQSPEVDUJPOEFQMPZNFOUPGUIFEPDLFSDPOUBJOFSXJUINBSBUIPO
    ⾣ "QBDIF.FTPTXJUI"NB[PO&$4QPU'MFFU
    ⾣ IUUQTTQFBLFSEFDLDPNLPUBUTVBQBDIFNFTPTXJUIBNB[POFDTQPUqFFU

    View Slide

  38. © 2017 VASILY,Inc.
    ໨࣍

    ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
    ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
    ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
    ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
    ⾣ εέʔϥϒϧͳΠϯϑϥ
    ⾣ %PDLFS.FTPT.BSBUIPOʹΑΔΦʔτεέʔϧ
    ⾣ ·ͱΊ

    View Slide

  39. © 2017 VASILY,Inc.
    ·ͱΊ

    ⾣ *20/ͷΫϩʔϥʔ͸ຖ೔਺ඦສ঎඼ΛΫϩʔϧ͍ͯ͠Δ
    ⾣ େن໛ͳ෼ࢄΫϩʔϥʔΛ3VCZͰϑϧεΫϥονͰ࣮૷
    ⾣ ඇಉظॲཧΛ׆༻ͨ͠ॊೈͳΞϓϦέʔγϣϯ
    ⾣ εέʔϥϒϧͳΠϯϑϥͷ্ͰɺεϐʔυΞοϓˍඅ༻࡟ݮ

    View Slide

  40. © 2017 VASILY,Inc.
    8FSF)JSJOH

    IUUQTWBTJMZKQSFDSVJU

    View Slide