1日あたり数百万商品をクロールする 大規模クローラーの裏側 / How IQON crawler works

Transcript

1. © 2017 VASILY,Inc.
   ೔͋ͨΓ਺ඦສ঎඼ΛΫϩʔϧ͢Δ
   େن໛Ϋϩʔϥʔͷཪଆ
   4QFFF$BGF.FFUVQ
   Ԙ㟒݈߂
   

   View Slide

2. © 2017 VASILY,Inc.
   ࣗݾ঺հ
   ⾣ Ԙ㟒݈߂
   ⾣ 7"4*-:৽ଔೖࣾ
   ⾣ όοΫΤϯυΤϯδχΞ
   ⾣ 3VCZ
   ⾣ (PPHMF#JH2VFSZ
   ⾣ "QBDIF4PMS
   ⾣ &NCVML
   ⾣ %JHEBH
   ▶ $ crontab -l

   0 0 7 8 * /bin/increment_age
   
   ฐٕࣾज़ސ໰.BU[ࢯ
   

   View Slide

3. © 2017 VASILY,Inc.
   ໊ࣾגࣜձࣾ7"4*-: ϰΝγϦʔ
   7"4*-: *OD
   ઃཱ೥݄
   ॴࡏ஍౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ'
   ैۀһ໊
   ࢿຊۚԯ

   ୅දऔక໾ۚࢁ༟थ

   औక໾ࠓଜխ޾ઍ༿େี

   ओཁגओ
   άϩʔϏεϕϯνϟʔΩϟϐλϧ
   ҏ౻஧ςΫϊϩδʔϕϯνϟʔζ
   (.0ϕϯνϟʔύʔτφʔζ
   ,%%*גࣜձࣾ
   גࣜձࣾߨஊࣾ
   

   View Slide

4. © 2017 VASILY,Inc.
   Ҏ্ͷϑΝογϣϯ&$αΠτ͔Βͷ΂ສ఺Λ௒͑Δ঎඼Λܝࡌ
   ݄ؒສਓҎ্͕ར༻͢Δ೔ຊ࠷େڃͷϑΝογϣϯαΠτ
   

   View Slide

5. © 2017 VASILY,Inc.
   ໨࣍
   
   ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
   ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
   ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
   ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
   ⾣ εέʔϥϒϧͳΠϯϑϥ
   ⾣ %PDLFS.FTPT.BSBUIPOʹΑΔΦʔτεέʔϧ
   ⾣ ·ͱΊ
   

   View Slide

6. © 2017 VASILY,Inc.
   ໨࣍
   
   ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
   ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
   ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
   ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
   ⾣ εέʔϥϒϧͳΠϯϑϥ
   ⾣ %PDLFS.FTPT.BSBUIPOʹΑΔΦʔτεέʔϧ
   ⾣ ·ͱΊ
   

   View Slide

7. © 2017 VASILY,Inc.
   *20/ͷΫϩʔϥʔ
   
   ⾣ ఏܞαΠτ਺Ҏ্
   ⾣ ৗ࣌ߪങՄೳ఺਺ɿສ఺Ҏ্
   

   View Slide

8. © 2017 VASILY,Inc.
   *20/ͷΫϩʔϥʔ
   
   ⾣ ໿߲໨ͷ঎඼৘ใΛऔಘ
   ঎඼ࣸਅ
   Ձ֨
   ࡏݿ਺
   ϒϥϯυ
   ঎඼໊
   

   View Slide

9. © 2017 VASILY,Inc.
   ໨࣍
   
   ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
   ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
   ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
   ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
   ⾣ εέʔϥϒϧͳΠϯϑϥ
   ⾣ %PDLFS.FTPT.BSBUIPOʹΑΔΦʔτεέʔϧ
   ⾣ ·ͱΊ
   

   View Slide

10. © 2017 VASILY,Inc.
    ෼ࢄΫϩʔϦϯά
    ⾣ ର৅ϖʔδ͕๲େͳͷͰ෼ࢄॲཧ
    ⾣ ෼ࢄΫϩʔϧ༻ͷϑϨʔϜϫʔΫ͸ͳ͍
    ⾣ 3VCZΛ༻͍ϑϧεΫϥονͰ࣮૷
    
    4DSBQZEPFTO`UQSPWJEFBOZCVJMUJOGBDJMJUZGPSSVOOJOH
    DSBXMTJOBEJTUSJCVUF NVMUJTFSWFS
    NBOOFS
    IUUQTEPDTDSBQZPSHFOMBUFTUUPQJDTQSBDUJDFTIUNMEJTUSJCVUFEDSBXMT
    

    View Slide

11. © 2017 VASILY,Inc.
    UBTLRVFVFΛհͨ͠෼ࢄɾඇಉظॲཧ
    ⾣ 424ΛUBTLRVFVFͱͯ͠࢖༻
    ⾣ UBTLΛ࣮ߦ͢ΔϓϩηεʢXPSLFSʣಉ࢜͸ૄ݁߹
    ⾣ ඇಉظॲཧϥΠϒϥϦͱͯ͠ɺ4IPSZVLFOΛ࢖༻
    
    XPSLFS XPSLFS
    FORVFVF EFRVFVF
    424
    

    View Slide

12. © 2017 VASILY,Inc.
    4IPSZVLFO
    
    ⾣ 3VCZͰॻ͔ΕͨඇಉظॲཧϑϨʔϜϫʔΫ
    ⾣ ෳ਺Ωϡʔͷ؅ཧػೳ
    ⾣ ϚϧνεϨου
    class HelloWorker
    include Shoryuken::Worker
    shoryuken_options queue: 'hello', auto_delete: true
    def perform(sqs_msg, name)
    puts "Hello, #{name}"
    end
    end
    HelloWorker.perform_async('joe')
    BTZODDBMM
    

    View Slide

13. © 2017 VASILY,Inc.
    શମߏ੒ਤ
    
    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB
    

    View Slide

14. © 2017 VASILY,Inc.
    શମߏ੒ਤ
    
    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB
    

    View Slide

15. © 2017 VASILY,Inc.
    $MPVE8BUDI-BNCEB
    ⾣ ఆظతʹ$MPVE8BUDI&WFOU͕ൃՐ͠ɺ-BNCEBΛىಈ
    ⾣ -BNCEB͕424ʹλεΫΛ౤ೖ͢Δ͜ͱͰɺΫϩʔϧ։࢝
    $MPVE8BUDI 4/4 -BNCEB 424
    JOWPLF FORVFVF
    
    

    View Slide

16. © 2017 VASILY,Inc.
    શମߏ੒ਤ
    
    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB
    

    View Slide

17. © 2017 VASILY,Inc.
    TUBSUDSBXMFSXPSLFS
    ⾣ 3%4͔Βͦͷ೔ͷΫϩʔϧର৅αΠτ৘ใΛऔಘ
    ⾣ ͦΕΒͷαΠτΛΫϩʔϧ͢ΔͨΊͷλεΫΛ౤ೖ
    TUBSUDSBXMFS
    424
    3%4
    Ϋϩʔϧର৅αΠτ৘ใऔಘ
    FORVFVF º&$αΠτͷ਺
    
    

    View Slide

18. © 2017 VASILY,Inc.
    શମߏ੒ਤ
    
    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB
    

    View Slide

19. © 2017 VASILY,Inc.
    MJTUQBHFFORVFVFXPSLFS
    ⾣ &$αΠτͷϖʔδૹΓ෦෼ΛεΫϨΠϐϯά
    ⾣ શϖʔδͷ63-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ
    MJTUQBHFFORVFVF
    424
    ϖʔδૹΓ෦෼ղੳ
    FORVFVF ºϖʔδͷ਺
    
    &$TJUF
    IUUQTFYBNQMFDPNJUFNT QBHF
    IUUQTFYBNQMFDPNJUFNT QBHF
    IUUQTFYBNQMFDPNJUFNT QBHF
    

    View Slide

20. © 2017 VASILY,Inc.
    શମߏ੒ਤ
    
    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB
    

    View Slide

21. © 2017 VASILY,Inc.
    JUFNQBHFFORVFVFXPSLFS
    ⾣ ঎඼Ϧετϖʔδ͔Β঎඼ৄࡉϖʔδ΁ͷ63-Λղੳ
    ⾣ શϖʔδͷ63-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ
    JUFNQBHFFORVFVF
    424
    Ϧετϖʔδˠৄࡉϖʔδͷ63-Λऔಘ
    FORVFVF ºϖʔδͷ਺
    
    &$TJUF
    IUUQTFYBNQMFDPNJUFNT
    IUUQTFYBNQMFDPNJUFNT
    IUUQTFYBNQMFDPNJUFNT
    

    View Slide

22. © 2017 VASILY,Inc.
    શମߏ੒ਤ
    
    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB
    

    View Slide

23. © 2017 VASILY,Inc.
    JUFNQBHFEPXOMPBEXPSLFS
    ⾣ ঎඼ৄࡉϖʔδ͔Β)5.-Λμ΢ϯϩʔυ
    ⾣ μ΢ϯϩʔυִؒͷௐ੔ͷͨΊʹ3FEJTʹ෼ࢄϩοΫʢޙड़ʣΛ࣮ݱ
    ⾣ )5.-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ
    JUFNQBHFFORVFVF
    424
    ৄࡉϖʔδͷ)5.-Λऔಘ
    FORVFVF
    &$TJUF
    
    トップス...
    ϩοΫऔಘ
    

    View Slide

24. © 2017 VASILY,Inc.
    શମߏ੒ਤ
    
    TUBSUDSBXMFS MJTUQBHFFORVFVF
    JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF
    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ
    $MPVE8BUDI-BNCEB
    

    View Slide

25. © 2017 VASILY,Inc.
    JUFNQBHFQBSTFXPSLFS
    ⾣ 91BUIɾਖ਼نදݱΛ࢖͍)5.-Λύʔε
    ⾣ ύʔεઃఆʢ91BUIɾਖ਼نදݱ͸XFCπʔϧͰೖߘ
    ⾣ ύʔε݁ՌΛ%#ʹॻ͖ࠐΉλεΫΛ౤ೖ
    JUFNQBHFQBSTF
    424
    ύʔεઃఆΛऔಘ
    FORVFVF
    {
    "title": "トップス",
    "price": 9800,
    3%4
    ύʔεઃఆΛೖߘ
    
    

    View Slide

26. © 2017 VASILY,Inc.
    ͜ΕҎ߱ͷॲཧ
    ⾣ ࣌ؒͷ౎߹Ͱࠓճ͸ׂѪ
    ⾣ Ϋϩʔϧ݁ՌΛ%#ʹॻ͖ࠐΈ
    ⾣ ը૾ॲཧ
    ⾣ ಁաॲཧ
    ⾣ ୅ද৭நग़
    ⾣ ঎඼ΧςΰϦʔࣗಈ෼ྨ
    
    

    View Slide

27. © 2017 VASILY,Inc.
    μ΢ϯϩʔυִؒ
    
    ⾣ 3FEJTͰ෼ࢄϩοΫΛ࣮ݱ͠ɺμ΢ϯϩʔυִؒΛௐ੔
    IUUQTSFEJTJPUPQJDTEJTUMPDL
    EPXOMPBEXPSLFS" EPXOMPBEXPSLFS#
    HFU@MPDLTVDDFTT
    MPDLFE
    HFU@MPDLGBJM
    EPXOMPBE
    HFU@MPDLGBJM
    FYQJSF
    HFU@MPDLTVDDFTT
    

    View Slide

28. © 2017 VASILY,Inc.
    จࣈίʔυ໰୊
    
    ⾣ NFUBDIBSTFU͸࢖༻͍ͯ͠ͳ͍
    ⾣ ,DPOWʢOLGϥούʔʣͷจࣈίʔυࣗಈਪଌػೳΛར༻
    ▶ ::Kconv.toutf8(str)͚ͩͰ0,
    

    View Slide

29. © 2017 VASILY,Inc.
    41" 4JOHMF1BHF"QQMJDBUJPO
    ͷରԠ
    
    ⾣ 41"͸)5.-ʹ͸΄ͱΜͲͷ৘ใ͕ͳ͍
    ⾣ 1IBOUPN+4Λ࢖ͬͨ1SPYZΛհ͢Δ
    ⾣ PO-PBEΠϕϯτൃՐޙͷ৘ใΛऔಘ
    EPXOMPBEXPSLFS
    &$TJUF
    EPXOMPBE
    

    View Slide

30. © 2017 VASILY,Inc.
    424ͷαΠζ੍ݶ
    
    ⾣ 424ʹ͸,#ҎԼͷςΩετσʔλ͔֨͠ೲͰ͖ͳ͍
    ⾣ Ұ෦ͷϖʔδͷ)5.-͸͜ΕΛ௒ա
    ⾣ )UNM$PNQSFTTPS ;MJC #BTFͰ)5.-Λѹॖ
    ⾣ ʙఔ౓ʹѹॖ
    Base64.encode64(

    Zlib::Deflate.deflate(

    HtmlCompressor::Compressor.new.compress(html)

    )

    )
    

    View Slide

31. © 2017 VASILY,Inc.
    ໨࣍
    
    ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
    ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
    ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
    ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
    ⾣ εέʔϥϒϧͳΠϯϑϥ
    ⾣ %PDLFS.FTPT.BSBUIPOʹΑΔΦʔτεέʔϧ
    ⾣ ·ͱΊ
    

    View Slide

32. © 2017 VASILY,Inc.
    Πϯϑϥߏ੒ਤ
    
    &$TQPUqFFUJOTUBODFT
    XPSLFSTJODPOUBJOFS
    %FQMPZ$POUBJOFS
    424
    -BNCEB
    $MPVE8BUDI
    8BUDI.FUSJDT
    "VUP4MBDF
    FORVFVFEFRVFVF
    

    View Slide

33. © 2017 VASILY,Inc.
    "QBDIF.FTPT.BSBUIPO
    
    ⾣ "QBDIF.FTPT
    ⾣ "EJTUSJCVUFETZTUFNTLFSOFM
    ⾣ ෳ਺ϚγϯΛͭͷܭࢉػϓʔϧͱͯ͠ந৅Խ
    ⾣ .BSBUIPO
    ⾣ .FTPT্Ͱಈ࡞͢ΔίϯςφΦʔέετϨʔγϣϯπʔϧ
    ⾣ .FTPTͷλεΫΛσʔϞϯԽ
    

    View Slide

34. © 2017 VASILY,Inc.
    EPXOMPBEXPSLFSͷՔಇ཰
    
    ίϯςφ਺
    ॲཧ଎౓
    
    ⾣ EPXOMPBEXPSLFSʹ͸ϩοΫ͕͋ΔͨΊɺ޻෉͕ඞཁ
    XJUI޻෉
    XJUIPVU޻෉
    

    View Slide

35. © 2017 VASILY,Inc.
    EPXOMPBEΩϡʔͷଟॏԽ
    
    ⾣ EPXOMPBEΩϡʔɾ෼ࢄϩοΫ͸&$αΠτຖʹಠཱ
    EPXOMPBEXPSLFS
    EPXOMPBEXPSLFS
    ϥϯμϜʹEFRVFVF
    αΠτʹରԠͨ͠
    ෼ࢄϩοΫΛऔಘ
    

    View Slide

36. © 2017 VASILY,Inc.
    ΦʔτεέʔϧͷͨΊͷϝτϦΫε
    
    424
    -BNCEB
    $MPVE8BUDI
    &WFOU
    ⾣ EPXOMPBEΩϡʔ͸ະॲཧͷλεΫͷ਺Λ؂ࢹͯ͠΋ҙຯ͕ͳ͍
    ⾣ ͦͷ୅ΘΓʹɺະॲཧλεΫͷΩϡʔͷݸ਺Λ؂ࢹ
    ⾣ $MPVE8BUDIͰ͸؂ࢹͰ͖ͳ͍
    ⾣ -BNCEBͰ424ͷ"1*Λୟ͘
    JOWPLF
    HFUUIFOVNCFSPG
    OPOFNQUZEPXOMPBERVFVFT
    DIBOHFUIFOVNCFSPGDPOUBJOFST
    

    View Slide

37. © 2017 VASILY,Inc.
    'PSNPSFJOGPSNBUJPO
    
    ⾣ %PDLFS"QBDIF.FTPT.BSBUIPOʹΑΔഒ଎͍*20/Ϋϩʔϥʔͷߏங
    ⾣ IUUQUFDIWBTJMZKQFOUSZJRPODSBXMFSCZEPDLFSBOENFTPTBOENBSBUIPO
    ⾣ "QBDIF.FTPT.BSBUIPOΛຊ൪Ͱӡ༻͢ΔͨΊͷͭͷ5JQT
    ⾣ IUUQUFDIWBTJMZKQFOUSZBQBDIFNFTPTBOENBSBUIPOUJQT
    ⾣ 1SPEVDUJPOEFQMPZNFOUPGUIF%PDLFSDPOUBJOFSXJUI.BSBUIPO
    ⾣ IUUQTTQFBLFSEFDLDPNLPUBUTVQSPEVDUJPOEFQMPZNFOUPGUIFEPDLFSDPOUBJOFSXJUINBSBUIPO
    ⾣ "QBDIF.FTPTXJUI"NB[PO&$4QPU'MFFU
    ⾣ IUUQTTQFBLFSEFDLDPNLPUBUTVBQBDIFNFTPTXJUIBNB[POFDTQPUqFFU
    

    View Slide

38. © 2017 VASILY,Inc.
    ໨࣍
    
    ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
    ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
    ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
    ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
    ⾣ εέʔϥϒϧͳΠϯϑϥ
    ⾣ %PDLFS.FTPT.BSBUIPOʹΑΔΦʔτεέʔϧ
    ⾣ ·ͱΊ
    

    View Slide

39. © 2017 VASILY,Inc.
    ·ͱΊ
    
    ⾣ *20/ͷΫϩʔϥʔ͸ຖ೔਺ඦສ঎඼ΛΫϩʔϧ͍ͯ͠Δ
    ⾣ େن໛ͳ෼ࢄΫϩʔϥʔΛ3VCZͰϑϧεΫϥονͰ࣮૷
    ⾣ ඇಉظॲཧΛ׆༻ͨ͠ॊೈͳΞϓϦέʔγϣϯ
    ⾣ εέʔϥϒϧͳΠϯϑϥͷ্ͰɺεϐʔυΞοϓˍඅ༻࡟ݮ
    

    View Slide

40. © 2017 VASILY,Inc.
    8FSF)JSJOH
    
    IUUQTWBTJMZKQSFDSVJU
    

    View Slide