Pro Yearly is on sale from $80 to $50! »

1日あたり数百万商品をクロールする 大規模クローラーの裏側 / How IQON crawler works

D0f23208dff6d54a2a4305e65af569c8?s=47 Takehiro Shiozaki
August 01, 2017
Technology
3
980

1日あたり数百万商品をクロールする 大規模クローラーの裏側 / How IQON crawler works

D0f23208dff6d54a2a4305e65af569c8?s=128

Takehiro Shiozaki

August 01, 2017
Tweet

More Decks by Takehiro Shiozaki

See All by Takehiro Shiozaki
D0f23208dff6d54a2a4305e65af569c8?s=48 shiozaki
5
3.9k
D0f23208dff6d54a2a4305e65af569c8?s=48 shiozaki
14
8.8k
D0f23208dff6d54a2a4305e65af569c8?s=48 shiozaki
0
310
D0f23208dff6d54a2a4305e65af569c8?s=48 shiozaki
1
1.2k
D0f23208dff6d54a2a4305e65af569c8?s=48 shiozaki
1
470
D0f23208dff6d54a2a4305e65af569c8?s=48 shiozaki
5
20k
D0f23208dff6d54a2a4305e65af569c8?s=48 shiozaki
0
750
D0f23208dff6d54a2a4305e65af569c8?s=48 shiozaki
0
4.8k
D0f23208dff6d54a2a4305e65af569c8?s=48 shiozaki
15
23k

Other Decks in Technology

See All in Technology
11d5350db3f389d0f5749b91ee11a916?s=48 indigo13love
1
180
79dcb04101764d7bf66f898dd446838e?s=48 tsukubachallenge
0
170
Eebedc2ee7ff95ffb9d9102c6d4a065c?s=48 line_devday2020
0
100
Eebedc2ee7ff95ffb9d9102c6d4a065c?s=48 line_devday2020
0
120
Fe06c08a0de4da36003bfeb7b749cbf8?s=48 ryotaumebayashi
1
210
9e7a653ed313b3da4b263a772bfc5026?s=48 clustervr
0
160
38f77168b7c5802adc9cc3a5cfcf031b?s=48 niharika28
0
270
23f4d5d797a91b6d17d627b90b5a42d9?s=48 kentaro
3
300
Eebedc2ee7ff95ffb9d9102c6d4a065c?s=48 line_devday2020
0
410
6bbb729118654b4f0d72dce782362487?s=48 kouxyz21
0
460
4b2f3a64637b51e81813accbe8a98083?s=48 miura55
0
120
Eebedc2ee7ff95ffb9d9102c6d4a065c?s=48 line_devday2020
0
110

Featured

See All Featured
1177e050db6bafe62885362edf6e3537?s=48 lauravandoore
436
28k
6bb82b13cda86d3b821fa44100335ddf?s=48 thoeni
4
490
0c6fd6a08d0f898159cda7cafcacf07f?s=48 afnizarnur
176
14k
C157b16234f1e75e8eac3698c1d4414a?s=48 ddemaree
274
31k
E76911cbe088e5b850d966de3fc7435b?s=48 robhawkes
52
2.7k
15f2b8221f247985a27a627d2ece72f0?s=48 pauljervisheath
195
15k
Ef32e0b1ac5082450dc8c1fca3a26b5c?s=48 cherdarchuk
66
260k
C4de88ea47538e4594750e3861a341c8?s=48 brettharned
92
2.9k
0bce06bd06ee7a2c4f5500f25dcf7f18?s=48 paulrobertlloyd
71
3.5k
Dad095ea7038f89f760419ce475d5d14?s=48 michaelherold
225
8.4k
A16eb159db895e3b01d3dc95767ad595?s=48 jonyablonski
10
940
32de0bd2ba869609d26fd052a4622778?s=48 cromwellryan
101
5.8k

Transcript

  1. © 2017 VASILY,Inc. ೔͋ͨΓ਺ඦສ঎඼ΛΫϩʔϧ͢Δ େن໛Ϋϩʔϥʔͷཪଆ 4QFFF$BGF.FFUVQ Ԙ㟒݈߂

  2. © 2017 VASILY,Inc. ࣗݾ঺հ ⾣ Ԙ㟒݈߂ ⾣ 7"4*-:৽ଔೖࣾ ⾣ όοΫΤϯυΤϯδχΞ

    ⾣ 3VCZ ⾣ (PPHMF#JH2VFSZ ⾣ "QBDIF4PMS ⾣ &NCVML ⾣ %JHEBH ▶ $ crontab -l
 0 0 7 8 * /bin/increment_age  ฐٕࣾज़ސ໰.BU[ࢯ

  3. © 2017 VASILY,Inc. ໊ࣾגࣜձࣾ7"4*-: ϰΝγϦʔ 7"4*-: *OD ઃཱ೥݄ ॴࡏ஍౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ' ैۀһ໊

    ࢿຊۚԯ
 ୅දऔక໾ۚࢁ༟थ
 औక໾ࠓଜխ޾ઍ༿େี
 ओཁגओ άϩʔϏεϕϯνϟʔΩϟϐλϧ ҏ౻஧ςΫϊϩδʔϕϯνϟʔζ (.0ϕϯνϟʔύʔτφʔζ ,%%*גࣜձࣾ גࣜձࣾߨஊࣾ

  4. © 2017 VASILY,Inc. Ҏ্ͷϑΝογϣϯ&$αΠτ͔Βͷ΂ສ఺Λ௒͑Δ঎඼Λܝࡌ ݄ؒສਓҎ্͕ར༻͢Δ೔ຊ࠷େڃͷϑΝογϣϯαΠτ

  5. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ

  6. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ

  7. © 2017 VASILY,Inc. *20/ͷΫϩʔϥʔ  ⾣ ఏܞαΠτ਺Ҏ্ ⾣ ৗ࣌ߪങՄೳ఺਺ɿສ఺Ҏ্

  8. © 2017 VASILY,Inc. *20/ͷΫϩʔϥʔ  ⾣ ໿߲໨ͷ঎඼৘ใΛऔಘ ঎඼ࣸਅ Ձ֨ ࡏݿ਺

    ϒϥϯυ ঎඼໊

  9. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ

  10. © 2017 VASILY,Inc. ෼ࢄΫϩʔϦϯά ⾣ ର৅ϖʔδ͕๲େͳͷͰ෼ࢄॲཧ ⾣ ෼ࢄΫϩʔϧ༻ͷϑϨʔϜϫʔΫ͸ͳ͍ ⾣ 3VCZΛ༻͍ϑϧεΫϥονͰ࣮૷

     4DSBQZEPFTO`UQSPWJEFBOZCVJMUJOGBDJMJUZGPSSVOOJOH DSBXMTJOBEJTUSJCVUF NVMUJTFSWFS NBOOFS IUUQTEPDTDSBQZPSHFOMBUFTUUPQJDTQSBDUJDFTIUNMEJTUSJCVUFEDSBXMT

  11. © 2017 VASILY,Inc. UBTLRVFVFΛհͨ͠෼ࢄɾඇಉظॲཧ ⾣ 424ΛUBTLRVFVFͱͯ͠࢖༻ ⾣ UBTLΛ࣮ߦ͢ΔϓϩηεʢXPSLFSʣಉ࢜͸ૄ݁߹ ⾣ ඇಉظॲཧϥΠϒϥϦͱͯ͠ɺ4IPSZVLFOΛ࢖༻

     XPSLFS XPSLFS FORVFVF EFRVFVF 424

  12. © 2017 VASILY,Inc. 4IPSZVLFO  ⾣ 3VCZͰॻ͔ΕͨඇಉظॲཧϑϨʔϜϫʔΫ ⾣ ෳ਺Ωϡʔͷ؅ཧػೳ ⾣

    ϚϧνεϨου class HelloWorker include Shoryuken::Worker shoryuken_options queue: 'hello', auto_delete: true def perform(sqs_msg, name) puts "Hello, #{name}" end end HelloWorker.perform_async('joe') BTZODDBMM

  13. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  14. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  15. © 2017 VASILY,Inc. $MPVE8BUDI-BNCEB ⾣ ఆظతʹ$MPVE8BUDI&WFOU͕ൃՐ͠ɺ-BNCEBΛىಈ ⾣ -BNCEB͕424ʹλεΫΛ౤ೖ͢Δ͜ͱͰɺΫϩʔϧ։࢝ $MPVE8BUDI 4/4

    -BNCEB 424 JOWPLF FORVFVF 

  16. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  17. © 2017 VASILY,Inc. TUBSUDSBXMFSXPSLFS ⾣ 3%4͔Βͦͷ೔ͷΫϩʔϧର৅αΠτ৘ใΛऔಘ ⾣ ͦΕΒͷαΠτΛΫϩʔϧ͢ΔͨΊͷλεΫΛ౤ೖ TUBSUDSBXMFS 424

    3%4 Ϋϩʔϧର৅αΠτ৘ใऔಘ FORVFVF º&$αΠτͷ਺

  18. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  19. © 2017 VASILY,Inc. MJTUQBHFFORVFVFXPSLFS ⾣ &$αΠτͷϖʔδૹΓ෦෼ΛεΫϨΠϐϯά ⾣ શϖʔδͷ63-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ MJTUQBHFFORVFVF 424

    ϖʔδૹΓ෦෼ղੳ FORVFVF ºϖʔδͷ਺ &$TJUF IUUQTFYBNQMFDPNJUFNT QBHF IUUQTFYBNQMFDPNJUFNT QBHF IUUQTFYBNQMFDPNJUFNT QBHF

  20. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  21. © 2017 VASILY,Inc. JUFNQBHFFORVFVFXPSLFS ⾣ ঎඼Ϧετϖʔδ͔Β঎඼ৄࡉϖʔδ΁ͷ63-Λղੳ ⾣ શϖʔδͷ63-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ JUFNQBHFFORVFVF 424

    Ϧετϖʔδˠৄࡉϖʔδͷ63-Λऔಘ FORVFVF ºϖʔδͷ਺ &$TJUF IUUQTFYBNQMFDPNJUFNT IUUQTFYBNQMFDPNJUFNT IUUQTFYBNQMFDPNJUFNT

  22. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  23. © 2017 VASILY,Inc. JUFNQBHFEPXOMPBEXPSLFS ⾣ ঎඼ৄࡉϖʔδ͔Β)5.-Λμ΢ϯϩʔυ ⾣ μ΢ϯϩʔυִؒͷௐ੔ͷͨΊʹ3FEJTʹ෼ࢄϩοΫʢޙड़ʣΛ࣮ݱ ⾣ )5.-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ

    JUFNQBHFFORVFVF 424 ৄࡉϖʔδͷ)5.-Λऔಘ FORVFVF &$TJUF <!DOCTYPE> <HTML><HEAD><TITLE>トップス... ϩοΫऔಘ

  24. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  25. © 2017 VASILY,Inc. JUFNQBHFQBSTFXPSLFS ⾣ 91BUIɾਖ਼نදݱΛ࢖͍)5.-Λύʔε ⾣ ύʔεઃఆʢ91BUIɾਖ਼نදݱ͸XFCπʔϧͰೖߘ ⾣ ύʔε݁ՌΛ%#ʹॻ͖ࠐΉλεΫΛ౤ೖ

    JUFNQBHFQBSTF 424 ύʔεઃఆΛऔಘ FORVFVF { "title": "トップス", "price": 9800, 3%4 ύʔεઃఆΛೖߘ 

  26. © 2017 VASILY,Inc. ͜ΕҎ߱ͷॲཧ ⾣ ࣌ؒͷ౎߹Ͱࠓճ͸ׂѪ ⾣ Ϋϩʔϧ݁ՌΛ%#ʹॻ͖ࠐΈ ⾣ ը૾ॲཧ

    ⾣ ಁաॲཧ ⾣ ୅ද৭நग़ ⾣ ঎඼ΧςΰϦʔࣗಈ෼ྨ 

  27. © 2017 VASILY,Inc. μ΢ϯϩʔυִؒ  ⾣ 3FEJTͰ෼ࢄϩοΫΛ࣮ݱ͠ɺμ΢ϯϩʔυִؒΛௐ੔ IUUQTSFEJTJPUPQJDTEJTUMPDL EPXOMPBEXPSLFS" EPXOMPBEXPSLFS#

    HFU@MPDLTVDDFTT MPDLFE HFU@MPDLGBJM EPXOMPBE HFU@MPDLGBJM FYQJSF HFU@MPDLTVDDFTT

  28. © 2017 VASILY,Inc. จࣈίʔυ໰୊  ⾣ NFUBDIBSTFU͸࢖༻͍ͯ͠ͳ͍ ⾣ ,DPOWʢOLGϥούʔʣͷจࣈίʔυࣗಈਪଌػೳΛར༻ ▶

    ::Kconv.toutf8(str)͚ͩͰ0,

  29. © 2017 VASILY,Inc. 41" 4JOHMF1BHF"QQMJDBUJPO ͷରԠ  ⾣ 41"͸)5.-ʹ͸΄ͱΜͲͷ৘ใ͕ͳ͍ ⾣

    1IBOUPN+4Λ࢖ͬͨ1SPYZΛհ͢Δ ⾣ PO-PBEΠϕϯτൃՐޙͷ৘ใΛऔಘ EPXOMPBEXPSLFS &$TJUF EPXOMPBE

  30. © 2017 VASILY,Inc. 424ͷαΠζ੍ݶ  ⾣ 424ʹ͸,#ҎԼͷςΩετσʔλ͔֨͠ೲͰ͖ͳ͍ ⾣ Ұ෦ͷϖʔδͷ)5.-͸͜ΕΛ௒ա ⾣

    )UNM$PNQSFTTPS ;MJC #BTFͰ)5.-Λѹॖ ⾣ ʙఔ౓ʹѹॖ Base64.encode64(
 Zlib::Deflate.deflate(
 HtmlCompressor::Compressor.new.compress(html)
 )
 )

  31. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ

  32. © 2017 VASILY,Inc. Πϯϑϥߏ੒ਤ  &$TQPUqFFUJOTUBODFT XPSLFSTJODPOUBJOFS %FQMPZ$POUBJOFS 424 -BNCEB

    $MPVE8BUDI 8BUDI.FUSJDT "VUP4MBDF FORVFVFEFRVFVF

  33. © 2017 VASILY,Inc. "QBDIF.FTPT.BSBUIPO  ⾣ "QBDIF.FTPT ⾣ "EJTUSJCVUFETZTUFNTLFSOFM ⾣

    ෳ਺ϚγϯΛͭͷܭࢉػϓʔϧͱͯ͠ந৅Խ ⾣ .BSBUIPO ⾣ .FTPT্Ͱಈ࡞͢ΔίϯςφΦʔέετϨʔγϣϯπʔϧ ⾣ .FTPTͷλεΫΛσʔϞϯԽ

  34. © 2017 VASILY,Inc. EPXOMPBEXPSLFSͷՔಇ཰  ίϯςφ਺ ॲཧ଎౓ ⾣ EPXOMPBEXPSLFSʹ͸ϩοΫ͕͋ΔͨΊɺ޻෉͕ඞཁ XJUI޻෉

    XJUIPVU޻෉

  35. © 2017 VASILY,Inc. EPXOMPBEΩϡʔͷଟॏԽ  ⾣ EPXOMPBEΩϡʔɾ෼ࢄϩοΫ͸&$αΠτຖʹಠཱ EPXOMPBEXPSLFS EPXOMPBEXPSLFS ϥϯμϜʹEFRVFVF

    αΠτʹରԠͨ͠ ෼ࢄϩοΫΛऔಘ

  36. © 2017 VASILY,Inc. ΦʔτεέʔϧͷͨΊͷϝτϦΫε  424 -BNCEB $MPVE8BUDI &WFOU ⾣

    EPXOMPBEΩϡʔ͸ະॲཧͷλεΫͷ਺Λ؂ࢹͯ͠΋ҙຯ͕ͳ͍ ⾣ ͦͷ୅ΘΓʹɺະॲཧλεΫͷΩϡʔͷݸ਺Λ؂ࢹ ⾣ $MPVE8BUDIͰ͸؂ࢹͰ͖ͳ͍ ⾣ -BNCEBͰ424ͷ"1*Λୟ͘ JOWPLF HFUUIFOVNCFSPG OPOFNQUZEPXOMPBERVFVFT DIBOHFUIFOVNCFSPGDPOUBJOFST

  37. © 2017 VASILY,Inc. 'PSNPSFJOGPSNBUJPO  ⾣ %PDLFS"QBDIF.FTPT.BSBUIPOʹΑΔഒ଎͍*20/Ϋϩʔϥʔͷߏங ⾣ IUUQUFDIWBTJMZKQFOUSZJRPODSBXMFSCZEPDLFSBOENFTPTBOENBSBUIPO ⾣

    "QBDIF.FTPT.BSBUIPOΛຊ൪Ͱӡ༻͢ΔͨΊͷͭͷ5JQT ⾣ IUUQUFDIWBTJMZKQFOUSZBQBDIFNFTPTBOENBSBUIPOUJQT ⾣ 1SPEVDUJPOEFQMPZNFOUPGUIF%PDLFSDPOUBJOFSXJUI.BSBUIPO ⾣ IUUQTTQFBLFSEFDLDPNLPUBUTVQSPEVDUJPOEFQMPZNFOUPGUIFEPDLFSDPOUBJOFSXJUINBSBUIPO ⾣ "QBDIF.FTPTXJUI"NB[PO&$4QPU'MFFU ⾣ IUUQTTQFBLFSEFDLDPNLPUBUTVBQBDIFNFTPTXJUIBNB[POFDTQPUqFFU

  38. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ

  39. © 2017 VASILY,Inc. ·ͱΊ  ⾣ *20/ͷΫϩʔϥʔ͸ຖ೔਺ඦສ঎඼ΛΫϩʔϧ͍ͯ͠Δ ⾣ େن໛ͳ෼ࢄΫϩʔϥʔΛ3VCZͰϑϧεΫϥονͰ࣮૷ ⾣

    ඇಉظॲཧΛ׆༻ͨ͠ॊೈͳΞϓϦέʔγϣϯ ⾣ εέʔϥϒϧͳΠϯϑϥͷ্ͰɺεϐʔυΞοϓˍඅ༻࡟ݮ

  40. © 2017 VASILY,Inc. 8FSF)JSJOH  IUUQTWBTJMZKQSFDSVJU