Upgrade to Pro — share decks privately, control downloads, hide ads and more …

1日あたり数百万商品をクロールする 大規模クローラーの裏側 / How IQON crawler works

Takehiro Shiozaki
August 01, 2017
Technology
3
1.2k

1日あたり数百万商品をクロールする 大規模クローラーの裏側 / How IQON crawler works

Takehiro Shiozaki

August 01, 2017
Tweet

More Decks by Takehiro Shiozaki

See All by Takehiro Shiozaki
タイムトラベルはじめました 〜時をかけるBigQuery〜 / Now serving Time Machine 〜BigQuery Which Leapt Through Time〜
shiozaki
0
2k
これからのZOZOを支える ログ収集基盤を設計した話 / Log collection infrastructure to support ZOZO in the future
shiozaki
4
6.7k
Amazon AuroraのデータをリアルタイムにGoogle BigQueryに連携してみた / Realtime data linkage from Amazon Aurora to Google BigQuery
shiozaki
9
7.6k
ZOZOTOWNの事業を支えるBigQueryの話 / BigQuery behind ZOZOTOWN
shiozaki
6
6.4k
ZOZOTOWNのDWHをRedshiftからBigQueryにお引越しした話 / Moving ZOZOTOWN DWH from Redshift to BigQuery
shiozaki
15
9.8k
ZOZOTOWNのバッチデータ転送基盤紹介 / ZOZOTOWN's data transfer batch
shiozaki
0
390
Digdagを仕事で使ってみて良かったこと、ハマったこと / Using Digdag in production environment
shiozaki
1
1.5k
ファッションIT業界あるある / fashion IT aruaru
shiozaki
1
600
MySQLからBigQueryの同期を差分更新にしたら4倍高速になった話 / Sync from MySQL to BigQuery become 4x faster by incremental updating
shiozaki
6
28k

Other Decks in Technology

See All in Technology
Confident Learning
asei
2
840
220914_ガンダム初学者が色々考えてみた_株式会社コミュカル Mitz
comucal
PRO
0
340
e-RT3 PlusでRclexをいごかしてみる
takasehideki
1
450
自然言語可視化ライブラリ 「nlplot」のご紹介
takapy
3
980
ポイント付与の非同期化 / Event Driven Point Assignment
komotani
0
110
CarPlay向けのアプリって開発できるの？
haseken_dev
0
190
DX_開発管理課
iketomo1207
1
150
SAP Community and Developer Update
sygyzmundovych
0
210
LINE iOSのビルド環境の変遷 / Changes in LINE iOS Build Environment
line_developers
PRO
3
470
JSONについて
miyakemito
1
560
AWSの「今」 - PHPのコードを素早く動かすための サービスのご紹介 / PHPCon2022 AWS Japan Session
koemu
2
940
Building smarter apps with machine learning, from magic to reality
picardparis
4
3.3k

Featured

See All Featured
Building an army of robots
kneath
300
40k
The World Runs on Bad Software
bkeepers
PRO
57
5.4k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
235
1.1M
jQuery: Nuts, Bolts and Bling
dougneiner
56
6.5k
BBQ
matthewcrist
74
8k
Fontdeck: Realign not Redesign
paulrobertlloyd
73
4.2k
Building Flexible Design Systems
yeseniaperezcruz
311
34k
The Pragmatic Product Professional
lauravandoore
19
3.2k
What the flash - Photography Introduction
edds
64
10k
Faster Mobile Websites
deanohume
294
29k
Thoughts on Productivity
jonyablonski
45
2.5k
Side Projects
sachag
451
37k

Transcript

  1. © 2017 VASILY,Inc. ೔͋ͨΓ਺ඦສ঎඼ΛΫϩʔϧ͢Δ େن໛Ϋϩʔϥʔͷཪଆ 4QFFF$BGF.FFUVQ Ԙ㟒݈߂

  2. © 2017 VASILY,Inc. ࣗݾ঺հ ⾣ Ԙ㟒݈߂ ⾣ 7"4*-:৽ଔೖࣾ ⾣ όοΫΤϯυΤϯδχΞ

    ⾣ 3VCZ ⾣ (PPHMF#JH2VFSZ ⾣ "QBDIF4PMS ⾣ &NCVML ⾣ %JHEBH ▶ $ crontab -l
 0 0 7 8 * /bin/increment_age  ฐٕࣾज़ސ໰.BU[ࢯ

  3. © 2017 VASILY,Inc. ໊ࣾגࣜձࣾ7"4*-: ϰΝγϦʔ 7"4*-: *OD ઃཱ೥݄ ॴࡏ஍౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ' ैۀһ໊

    ࢿຊۚԯ
 ୅දऔక໾ۚࢁ༟थ
 औక໾ࠓଜխ޾ઍ༿େี
 ओཁגओ άϩʔϏεϕϯνϟʔΩϟϐλϧ ҏ౻஧ςΫϊϩδʔϕϯνϟʔζ (.0ϕϯνϟʔύʔτφʔζ ,%%*גࣜձࣾ גࣜձࣾߨஊࣾ

  4. © 2017 VASILY,Inc. Ҏ্ͷϑΝογϣϯ&$αΠτ͔Βͷ΂ສ఺Λ௒͑Δ঎඼Λܝࡌ ݄ؒສਓҎ্͕ར༻͢Δ೔ຊ࠷େڃͷϑΝογϣϯαΠτ

  5. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ

  6. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ

  7. © 2017 VASILY,Inc. *20/ͷΫϩʔϥʔ  ⾣ ఏܞαΠτ਺Ҏ্ ⾣ ৗ࣌ߪങՄೳ఺਺ɿສ఺Ҏ্

  8. © 2017 VASILY,Inc. *20/ͷΫϩʔϥʔ  ⾣ ໿߲໨ͷ঎඼৘ใΛऔಘ ঎඼ࣸਅ Ձ֨ ࡏݿ਺

    ϒϥϯυ ঎඼໊

  9. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ

  10. © 2017 VASILY,Inc. ෼ࢄΫϩʔϦϯά ⾣ ର৅ϖʔδ͕๲େͳͷͰ෼ࢄॲཧ ⾣ ෼ࢄΫϩʔϧ༻ͷϑϨʔϜϫʔΫ͸ͳ͍ ⾣ 3VCZΛ༻͍ϑϧεΫϥονͰ࣮૷

     4DSBQZEPFTO`UQSPWJEFBOZCVJMUJOGBDJMJUZGPSSVOOJOH DSBXMTJOBEJTUSJCVUF NVMUJTFSWFS NBOOFS IUUQTEPDTDSBQZPSHFOMBUFTUUPQJDTQSBDUJDFTIUNMEJTUSJCVUFEDSBXMT

  11. © 2017 VASILY,Inc. UBTLRVFVFΛհͨ͠෼ࢄɾඇಉظॲཧ ⾣ 424ΛUBTLRVFVFͱͯ͠࢖༻ ⾣ UBTLΛ࣮ߦ͢ΔϓϩηεʢXPSLFSʣಉ࢜͸ૄ݁߹ ⾣ ඇಉظॲཧϥΠϒϥϦͱͯ͠ɺ4IPSZVLFOΛ࢖༻

     XPSLFS XPSLFS FORVFVF EFRVFVF 424

  12. © 2017 VASILY,Inc. 4IPSZVLFO  ⾣ 3VCZͰॻ͔ΕͨඇಉظॲཧϑϨʔϜϫʔΫ ⾣ ෳ਺Ωϡʔͷ؅ཧػೳ ⾣

    ϚϧνεϨου class HelloWorker include Shoryuken::Worker shoryuken_options queue: 'hello', auto_delete: true def perform(sqs_msg, name) puts "Hello, #{name}" end end HelloWorker.perform_async('joe') BTZODDBMM

  13. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  14. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  15. © 2017 VASILY,Inc. $MPVE8BUDI-BNCEB ⾣ ఆظతʹ$MPVE8BUDI&WFOU͕ൃՐ͠ɺ-BNCEBΛىಈ ⾣ -BNCEB͕424ʹλεΫΛ౤ೖ͢Δ͜ͱͰɺΫϩʔϧ։࢝ $MPVE8BUDI 4/4

    -BNCEB 424 JOWPLF FORVFVF 

  16. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  17. © 2017 VASILY,Inc. TUBSUDSBXMFSXPSLFS ⾣ 3%4͔Βͦͷ೔ͷΫϩʔϧର৅αΠτ৘ใΛऔಘ ⾣ ͦΕΒͷαΠτΛΫϩʔϧ͢ΔͨΊͷλεΫΛ౤ೖ TUBSUDSBXMFS 424

    3%4 Ϋϩʔϧର৅αΠτ৘ใऔಘ FORVFVF º&$αΠτͷ਺

  18. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  19. © 2017 VASILY,Inc. MJTUQBHFFORVFVFXPSLFS ⾣ &$αΠτͷϖʔδૹΓ෦෼ΛεΫϨΠϐϯά ⾣ શϖʔδͷ63-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ MJTUQBHFFORVFVF 424

    ϖʔδૹΓ෦෼ղੳ FORVFVF ºϖʔδͷ਺ &$TJUF IUUQTFYBNQMFDPNJUFNT QBHF IUUQTFYBNQMFDPNJUFNT QBHF IUUQTFYBNQMFDPNJUFNT QBHF

  20. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  21. © 2017 VASILY,Inc. JUFNQBHFFORVFVFXPSLFS ⾣ ঎඼Ϧετϖʔδ͔Β঎඼ৄࡉϖʔδ΁ͷ63-Λղੳ ⾣ શϖʔδͷ63-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ JUFNQBHFFORVFVF 424

    Ϧετϖʔδˠৄࡉϖʔδͷ63-Λऔಘ FORVFVF ºϖʔδͷ਺ &$TJUF IUUQTFYBNQMFDPNJUFNT IUUQTFYBNQMFDPNJUFNT IUUQTFYBNQMFDPNJUFNT

  22. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  23. © 2017 VASILY,Inc. JUFNQBHFEPXOMPBEXPSLFS ⾣ ঎඼ৄࡉϖʔδ͔Β)5.-Λμ΢ϯϩʔυ ⾣ μ΢ϯϩʔυִؒͷௐ੔ͷͨΊʹ3FEJTʹ෼ࢄϩοΫʢޙड़ʣΛ࣮ݱ ⾣ )5.-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ

    JUFNQBHFFORVFVF 424 ৄࡉϖʔδͷ)5.-Λऔಘ FORVFVF &$TJUF <!DOCTYPE> <HTML><HEAD><TITLE>トップス... ϩοΫऔಘ

  24. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF

    ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI-BNCEB

  25. © 2017 VASILY,Inc. JUFNQBHFQBSTFXPSLFS ⾣ 91BUIɾਖ਼نදݱΛ࢖͍)5.-Λύʔε ⾣ ύʔεઃఆʢ91BUIɾਖ਼نදݱ͸XFCπʔϧͰೖߘ ⾣ ύʔε݁ՌΛ%#ʹॻ͖ࠐΉλεΫΛ౤ೖ

    JUFNQBHFQBSTF 424 ύʔεઃఆΛऔಘ FORVFVF { "title": "トップス", "price": 9800, 3%4 ύʔεઃఆΛೖߘ 

  26. © 2017 VASILY,Inc. ͜ΕҎ߱ͷॲཧ ⾣ ࣌ؒͷ౎߹Ͱࠓճ͸ׂѪ ⾣ Ϋϩʔϧ݁ՌΛ%#ʹॻ͖ࠐΈ ⾣ ը૾ॲཧ

    ⾣ ಁաॲཧ ⾣ ୅ද৭நग़ ⾣ ঎඼ΧςΰϦʔࣗಈ෼ྨ 

  27. © 2017 VASILY,Inc. μ΢ϯϩʔυִؒ  ⾣ 3FEJTͰ෼ࢄϩοΫΛ࣮ݱ͠ɺμ΢ϯϩʔυִؒΛௐ੔ IUUQTSFEJTJPUPQJDTEJTUMPDL EPXOMPBEXPSLFS" EPXOMPBEXPSLFS#

    HFU@MPDLTVDDFTT MPDLFE HFU@MPDLGBJM EPXOMPBE HFU@MPDLGBJM FYQJSF HFU@MPDLTVDDFTT

  28. © 2017 VASILY,Inc. จࣈίʔυ໰୊  ⾣ NFUBDIBSTFU͸࢖༻͍ͯ͠ͳ͍ ⾣ ,DPOWʢOLGϥούʔʣͷจࣈίʔυࣗಈਪଌػೳΛར༻ ▶

    ::Kconv.toutf8(str)͚ͩͰ0,

  29. © 2017 VASILY,Inc. 41" 4JOHMF1BHF"QQMJDBUJPO ͷରԠ  ⾣ 41"͸)5.-ʹ͸΄ͱΜͲͷ৘ใ͕ͳ͍ ⾣

    1IBOUPN+4Λ࢖ͬͨ1SPYZΛհ͢Δ ⾣ PO-PBEΠϕϯτൃՐޙͷ৘ใΛऔಘ EPXOMPBEXPSLFS &$TJUF EPXOMPBE

  30. © 2017 VASILY,Inc. 424ͷαΠζ੍ݶ  ⾣ 424ʹ͸,#ҎԼͷςΩετσʔλ͔֨͠ೲͰ͖ͳ͍ ⾣ Ұ෦ͷϖʔδͷ)5.-͸͜ΕΛ௒ա ⾣

    )UNM$PNQSFTTPS ;MJC #BTFͰ)5.-Λѹॖ ⾣ ʙఔ౓ʹѹॖ Base64.encode64(
 Zlib::Deflate.deflate(
 HtmlCompressor::Compressor.new.compress(html)
 )
 )

  31. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ

  32. © 2017 VASILY,Inc. Πϯϑϥߏ੒ਤ  &$TQPUqFFUJOTUBODFT XPSLFSTJODPOUBJOFS %FQMPZ$POUBJOFS 424 -BNCEB

    $MPVE8BUDI 8BUDI.FUSJDT "VUP4MBDF FORVFVFEFRVFVF

  33. © 2017 VASILY,Inc. "QBDIF.FTPT.BSBUIPO  ⾣ "QBDIF.FTPT ⾣ "EJTUSJCVUFETZTUFNTLFSOFM ⾣

    ෳ਺ϚγϯΛͭͷܭࢉػϓʔϧͱͯ͠ந৅Խ ⾣ .BSBUIPO ⾣ .FTPT্Ͱಈ࡞͢ΔίϯςφΦʔέετϨʔγϣϯπʔϧ ⾣ .FTPTͷλεΫΛσʔϞϯԽ

  34. © 2017 VASILY,Inc. EPXOMPBEXPSLFSͷՔಇ཰  ίϯςφ਺ ॲཧ଎౓ ⾣ EPXOMPBEXPSLFSʹ͸ϩοΫ͕͋ΔͨΊɺ޻෉͕ඞཁ XJUI޻෉

    XJUIPVU޻෉

  35. © 2017 VASILY,Inc. EPXOMPBEΩϡʔͷଟॏԽ  ⾣ EPXOMPBEΩϡʔɾ෼ࢄϩοΫ͸&$αΠτຖʹಠཱ EPXOMPBEXPSLFS EPXOMPBEXPSLFS ϥϯμϜʹEFRVFVF

    αΠτʹରԠͨ͠ ෼ࢄϩοΫΛऔಘ

  36. © 2017 VASILY,Inc. ΦʔτεέʔϧͷͨΊͷϝτϦΫε  424 -BNCEB $MPVE8BUDI &WFOU ⾣

    EPXOMPBEΩϡʔ͸ະॲཧͷλεΫͷ਺Λ؂ࢹͯ͠΋ҙຯ͕ͳ͍ ⾣ ͦͷ୅ΘΓʹɺະॲཧλεΫͷΩϡʔͷݸ਺Λ؂ࢹ ⾣ $MPVE8BUDIͰ͸؂ࢹͰ͖ͳ͍ ⾣ -BNCEBͰ424ͷ"1*Λୟ͘ JOWPLF HFUUIFOVNCFSPG OPOFNQUZEPXOMPBERVFVFT DIBOHFUIFOVNCFSPGDPOUBJOFST

  37. © 2017 VASILY,Inc. 'PSNPSFJOGPSNBUJPO  ⾣ %PDLFS"QBDIF.FTPT.BSBUIPOʹΑΔഒ଎͍*20/Ϋϩʔϥʔͷߏங ⾣ IUUQUFDIWBTJMZKQFOUSZJRPODSBXMFSCZEPDLFSBOENFTPTBOENBSBUIPO ⾣

    "QBDIF.FTPT.BSBUIPOΛຊ൪Ͱӡ༻͢ΔͨΊͷͭͷ5JQT ⾣ IUUQUFDIWBTJMZKQFOUSZBQBDIFNFTPTBOENBSBUIPOUJQT ⾣ 1SPEVDUJPOEFQMPZNFOUPGUIF%PDLFSDPOUBJOFSXJUI.BSBUIPO ⾣ IUUQTTQFBLFSEFDLDPNLPUBUTVQSPEVDUJPOEFQMPZNFOUPGUIFEPDLFSDPOUBJOFSXJUINBSBUIPO ⾣ "QBDIF.FTPTXJUI"NB[PO&$4QPU'MFFU ⾣ IUUQTTQFBLFSEFDLDPNLPUBUTVBQBDIFNFTPTXJUIBNB[POFDTQPUqFFU

  38. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ ⾣

    ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़ ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ ⾣ εέʔϥϒϧͳΠϯϑϥ ⾣ %PDLFS .FTPT .BSBUIPOʹΑΔΦʔτεέʔϧ ⾣ ·ͱΊ

  39. © 2017 VASILY,Inc. ·ͱΊ  ⾣ *20/ͷΫϩʔϥʔ͸ຖ೔਺ඦສ঎඼ΛΫϩʔϧ͍ͯ͠Δ ⾣ େن໛ͳ෼ࢄΫϩʔϥʔΛ3VCZͰϑϧεΫϥονͰ࣮૷ ⾣

    ඇಉظॲཧΛ׆༻ͨ͠ॊೈͳΞϓϦέʔγϣϯ ⾣ εέʔϥϒϧͳΠϯϑϥͷ্ͰɺεϐʔυΞοϓˍඅ༻࡟ݮ

  40. © 2017 VASILY,Inc. 8FSF)JSJOH  IUUQTWBTJMZKQSFDSVJU