高速なBitTorrent DHTクローラ

Transcript

1. ߴ଎ͳ#JU5PSSFOU
   %)5Ϋϩʔϥ ೥݄೔
   શମྠߨϑϦʔτʔΫ
   Լᑍݚ
   %
   ߴڮܛஐ
2. #JU5PSSFOU w ੈք࠷େͷ11ϑΝΠϧڞ༗ιϑτ΢ΣΞ
   ‣ ৗ࣌ສਓʙສਓͷΦϯϥΠϯϢʔβ
   ೥
   ‣ ԯສਓͷΞΫςΟϒϢʔβ
   ೥
   w ༷ʑͳԠ༻ྫ
   ‣ #JU5PSSFOU
   4ZOD
   11൛%SPQCPY
   ‣ ΦϯϥΠϯήʔϜͷΞοϓσʔτͷ഑৴
   $"1$0.
   #MJ[[BSE
   ʜ
   ‣ Ϋϥελʹ͓͚ΔΞϓϦέʔγϣϯͷσϓϩΠπʔϧɾ
 ڞ༗ϑΝΠϧγεςϜͱͯ͠ 
3. #JU5PSSFOUͷ࢓૊Έ
   %)5ແ  5PSSFOUϑΝΠϧ ݕࡧΫΤϦ ϐΞऔಘϦΫΤετ ϐΞ৘ใ ϐʔεϦΫΤετ ϐʔε 4FFE1FFS 5SBDLFS *OEFYJOH
   8FCTJUF
4. %JTUSJCVUFE
   )BTI
   5BCMF
   %)5 w )BTI
   5BCMF
   ‣ Ωʔ͔Βޮ཰తʹ஋ΛݕࡧͰ͖Δσʔλߏ଄
   w %JTUSJCVUFE
   )BTI
   5BCMF
   ‣ σʔλΛෳ਺ͷίϯϐϡʔλ্ʹ෼ࢄ͢Δ)BTI
   5BCMF
   w ैདྷͷ#JU5PSSFOUͷऑ఺
   ‣ 5SBDLFS͕ఀࢭ͢ΔͱɺϐΞͷ৘ใͷ୳ࡧ͕Ͱ͖ͳ͘ͳΓɺ
 μ΢ϯϩʔυɾΞοϓϩʔυෆೳʹͳΔ
   ‣ ϐΞ਺͕૿Ճ͢Δͱ5SBDLFS͕ϘτϧωοΫʹͳΓ͏Δ
   w #JU5PSSFOU
   .BJOMJOF
   %)5
   ‣ ϐΞͷ৘ใΛ%)5ʹอଘ͢Δ͜ͱͰɺ଱ো֐ੑͱεέʔϥϏϦςΟ Λ޲্ 
5. #JU5PSSFOUͷ࢓૊Έ
   %)5༗  4FFE1FFS
   
   %)5
   /PEF %)5ʹࢀՃ ϐΞ৘ใऔಘ ϐʔεऔಘ
6. #JU5PSSFOU
   .BJOMJOF
   %)5ͷ࢓૊Έ w ,BEFNMJBΛϕʔεʹͨ͠%)5
   ‣ CJUͷΞυϨεۭؒ
   ‣ 903ܭྔͱ෼໦ͰΞυϨεۭؒΛߏ଄Խ
   w %)5ϊʔυؒ͸,31$ͱ͍͏ϓϩτίϧͰ௨৴
   ‣ 6%1্ͷඇಉظతͳϓϩτίϧ
   w QJOH
   pOE@OPEF
   HFU@QFFS
   BOOOPVODF@QFFSͷछͷϝοηʔδ
   ‣ QJOH JE 
   ϊʔυͷੜࢮΛ֬ೝ
   ‣ pOE@OPEF JE
   UBSHFU 
   UBSHFUʹۙͦ͏ͳϊʔυΛऔಘ
   ‣ HFU@QFFS JE
   JOGPIBTI 
   JOGPIBTIΛ͍࣋ͬͯͦ͏ͳϐΞΛऔಘ
   ‣ BOOPVDF@QFFS JE
   JNQMJFE@QPSU
   JOGP@IBTI
   QPSU
   UPLFO 
   
 ϐΞͷμ΢ϯϩʔυΛΞφ΢ϯε 
7. #JU5PSSFOU
   %)5ͷΫϩʔϥ w ͳͥͭ͘Δ͔
   ‣ #JU5PSSFOUͱ%)5ͷཧղΛਂΊΔ
   ‣ Ϋϩʔϥͷઃܭɾ࣮૷ٕज़Λशಘ͢Δ
   ‣ Ԡ༻
    #JU5PSSFOUωοτϫʔΫͷ؂ࢹɾ෼ੳ
    ྲྀ௨͍ͯ͠Δ5PSSFOU৘ใͷऩू
   ‣ ָ͍͔͠Β
   w ͳʹΛͭ͘Δ͔
   ‣ %)5ʹࢀՃ͍ͯ͠ΔϊʔυΛҶͮΔࣜʹ୳ࡧ͍ͯ͘͠Ϋϩʔϥ
   ‣ ͨͩ࡞Δ͚ͩͰ͸໘ന͘ͳ͍ͷͰɺΫϩʔϦϯά଎౓ΛۃΊΔ 
8. ઌߦݚڀ  1 DHTの迅速で膨大なモニタリング : 24時間以内に 1000万ノードの情報をクローリングする PacSec Tokyo November 2011 In this presentation, we present our high-speed DHT Crawler to monitor 10 millions of nodes in 24 hours ! 情報通信研究機構 ネットエージェント株式会社 25 Rapid crawling: 24 hours to reach 10000000 nodes ! node 0 2000000 4000000 6000000 8000000 10000000 12000000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 hour diff 10000 100000 1000000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 PacSec 2011 ·ͣ͸͜ΕΛ໨ඪʹ
9. "SBDIOF
   ., w ࣌ؒͰສϊʔυΛΫϩʔϧ
   ϫʔΧ
   w ϐʔΫੑೳ͸ϊʔυඵϫʔΧ
   w OPEFKTͰૉ௚ʹ࣮૷
   ಛผʹ࠷దԽͳͲ͸͍ͯ͠ͳ͍
   w ϫʔΧΛ૿΍͢͜ͱͰεέʔϧΞ΢τՄೳ
   w %JHJUBM0DFBO
   714ϓϩόΠμ
   ্ʹߏஙͨ͠ΫϥελͰಈ࡞
   ‣ ϫʔΧ
   ίΞ
   .#
   
   
   7.
   ‣ δϣϒΩϡʔ
   ίΞ.#
   
   7.
   ‣ σʔλϕʔε
   ίΞ
   (#
   
   7.
   w ֤ϫʔΧͰ͸ฒྻͰಈ࡞  ^IPVS IUUQTHJUIVCDPNLFJDIJBSBDIOFNL
10. "SBDIOF
    .,ͷ࢓૊Έ  ϫʔΧ 
    ࣍ʹϦΫΤετΛૹΔ
 ϊʔυΛσΩϡʔ 
    pOE@OPEFϦΫΤετΛૹ৴
    
    ϦϓϥΠ͔ΒϊʔυΛநग़ 
    ൃݟͨ͠ϊʔυΛ
 ΤϯΩϡʔ δϣϒΩϡʔ
    3FEJT 3%#
    .Z42- 
    ॏෳνΣοΫ
    
    ϊʔυ৘ใొ࿥ Πϯλʔωοτ
11. ΋ͬͱ଎͘ͳΔ͸ͣ
12. "SBDIOF
    ., w ෼ͰສϊʔυΛΫϩʔϧ
    ‣ %JHJUBM0DFBOͷίΞ
    (#
    44%ͷ7.୆্Ͱͷੑೳ
    w ϐʔΫੑೳ͸ສϊʔυඵ
    w ݱࡏ
    ͨͿΜ
    ੈք࠷଎
    w ϊʔυͰͷ࣮ߦΛલఏͱͯ͠ઃܭ
    w ৭ʑ࠷దԽΛࢪͯ͋͠Δ  IUUQTHJUIVCDPNLFJDIJBSBDIOFNL ஫
    /"5഑ԼͰ࣮ߦ͢Δͱ
 /"5ςʔϒϧ͕ॠ࣌ʹᷓΕΔ
13. "SBDIOF
    .,
    ࠷దԽͷΞΠσΟΞ w ,31$ϝοηʔδͷૹ৴ͱड৴Λඇಉظʹ࣮ߦ͢Δ
    ‣ ,31$͸6%1্Ͱͷಈ࡞͢ΔɺίωΫγϣϯ͕ෆཁͳϓϩτίϧ
    ‣ ,31$ϦΫΤετΛ౤͔͛ͯΒɺϦεϙϯε͕དྷΔ·Ͱ଴ͭඞཁͳ͠
    w ΫϩʔϦϯάʹඞཁͳػೳҎ֎Λపఈతʹ࡟Δ
    ‣ ,31$ϝοηʔδͷγϦΞϥΠβɾσγϦΞϥΠβͳͲ
    w ΠϯϝϞϦͰಈ࡞
    ‣ σΟεΫ*0ΛߦΘͳ͍
    ‣ %#Λ࢖Θͳ͍ 
14. "SBDIOF
    .,ͷ࢓૊Έ  pOE@OPEF
 ϦΫΤετߏங ૹ৴εϨου ϦΫΤετૹ৴ pOE@OPEF
 ϦϓϥΠͷύʔε ड৴εϨου ϦϓϥΠड৴ ϊʔυΛ
 σΩϡʔ ཚ਺ιʔε 6%1ιέοτ ϊʔυΛ
 ΤϯΩϡʔ ΠϯϝϞϦ%# ϊʔυͷΩϡʔ
15. ฒྻΩϡʔͷ࠷దԽ w Ωϡʔͷ࣮૷
    ‣ ૒ํ޲࿈݁Ϧετ
     ཁૉΛḷΔࡍʹϝϞϦΞΫηε͕ඈͼඈͼʹͳΔ
    ‣ ϦϯάόοϑΝ
     ϝϞϦΞΫηε͕γʔέϯγϟϧʹͳΓΩϟογϡώοτ޲্
     ݻఆ௕ͳͷͰࣄલʹϝϞϦΛ֬อ͓͚ͯ͠Δ
    w ഉଞ੍ޚ
    ‣ ϛϡʔςοΫεͰΩϡʔશମΛϩοΫ
     ϩοΫɾΞϯϩοΫͷίετ͕ߴ͍
    ‣ ΞτϛοΫ໋ྩΛ࢖ͬͨϩοΫϑϦʔͳΩϡʔ 
16. Ϋϩʔϧ݁Ռͷอଘɾ໰͍߹Θͤ w Ϋϩʔϧͨ͠ϊʔυͷ৘ใΛอଘ͠ͳ͚Ε͹͍͚ͳ͍
    w ϥϯμϜϦʔυɾϥΠτͷϨΠςϯγΛ࠷খԽ͍ͨ͠
    ‣ ελϯυΞϩʔϯ%#
     %#ͱͷ௨৴ͷϨΠςϯγ͕େ͖͗͢Δ
    ‣ Τϯϕου,74
     #PMU%#ͱ-FWFM%#Ͱࢼͨ݁͠ՌɺϨΠςϯγ͕େ͖͗͢Δ
    ‣ ΠϯϝϞϦ%#
     ࣮ࡍʹ͸ͨͩͷHPඪ४ͷNBQ
     ࠷଎͕ͩҰ੾ӬଓԽ͠ͳ͍
     Ωϟογϡͱͯ͠࢖͍ɺఆظతʹ42-JUFͳͲʹॻ͘ͷ͕ݱ࣮త 
17. ͦͷଞͷࡉ͔͍࠷దԽ w CFODPEFͷγϦΞϥΠβɾσγϦΞϥΠβ
    ‣ طଘϥΠϒϥϦ͸ϦϑϨΫγϣϯͳͲΛ࢖͍ͬͯΔͨΊ஗͗͢Δ
    ‣ γϦΞϥΠβ͸pOE@OPEFϦΫΤετͷΈʹରԠ
    ‣ σγϦΞϥΠβ͸pOE@OPEFϦΫΤετͷϦεϙϯεͷΈʹରԠ
    w ϝϞϦΞϩέʔγϣϯͷ࡟ݮ
    ‣ ՄೳͳݶΓࣄલʹϝϞϦ֬อͯ͠࢖͍ճ͢
    w ϊʔυʹෳ਺ճpOE@OPEFϦΫΤετΛૹ৴
    ‣ ֤ϊʔυ͔Βϊʔυͷ৘ใΛͰ͖Δ͚ͩଟ͘Ҿ͖ग़͢
    w (PݴޠͰ࣮૷
    ‣ OPEFKT଎͍ͱ͸͍͑ɺػցޠʹίϯύΠϧ͢Δݴޠͷํ͕଎͍ 
18. ·ͱΊ w #JU5PSSFOU
    %)5͸଱ো֐ੑͱεέʔϥϏϦςΟΛ޲্͢ΔͨΊͷ࢓૊Έ
    w %)5ͷΫϩʔϥΛछྨ։ൃ
    ‣ "SBDIOF
    .,
     ฒྻ෼ࢄܕͷΫϩʔϥ
     ࣌ؒͰສϊʔυΛൃݟ
    ‣ "SBDIOF
    .,
     ୯ҰϊʔυͰͷ࣮ߦ͢ΔΫϩʔϥ
     ෼ͰສϊʔυΛൃݟ
    w কདྷ՝୊
    ‣ ͞ΒͳΔ࠷దԽ
    ‣ UPSSFOUϝλ৘ใͷऩू 
19. ࢀߟจݙ w #&1
    ‣ IUUQXXXCJUUPSSFOUPSHCFQTCFQ@IUNM
    w ͳͥͳʹ5PSSFOU
    ‣ IUUQTXXXHJUCPPLDPNCPPLLZPSPIJSPEPD@IFUJNBUPSSFOUEFUBJMT
    w %)5ͷਝ଎Ͱ๲େͳϞχλϦϯά
    ࣌ؒҎ಺ʹ
    ສϊʔυͷ৘ใΛΫ ϩʔϦϯά͢Δ
    ‣ IUUQTQBDTFDKQQTK1BD4FD@.BTTJWF.POJUPSJOH@KQQEG
    w %)5᠓઎ᗡ஬
    ‣ IUUQFOHJZDPNUFDI
    w ,BEFNMJB
    ‣ IUUQXXXTIVEPOFUBSUJDMF,BEFNMJBTIVEP,BEFNMJBQEG