高速なBitTorrent DHTクローラ

ߴ଎ͳ#JU5PSSFOU%)5Ϋϩʔϥ ೥݄೔ શମྠߨϑϦʔτʔΫ Լᑍݚ%ߴڮܛஐ

#JU5PSSFOU w ੈք࠷େͷ11ϑΝΠϧڞ༗ιϑτ΢ΣΞ ‣ ৗ࣌ສਓʙສਓͷΦϯϥΠϯϢʔβ ೥ ‣ ԯສਓͷΞΫςΟϒϢʔβ ೥
w ༷ʑͳԠ༻ྫ ‣ #JU5PSSFOU4ZOD 11൛%SPQCPY ‣ ΦϯϥΠϯήʔϜͷΞοϓσʔτͷ഑৴ $"1$0. #MJ[[BSE ʜ ‣ Ϋϥελʹ͓͚ΔΞϓϦέʔγϣϯͷσϓϩΠπʔϧɾ  ڞ༗ϑΝΠϧγεςϜͱͯ͠

#JU5PSSFOUͷ࢓૊Έ %)5ແ 5PSSFOUϑΝΠϧ ݕࡧΫΤϦ ϐΞऔಘϦΫΤετ ϐΞ৘ใ ϐʔεϦΫΤετ ϐʔε 4FFE1FFS
5SBDLFS *OEFYJOH8FCTJUF

%JTUSJCVUFE)BTI5BCMF %)5 w )BTI5BCMF ‣ Ωʔ͔Βޮ཰తʹ஋ΛݕࡧͰ͖Δσʔλߏ଄ w %JTUSJCVUFE)BTI5BCMF ‣ σʔλΛෳ਺ͷίϯϐϡʔλ্ʹ෼ࢄ͢Δ)BTI5BCMF
w ैདྷͷ#JU5PSSFOUͷऑ఺ ‣ 5SBDLFS͕ఀࢭ͢ΔͱɺϐΞͷ৘ใͷ୳ࡧ͕Ͱ͖ͳ͘ͳΓɺ  μ΢ϯϩʔυɾΞοϓϩʔυෆೳʹͳΔ ‣ ϐΞ਺͕૿Ճ͢Δͱ5SBDLFS͕ϘτϧωοΫʹͳΓ͏Δ w #JU5PSSFOU.BJOMJOF%)5 ‣ ϐΞͷ৘ใΛ%)5ʹอଘ͢Δ͜ͱͰɺ଱ো֐ੑͱεέʔϥϏϦςΟ Λ޲্

#JU5PSSFOUͷ࢓૊Έ %)5༗ 4FFE1FFS%)5/PEF %)5ʹࢀՃ ϐΞ৘ใऔಘ ϐʔεऔಘ

#JU5PSSFOU.BJOMJOF%)5ͷ࢓૊Έ w ,BEFNMJBΛϕʔεʹͨ͠%)5 ‣ CJUͷΞυϨεۭؒ ‣ 903ܭྔͱ෼໦ͰΞυϨεۭؒΛߏ଄Խ w %)5ϊʔυؒ͸,31$ͱ͍͏ϓϩτίϧͰ௨৴ ‣
6%1্ͷඇಉظతͳϓϩτίϧ w QJOH pOE@OPEF HFU@QFFS BOOOPVODF@QFFSͷछͷϝοηʔδ ‣ QJOH JE ϊʔυͷੜࢮΛ֬ೝ ‣ pOE@OPEF JE UBSHFU UBSHFUʹۙͦ͏ͳϊʔυΛऔಘ ‣ HFU@QFFS JE JOGPIBTI JOGPIBTIΛ͍࣋ͬͯͦ͏ͳϐΞΛऔಘ ‣ BOOPVDF@QFFS JE JNQMJFE@QPSU JOGP@IBTI QPSU UPLFO   ϐΞͷμ΢ϯϩʔυΛΞφ΢ϯε

#JU5PSSFOU%)5ͷΫϩʔϥ w ͳͥͭ͘Δ͔ ‣ #JU5PSSFOUͱ%)5ͷཧղΛਂΊΔ ‣ Ϋϩʔϥͷઃܭɾ࣮૷ٕज़Λशಘ͢Δ ‣ Ԡ༻
#JU5PSSFOUωοτϫʔΫͷ؂ࢹɾ෼ੳ ྲྀ௨͍ͯ͠Δ5PSSFOU৘ใͷऩू ‣ ָ͍͔͠Β w ͳʹΛͭ͘Δ͔ ‣ %)5ʹࢀՃ͍ͯ͠ΔϊʔυΛҶͮΔࣜʹ୳ࡧ͍ͯ͘͠Ϋϩʔϥ ‣ ͨͩ࡞Δ͚ͩͰ͸໘ന͘ͳ͍ͷͰɺΫϩʔϦϯά଎౓ΛۃΊΔ

ઌߦݚڀ 1 DHTの迅速で膨大なモニタリング : 24時間以内に 1000万ノードの情報をクローリングする PacSec Tokyo November
2011 In this presentation, we present our high-speed DHT Crawler to monitor 10 millions of nodes in 24 hours ! 情報通信研究機構ネットエージェント株式会社 25 Rapid crawling: 24 hours to reach 10000000 nodes ! node 0 2000000 4000000 6000000 8000000 10000000 12000000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 hour diff 10000 100000 1000000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 PacSec 2011 ·ͣ͸͜ΕΛ໨ඪʹ

"SBDIOF., w ࣌ؒͰສϊʔυΛΫϩʔϧ ϫʔΧ w ϐʔΫੑೳ͸ϊʔυඵϫʔΧ w OPEFKTͰૉ௚ʹ࣮૷ ಛผʹ࠷దԽͳͲ͸͍ͯ͠ͳ͍
w ϫʔΧΛ૿΍͢͜ͱͰεέʔϧΞ΢τՄೳ w %JHJUBM0DFBO 714ϓϩόΠμ ্ʹߏஙͨ͠ΫϥελͰಈ࡞ ‣ ϫʔΧ ίΞ.# 7. ‣ δϣϒΩϡʔ ίΞ.# 7. ‣ σʔλϕʔε ίΞ(# 7. w ֤ϫʔΧͰ͸ฒྻͰಈ࡞ ^IPVS IUUQTHJUIVCDPNLFJDIJBSBDIOFNL

"SBDIOF.,ͷ࢓૊Έ ϫʔΧ ࣍ʹϦΫΤετΛૹΔ  ϊʔυΛσΩϡʔ pOE@OPEFϦΫΤετΛૹ৴ ϦϓϥΠ͔ΒϊʔυΛநग़ ൃݟͨ͠ϊʔυΛ  ΤϯΩϡʔ δϣϒΩϡʔ
3FEJT 3%# .Z42- ॏෳνΣοΫ ϊʔυ৘ใొ࿥ Πϯλʔωοτ

΋ͬͱ଎͘ͳΔ͸ͣ

"SBDIOF., w ෼ͰສϊʔυΛΫϩʔϧ ‣ %JHJUBM0DFBOͷίΞ(#44%ͷ7.୆্Ͱͷੑೳ w ϐʔΫੑೳ͸ສϊʔυඵ w ݱࡏ ͨͿΜ
ੈք࠷଎ w ϊʔυͰͷ࣮ߦΛલఏͱͯ͠ઃܭ w ৭ʑ࠷దԽΛࢪͯ͋͠Δ IUUQTHJUIVCDPNLFJDIJBSBDIOFNL ஫/"5഑ԼͰ࣮ߦ͢Δͱ  /"5ςʔϒϧ͕ॠ࣌ʹᷓΕΔ

"SBDIOF.,࠷దԽͷΞΠσΟΞ w ,31$ϝοηʔδͷૹ৴ͱड৴Λඇಉظʹ࣮ߦ͢Δ ‣ ,31$͸6%1্Ͱͷಈ࡞͢ΔɺίωΫγϣϯ͕ෆཁͳϓϩτίϧ ‣ ,31$ϦΫΤετΛ౤͔͛ͯΒɺϦεϙϯε͕དྷΔ·Ͱ଴ͭඞཁͳ͠ w ΫϩʔϦϯάʹඞཁͳػೳҎ֎Λపఈతʹ࡟Δ ‣
,31$ϝοηʔδͷγϦΞϥΠβɾσγϦΞϥΠβͳͲ w ΠϯϝϞϦͰಈ࡞ ‣ σΟεΫ*0ΛߦΘͳ͍ ‣ %#Λ࢖Θͳ͍

"SBDIOF.,ͷ࢓૊Έ pOE@OPEF  ϦΫΤετߏங ૹ৴εϨου ϦΫΤετૹ৴ pOE@OPEF  ϦϓϥΠͷύʔε ड৴εϨου ϦϓϥΠड৴
ϊʔυΛ  σΩϡʔ ཚ਺ιʔε 6%1ιέοτ ϊʔυΛ  ΤϯΩϡʔ ΠϯϝϞϦ%# ϊʔυͷΩϡʔ

ฒྻΩϡʔͷ࠷దԽ w Ωϡʔͷ࣮૷ ‣ ૒ํ޲࿈݁Ϧετ ཁૉΛḷΔࡍʹϝϞϦΞΫηε͕ඈͼඈͼʹͳΔ ‣ ϦϯάόοϑΝ
ϝϞϦΞΫηε͕γʔέϯγϟϧʹͳΓΩϟογϡώοτ޲্ ݻఆ௕ͳͷͰࣄલʹϝϞϦΛ֬อ͓͚ͯ͠Δ w ഉଞ੍ޚ ‣ ϛϡʔςοΫεͰΩϡʔશମΛϩοΫ ϩοΫɾΞϯϩοΫͷίετ͕ߴ͍ ‣ ΞτϛοΫ໋ྩΛ࢖ͬͨϩοΫϑϦʔͳΩϡʔ

Ϋϩʔϧ݁Ռͷอଘɾ໰͍߹Θͤ w Ϋϩʔϧͨ͠ϊʔυͷ৘ใΛอଘ͠ͳ͚Ε͹͍͚ͳ͍ w ϥϯμϜϦʔυɾϥΠτͷϨΠςϯγΛ࠷খԽ͍ͨ͠ ‣ ελϯυΞϩʔϯ%# %#ͱͷ௨৴ͷϨΠςϯγ͕େ͖͗͢Δ ‣
Τϯϕου,74 #PMU%#ͱ-FWFM%#Ͱࢼͨ݁͠ՌɺϨΠςϯγ͕େ͖͗͢Δ ‣ ΠϯϝϞϦ%# ࣮ࡍʹ͸ͨͩͷHPඪ४ͷNBQ ࠷଎͕ͩҰ੾ӬଓԽ͠ͳ͍ Ωϟογϡͱͯ͠࢖͍ɺఆظతʹ42-JUFͳͲʹॻ͘ͷ͕ݱ࣮త

ͦͷଞͷࡉ͔͍࠷దԽ w CFODPEFͷγϦΞϥΠβɾσγϦΞϥΠβ ‣ طଘϥΠϒϥϦ͸ϦϑϨΫγϣϯͳͲΛ࢖͍ͬͯΔͨΊ஗͗͢Δ ‣ γϦΞϥΠβ͸pOE@OPEFϦΫΤετͷΈʹରԠ ‣ σγϦΞϥΠβ͸pOE@OPEFϦΫΤετͷϦεϙϯεͷΈʹରԠ w
ϝϞϦΞϩέʔγϣϯͷ࡟ݮ ‣ ՄೳͳݶΓࣄલʹϝϞϦ֬อͯ͠࢖͍ճ͢ w ϊʔυʹෳ਺ճpOE@OPEFϦΫΤετΛૹ৴ ‣ ֤ϊʔυ͔Βϊʔυͷ৘ใΛͰ͖Δ͚ͩଟ͘Ҿ͖ग़͢ w (PݴޠͰ࣮૷ ‣ OPEFKT଎͍ͱ͸͍͑ɺػցޠʹίϯύΠϧ͢Δݴޠͷํ͕଎͍

·ͱΊ w #JU5PSSFOU%)5͸଱ো֐ੑͱεέʔϥϏϦςΟΛ޲্͢ΔͨΊͷ࢓૊Έ w %)5ͷΫϩʔϥΛछྨ։ൃ ‣ "SBDIOF., ฒྻ෼ࢄܕͷΫϩʔϥ
࣌ؒͰສϊʔυΛൃݟ ‣ "SBDIOF., ୯ҰϊʔυͰͷ࣮ߦ͢ΔΫϩʔϥ ෼ͰສϊʔυΛൃݟ w কདྷ՝୊ ‣ ͞ΒͳΔ࠷దԽ ‣ UPSSFOUϝλ৘ใͷऩू

ࢀߟจݙ w #&1 ‣ IUUQXXXCJUUPSSFOUPSHCFQTCFQ@IUNM w ͳͥͳʹ5PSSFOU ‣ IUUQTXXXHJUCPPLDPNCPPLLZPSPIJSPEPD@IFUJNBUPSSFOUEFUBJMT w
%)5ͷਝ଎Ͱ๲େͳϞχλϦϯά࣌ؒҎ಺ʹສϊʔυͷ৘ใΛΫ ϩʔϦϯά͢Δ ‣ IUUQTQBDTFDKQQTK1BD4FD@.BTTJWF.POJUPSJOH@KQQEG w %)5᠓઎ᗡ஬ ‣ IUUQFOHJZDPNUFDI w ,BEFNMJB ‣ IUUQXXXTIVEPOFUBSUJDMF,BEFNMJBTIVEP,BEFNMJBQEG

高速なBitTorrent DHTクローラ

高速なBitTorrent DHTクローラ

Keichi Takahashi

More Decks by Keichi Takahashi

Other Decks in Programming

Featured

Transcript

ߴ଎ͳ#JU5PSSFOU%)5Ϋϩʔϥ ೥݄೔ શମྠߨϑϦʔτʔΫ Լᑍݚ%ߴڮܛஐ

#JU5PSSFOU w ੈք࠷େͷ11ϑΝΠϧڞ༗ιϑτ΢ΣΞ ‣ ৗ࣌ສਓʙສਓͷΦϯϥΠϯϢʔβ ೥ ‣ ԯສਓͷΞΫςΟϒϢʔβ ೥

#JU5PSSFOUͷ࢓૊Έ %)5ແ 5PSSFOUϑΝΠϧ ݕࡧΫΤϦ ϐΞऔಘϦΫΤετ ϐΞ৘ใ ϐʔεϦΫΤετ ϐʔε 4FFE1FFS

%JTUSJCVUFE)BTI5BCMF %)5 w )BTI5BCMF ‣ Ωʔ͔Βޮ཰తʹ஋ΛݕࡧͰ͖Δσʔλߏ଄ w %JTUSJCVUFE)BTI5BCMF ‣ σʔλΛෳ਺ͷίϯϐϡʔλ্ʹ෼ࢄ͢Δ)BTI5BCMF

#JU5PSSFOUͷ࢓૊Έ %)5༗ 4FFE1FFS%)5/PEF %)5ʹࢀՃ ϐΞ৘ใऔಘ ϐʔεऔಘ

#JU5PSSFOU.BJOMJOF%)5ͷ࢓૊Έ w ,BEFNMJBΛϕʔεʹͨ͠%)5 ‣ CJUͷΞυϨεۭؒ ‣ 903ܭྔͱ෼໦ͰΞυϨεۭؒΛߏ଄Խ w %)5ϊʔυؒ͸,31$ͱ͍͏ϓϩτίϧͰ௨৴ ‣

#JU5PSSFOU%)5ͷΫϩʔϥ w ͳͥͭ͘Δ͔ ‣ #JU5PSSFOUͱ%)5ͷཧղΛਂΊΔ ‣ Ϋϩʔϥͷઃܭɾ࣮૷ٕज़Λशಘ͢Δ ‣ Ԡ༻

ઌߦݚڀ 1 DHTの迅速で膨大なモニタリング : 24時間以内に 1000万ノードの情報をクローリングする PacSec Tokyo November

"SBDIOF., w ࣌ؒͰສϊʔυΛΫϩʔϧ ϫʔΧ w ϐʔΫੑೳ͸ϊʔυඵϫʔΧ w OPEFKTͰૉ௚ʹ࣮૷ ಛผʹ࠷దԽͳͲ͸͍ͯ͠ͳ͍

"SBDIOF.,ͷ࢓૊Έ ϫʔΧ ࣍ʹϦΫΤετΛૹΔ  ϊʔυΛσΩϡʔ pOE@OPEFϦΫΤετΛૹ৴ ϦϓϥΠ͔ΒϊʔυΛநग़ ൃݟͨ͠ϊʔυΛ  ΤϯΩϡʔ δϣϒΩϡʔ

΋ͬͱ଎͘ͳΔ͸ͣ

"SBDIOF., w ෼ͰສϊʔυΛΫϩʔϧ ‣ %JHJUBM0DFBOͷίΞ(#44%ͷ7.୆্Ͱͷੑೳ w ϐʔΫੑೳ͸ສϊʔυඵ w ݱࡏ ͨͿΜ

"SBDIOF.,࠷దԽͷΞΠσΟΞ w ,31$ϝοηʔδͷૹ৴ͱड৴Λඇಉظʹ࣮ߦ͢Δ ‣ ,31$͸6%1্Ͱͷಈ࡞͢ΔɺίωΫγϣϯ͕ෆཁͳϓϩτίϧ ‣ ,31$ϦΫΤετΛ౤͔͛ͯΒɺϦεϙϯε͕དྷΔ·Ͱ଴ͭඞཁͳ͠ w ΫϩʔϦϯάʹඞཁͳػೳҎ֎Λపఈతʹ࡟Δ ‣

"SBDIOF.,ͷ࢓૊Έ pOE@OPEF  ϦΫΤετߏங ૹ৴εϨου ϦΫΤετૹ৴ pOE@OPEF  ϦϓϥΠͷύʔε ड৴εϨου ϦϓϥΠड৴

ฒྻΩϡʔͷ࠷దԽ w Ωϡʔͷ࣮૷ ‣ ૒ํ޲࿈݁Ϧετ ཁૉΛḷΔࡍʹϝϞϦΞΫηε͕ඈͼඈͼʹͳΔ ‣ ϦϯάόοϑΝ

Ϋϩʔϧ݁Ռͷอଘɾ໰͍߹Θͤ w Ϋϩʔϧͨ͠ϊʔυͷ৘ใΛอଘ͠ͳ͚Ε͹͍͚ͳ͍ w ϥϯμϜϦʔυɾϥΠτͷϨΠςϯγΛ࠷খԽ͍ͨ͠ ‣ ελϯυΞϩʔϯ%# %#ͱͷ௨৴ͷϨΠςϯγ͕େ͖͗͢Δ ‣

ͦͷଞͷࡉ͔͍࠷దԽ w CFODPEFͷγϦΞϥΠβɾσγϦΞϥΠβ ‣ طଘϥΠϒϥϦ͸ϦϑϨΫγϣϯͳͲΛ࢖͍ͬͯΔͨΊ஗͗͢Δ ‣ γϦΞϥΠβ͸pOE@OPEFϦΫΤετͷΈʹରԠ ‣ σγϦΞϥΠβ͸pOE@OPEFϦΫΤετͷϦεϙϯεͷΈʹରԠ w

·ͱΊ w #JU5PSSFOU%)5͸଱ো֐ੑͱεέʔϥϏϦςΟΛ޲্͢ΔͨΊͷ࢓૊Έ w %)5ͷΫϩʔϥΛछྨ։ൃ ‣ "SBDIOF., ฒྻ෼ࢄܕͷΫϩʔϥ

ࢀߟจݙ w #&1 ‣ IUUQXXXCJUUPSSFOUPSHCFQTCFQ@IUNM w ͳͥͳʹ5PSSFOU ‣ IUUQTXXXHJUCPPLDPNCPPLLZPSPIJSPEPD@IFUJNBUPSSFOUEFUBJMT w