Tweet
Tweet

Slide 1

Slide 1 text

BEHIND the iQON CRAWLER iQONΛࢧ͑ΔΫϩʔϥʔ VASILY,Inc. @kyuns IVS CTO Night & Day 2015 Spring powered by AWS

Slide 2

Slide 2 text

ࠓଜխ޾ @kyuns / Ωϡϯ VASILY,Inc औక໾CTO / Co-Founder 2006೥ʹYahoo!JAPANʹ৽ଔೖࣾ Yahoo!FASHIONɺXBRANDͳͲͷϝσΟΞͷ্ཱͪ͛ 2009೥ʹಠཱɺVASILYΛ૑ۀɺऔక໾CTOʹब೚ ޷͖ͳ"84αʔϏε͸4޷͖ͳ੠༏͸ਫथಸʑ

Slide 3

Slide 3 text

ઃཱ :2008೥11݄ ैۀһ :53ਓ / (ΤϯδχΞ17໊) ࢿຊۚ :8ԯ2,458ສ גओɹ :KDDI ITV GCP GMOVP

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

೔ຊதͷECαΠτͷ ঎඼σʔλΛܝࡌ ೔ຊ࠷େڃͷ ϑΝογϣϯΞϓϦʮΞΠίϯʯ ձһ਺ 230ສਓ

Slide 6

Slide 6 text

J20/ͷ ΫϩʔϥʔͬͯԿʁ

Slide 7

Slide 7 text

Λ௒͑Δ೔ຊதͷϑΝογϣϯ&$αΠτͷσʔλΛ શͯΫϩʔϧɺ஋ஈɺࣸਅɺઆ໌ɺࡏݿΛ௥੻ॲཧ

Slide 8

Slide 8 text

ͱ͋ΔனԼ͕Γʜ

Slide 9

Slide 9 text

ਓؾ501ϒϥϯυ ͍͍ײ͡ʹશ෦Ϋϩʔϧͯ͠Α

Slide 10

Slide 10 text

ཱͪ͸͔ͩΔ՝୊ w ϒϥϯυҎ্Λຖ೔Ϋϩʔϧ௥੻͢Δʹ͸ εέδϡʔϥʔΛޮ཰Խ͠ͳ͍ͱ͍͚ͳ͍ w ΧςΰϦ൑ఆ΍ϞϊɺϞσϧɺτϧιʔͷ൑ఆͳ Ͳɺਓ͕͍ؒͯͨ͜͠ͱΛແ͘͞ͳ͍ͱ͍͚ͳ͍ w ͦ΋ͦ΋91"5)Λਓ͕ؒௐ΂ΔͷΛͳΜͱ͔͠ͳ ͍ͱ͍͚ͳ͍

Slide 11

Slide 11 text

iQON Crawler 4.0 = શࣗಈԽ

Slide 12

Slide 12 text

ΧςΰϦ൑ఆͷࣗಈԽ w ͋ΒΏΔϑΝογϣϯ༻ޠΛ໢ཏͨ͠
 ϑΝογϣϯʹಛԽͨࣙ͠ॻΛੜ੒ w &$αΠτͷλΠτϧɺઆ໌จɺͺΜͣ͘Ϧετ͔ΒεςοϓΛܦͯ൑ఆ
 จ຺ͷ൑ఆͳͲ΋ߦ͍ͬͯΔ
 ྫʮΊͪΌͪ͘Ό͔Θ͍͍όοάʂεΧʔτʹ΋͋͏ʂʯ౳ͷઆ໌จͷ৔߹͸εΧʔ τʹޡ൑ఆ͠ͳ͍Α͏ͳจ຺Λҙࣝͨ͠ॲཧʣ w ͷ঎඼͸্هεςοϓ͔Β൑ఆͰ͖Δ w ϓϧΦʔόʔͱ͔5γϟπʹ΋ϒϥ΢εʹ΋෼ྨͰ͖ͯ͠·͏
 αΠτ͝ͱʹ΋ఆ͕ٛҧͬͨΓ͢Δ w ࢒Γͷෳࡶͳ΋ͷ͸ػցֶशʹ͔͚ͯ൑ఆ

Slide 13

Slide 13 text

ը૾൑ఆͷࣗಈԽ ϞϊɺϞσϧɺτϧιʔΛը૾ղੳΛۦ࢖ͯ͠ɺࣗಈతʹ൑ఆɻ ෼ྨਫ਼౓͸ˋ ͞ΒʹɺϞϊը૾͸ΤσΟλͰ࢖͏ͨΊʹനൈ͖ॲཧΛશࣗಈͰߦ͏

Slide 14

Slide 14 text

91"5)நग़ͷࣗಈԽ &$αΠτͷߏ଄ΛӡӦ ϕϯμʔ΍աڈͷ৘ใΛ ݩʹಛ௃Խ %0.ͷߏ଄͔Β֤ཁૉ Β͖͠΋ͷΛ൑அ ஋ஈɺ λΠτϧ πʔϧͰ91"5)΋؅ཧ

Slide 15

Slide 15 text

ޮ཰ͷ͍͍Ϋϩʔϧͱ͸ w ΫϩʔϧͷXPSLFSΛHPͰॻ͖ͳ͓ͯ͠ EPDLFS&MBTUJD#FBOTUBMLͰBVUPTDBMFͰΫϩʔϧ͠·͘Γʂ
 ઌํͷαΠτ͕͙͢ࢮ͵ w ͷน
 αΠτඵΞΫηε·Ͱͱ͍͏ਈ࢜ڠఆ w ෳ਺XPSLFSΛ·͍ͨͩ෼ࢄϩοΫػߏ͕ඞཁ

Slide 16

Slide 16 text

w %JTUSJCVUFE-PDLXJUI3FEJT %-. IUUQSFEJTJPUPQJDTEJTUMPDL w ಠࣗʹ࣌ݶ෇͖෼ࢄϩοΫSFEJTEJTUNVUFYΛ։ൃ WBTJMZKQSFEJTEJTUNVUFY w ٕज़ϒϩάIUUQUFDIWBTJMZKQDSBXMTJUFTXJUISFEJTEJTUNVUFY w αΠτͷϨεϙϯεੑೳʹؔ܎ͳ͘ҰఆִؒͰޮ཰తʹΫϩʔϧͰ͖Δ Resque (fetch) Fetch worker &$4JUF Dist Mutex Resque (process) Fetch worker Fetch worker Sender 1.enqueue 2.dequeue 3.try_lock 4.get DB 5.save 6.enqueue

Slide 17

Slide 17 text

·ͱΊ w $&0ͷແ஡ͿΓʹ͸ٕज़ͰԠ͑Α͏ w Ϋϩʔϥʔʹ͍ͭͯฉ͖͍ͨਓ͓଴ͪͯ͠·͢