jdk-1812-mugt-session-latter-half-iqon-crawler-on-mesos

Japan Container Days v12.18 - DAY 2 [ 2W4 ]
Hashtags : #containerdaysjp #HallWEST ( #MUGT )

Mesos User Group Tokyo Community Session
IQON Crawler on Mesos

Transcript

1. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   Japan Container Days v12.18 - DAY 2 [ 2W4 ]
   Hashtags : #containerdaysjp #HallWEST ( #MUGT )
   ZOZO Technologies, Inc.
   ։ൃ෦ SRE ςοΫϦʔυ
   ޫ໺ ୡ࿕
   *20/Ϋϩʔϥʔج൫
   ʲ.FTPTϢʔβࣄྫʳ
   

   View Slide

2. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   ϓϩϑΟʔϧ
   ϠϑʔגࣜձࣾʹͯϠϑΦΫ!ɾ޿ࠂͷ։ൃʹܞΘΔɻ2016೥͔
   Β͸גࣜձࣾVASILYɺ๏ਓ౷߹ʹ൐͍2018೥4݄͔Βݱ৬ɻݱ
   ࡏ͸SREΤϯδχΞͱͯ͠ɺ೔ʑαʔϏεͷ৴པੑ޲্ʹ౒Ί
   ΔɻVASILY࣌୅ͷऔΓ૊Έ͔Βԑ͋ͬͯMesos User Group
   TokyoʹࢀՃɻຊ൪؀ڥͰͷ࣮ફωλ୲౰ɻ΢ΟεΩʔͱ༿ר͕
   ޷͖ɻѪͯ͠Δɻ
   ;0;05FDIOPMPHJFT *OD
   ։ൃ෦43&ςοΫϦʔυ
   ޫ໺ୡ࿕
   Icon illustrated by YOSHI 

   https://ja-jp.facebook.com/yoshi.yone.7
   

   View Slide

3. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   ;0;05FDIOPMPHJFT *OD
   ZOZOグループのサービス運用・技術開発部門
   

   View Slide

4. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   ˓೔ຊ࠷େڃͷϑΝογϣϯγϣοϐϯάαΠτΞϓϦ
   ˓ Ҏ্ͷγϣοϓɺ Ҏ্ͷϒϥϯυͷऔΓѻ͍ʢ
   ೥݄຤࣌఺ʣ
   ˓ৗ࣌ສ఺Ҏ্ͷ঎඼ΞΠςϜ਺ͱຖ೔ฏۉ ఺Ҏ্ͷ৽ண঎
   ඼Λܝࡌ
   ˓ଈ೔഑ૹαʔϏεΪϑτϥοϐϯάαʔϏεπέ෷͍ͳͲ
   IUUQ[P[PKQ
   

   View Slide

5. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   ˓೔ຊ࠷େڃͷϑΝογϣϯίʔσΟωʔτΞϓϦ
   ˓ ສμ΢ϯϩʔυಥഁɺίʔσΟωʔτ౤ߘ૯਺͸ສ݅Ҏ
   ্ʢͱ΋ʹ೥݄຤࣌఺ʣ
   ˓શੈքʢ"QQ4UPSF(PPHMFQMBZ͕ར༻ՄೳͳશͯͷࠃʣͰμ
   ΢ϯϩʔυ͕Մೳ
   ˓ສਓҎ্ͷϑΥϩϫʔΛ࣋ͭϢʔβʔʢ8&"3*45"ʣ΋஀ੜ
   IUUQTXFBSKQ
   

   View Slide

6. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   ˓౰͕ࣾಠࣗʹ։ൃͨ͠࠾ੇ༻ϘσΟʔεʔπ
   ˓શମʹࢪ͞ΕͨυοτϚʔΧʔΛεϚʔτϑΥϯΧϝϥͰ౓ࡱ
   Ө͢Δ͜ͱͰɺମܕσʔλΛܭଌ
   ˓ܭଌͨ͠ମܕσʔλ͸ɺॠ࣌ʹ%ϞσϧԽ͞Εɺ;0;0508/Ξ
   ϓϦʹอଘɻ%Ϟσϧ͸͋ΒΏΔ֯౓ʹಈ͔͢͜ͱ͕Ͱ͖ɺମܕΛ
   ౓νΣοΫ͢Δ͜ͱ͕Մೳ
   IUUQ[P[PKQ[P[PTVJU
   

   View Slide

7. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   ˓ʮ;0;046*5ʯͰܭଌͨ͠ମܕσʔλΛ΋ͱʹɺҰਓͻͱΓͷମ
   ܕʹ߹ͬͨʮ͋ͳͨαΠζʯͷΞΠςϜ
   ˓ʮڀۃͷϑΟοτײʯΛ࣮ݱͨ͠ϕʔγοΫΞΠςϜΛఏڙ
   άϩʔόϧαΠτʮ;0;0DPNʯͰւ֎ల։
   ˓ΞΠςϜ̩γϟπɺσχϜύϯπɺγϟπɺϏδωεεʔπɺ
   ωΫλΠɺϘʔμʔ̩γϟπɺ௕କΫϧʔωοΫ̩γϟπͳͲ
   IUUQ[P[PKQQC
   

   View Slide

8. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   

   View Slide

9. Ҏ্ͷϑΝογϣϯ&$αΠτ͔Βͷ΂ສ఺Λ௒͑Δ঎඼Λܝࡌ
   ݄ؒສਓҎ্͕ར༻͢Δ೔ຊ࠷େڃͷϑΝογϣϯαΠτ
   

   View Slide

10. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ✖
    

    View Slide

11. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    *20/ͷதͰಈ͍͍ͯΔ.FTPT
    ΫϩʔϥʔγεςϜͷج൫ͱͯ͠
    

    View Slide

12. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    *20/ͷϏδωε
    EC sites
    
    1.
    ঎඼৘ใ
    ू໿
    3. ߪೖՄೳͳECαΠτ΁ૹ٬
    4. ܦ༝ച্ͷҰ෦
    2. Ӿཡ
    

    View Slide

13. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    *20/ͷϏδωε
    EC sites
    
    ঎඼৘ใ
    ू໿
    ॏཁ
    

    View Slide

14. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    *20/ͷϏδωε
    EC sites
    
    ॏཁ
    IQON
    Ϋϩʔϥ
    

    View Slide

15. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    εέʔϧײʢ೥౰࣌ʣ
    online-shot: Icons made by Vectors Market from www.flaticon.com is licensed by CC 3.0 BY
    dress: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY
    more than
    300 EC sites
    per day
    more than
    1,000,000 fashion items
    per day
    

    View Slide

16. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Application
    Infrastructure
    RubyͰ࣮૷
    ෼ࢄฒྻ
    γεςϜԽ
    EC2
    Ϋϩʔϥʔͷਐา
    ߋͳΔଟஈԽ
    XPSLFS
    #
    XPSLFS
    "
    &$*OTUBODF
    XPSLFS
    "
    XPSLFS
    $
    XPSLFS
    %
    XPSLFS
    "
    

    View Slide

17. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Application
    Infrastructure
    Ϋϩʔϥʔͷਐา
    ߋͳΔଟஈԽ
    XPSLFS
    #
    XPSLFS
    "
    &$*OTUBODF
    XPSLFS
    "
    XPSLFS
    $
    XPSLFS
    %
    XPSLFS
    "
    ࣍ͳΔεςʔδ
    ʢ2016೥ʣ
    EC2
    

    View Slide

18. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    XPSLFS
    #
    XPSLFS
    "
    XPSLFS
    $
    XPSLFS
    %
    XPSLFS
    "
    XPSLFS
    "
    XPSLFS
    #
    XPSLFS
    "
    XPSLFS
    $
    XPSLFS
    %
    XPSLFS
    %
    XPSLFS
    %
    XPSLFS
    #
    XPSLFS
    $
    XPSLFS
    %
    XPSLFS
    %
    TIME
    Ϋϩʔϥʔͷཧ૝
    

    View Slide

19. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    XPSLFS
    #
    XPSLFS
    "
    XPSLFS
    $
    XPSLFS
    %
    XPSLFS
    "
    XPSLFS
    "
    XPSLFS
    #
    XPSLFS
    "
    XPSLFS
    $
    XPSLFS
    %
    XPSLFS
    %
    XPSLFS
    %
    XPSLFS
    #
    XPSLFS
    $
    XPSLFS
    %
    XPSLFS
    %
    TIME
    ՝୊
    • ϫʔΧʔຖʹCPU, ϝϞϦͱ͍ͬͨඞཁͳϦιʔε͕ҟͳΔ
    • ͦͷ࣌ʑͰݸʑͷΠϯελϯεϦιʔεΛߟྀ͢Δඞཁ͕͋Δ

    ʢͨ͘͠ͳ͍ʣ
    • ϫʔΧʔͷϓϩηεΛίϯτϩʔϧϓϨʔϯ͔Βૢ࡞Մೳʹ͢Δඞཁ͕͋Δ

    ʢͨ͘͠ͳ͍ʣ
    

    View Slide

20. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    XPSLFS
    #
    XPSLFS
    "
    XPSLFS
    $
    XPSLFS
    %
    XPSLFS
    "
    XPSLFS
    "
    XPSLFS
    #
    XPSLFS
    "
    XPSLFS
    $
    XPSLFS
    %
    XPSLFS
    %
    XPSLFS
    %
    XPSLFS
    #
    XPSLFS
    $
    XPSLFS
    %
    XPSLFS
    %
    TIME
    Ϋϩʔϥʔͷཧ૝
    

    View Slide

21. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ىಈɾഇغ͕
    ༰қͳശ
    ىಈɾഇغ͕
    ༰қͳശ
    ىಈɾഇغ͕
    ༰қͳശ
    ىಈɾഇغ͕
    ༰қͳശ
    $PNQVUFS $PNQVUFS
    Կ͔ૉఢͳந৅Խ૚
    XPSLFS
    "
    XPSLFS
    #
    $PNQVUFS
    XPSLFS
    $
    XPSLFS
    %
    $PNQVUFS $PNQVUFS $PNQVUFS
    

    View Slide

22. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ىಈɾഇغ͕
    ༰қͳശ
    ىಈɾഇغ͕
    ༰қͳശ
    ىಈɾഇغ͕
    ༰қͳശ
    ىಈɾഇغ͕
    ༰қͳശ
    $PNQVUFS $PNQVUFS
    ίϯςφΦʔέετϨʔγϣϯ૚
    XPSLFS
    "
    XPSLFS
    #
    $PNQVUFS
    XPSLFS
    $
    XPSLFS
    %
    $PNQVUFS $PNQVUFS $PNQVUFS
    

    View Slide

23. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    

    View Slide

24. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ΫϩʔϥʔPO.FTPT
    ܭࢉࢿݯ
    ܭࢉࢿݯͷ؅ཧɾ
    ू໿ɾந৅Խ
    ίϯςφͷঢ়گ؂ࢹ
    ΞϓϦέʔγϣϯ
    

    View Slide

25. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ΫϩʔϥʔPO.FTPT
    ෼ࢄγεςϜ༻
    λεΫQueue
    Queue
    ؂ࢹ
    Queue؂ࢹ
    APIʹΑΔίϯςφૢ࡞
    

    View Slide

26. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ΫϩʔϥʔPO.FTPT
    ϋʔυ΢ΣΞো֐Ͱ
    λεΫ͕ݮগͨ͠৔߹
    

    View Slide

27. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ΫϩʔϥʔPO.FTPT
    Marathon͕λεΫ਺Λҡ࣋
    Mesos͕λεΫΛ࠶෼഑͢Δ
    

    View Slide

28. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    racing: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY
    crawler operating time
    67% down
    

    View Slide

29. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ͳͥ.FTPTʁ
    shakehand, home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY
    open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY
    

    View Slide

30. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ϙΠϯτͦͷɿ୳͍ͯͨ͠΋ͷͱࢥ૝͕ϕετϚον
    shakehand, home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY
    open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY
    • Mesosͷࢥ૝ɹɿ෼ࢄ؀ڥΛڊେͳܭࢉࢿݯͱͯ͠Έͳ͢
    • ୳͍ͯͨ͠΋ͷ ɿϫʔΧʔΛ౎߹Α͘഑ஔ͢ΔͨΊͷܭࢉࢿݯ

    • ϫʔΧʔͷಛੑɿ෼ࢄɾඇಉظॲཧ

    ʹʼ CPU΍ϝϞϦͱ͍ͬͨܭࢉࢿݯʹͷΈґଘ
    • Mesosͷࢥ૝͸γϯϓϧͰɺ͔ͭϕετϚονͩͬͨ
    

    View Slide

31. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ϙΠϯτͦͷɿγϯϓϧͳΫϥελ
    shakehand, home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY
    open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY
    Mesos Master
    Zookeeper
    Mesos Slave
    Framework
    (Optional)
    • ࠷খ3ཁૉ
    • ϑϨʔϜϫʔΫ͸޷͖ͳ͚ͩ
    • LinuxσΟετϦʹΑͬͯ͸

    apt/yumͰΠϯετʔϧՄೳ
    

    View Slide

32. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ϙΠϯτͦͷɿ๛෋ͳυΩϡϝϯτ
    shakehand, home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY
    open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY
    • ৄࡉͳެࣜυΩϡϝϯτ
    • ΤʔδΣϯτͷύϥϝʔλ
    • Ξυόϯευͳ࢖͍ํͱ۩ମతͳྫ
    • APIͷ࢓༷
    • ʮެࣜΛ୳ͤ͹ྑ͍ʯ

    ͱ͍͏҆৺ײ
    

    View Slide

33. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ͳͥ.FTPTʁ
    shakehand, home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY
    open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY
    

    View Slide

34. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ͱ͜ΖͰ%$04ʁ
    ܭࢉࢿݯ
    Mesos
    Marathon
    ʢDC/OS͕ࣗ਎Λ؅ཧ͢ΔͨΊʣ
    DC/OS
    Marathon
    ʢΞϓϦέʔγϣϯ༻ʣ
    ΞϓϦέʔγϣϯ
    }1.ϑϨʔϜϫʔΫͷ૚͕ް͘ͳΔ
    2.DC/OSͷར఺Ͱ͋Δͦͷଞͷ෼ࢄγεςϜΛಈ͔͢༧ఆ͕ͳ͍
    3.DC/OS͕࣋ͭϩʔυόϥϯαɾDNS͕ෆཁ
    γϯϓϧʹѻ͏ͨΊɺDC/OS͸ෆ࠾༻
    

    View Slide

35. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ·ͱΊ
    •IQONΫϩʔϥʔج൫͸Mesos͕࢖ΘΕ͍ͯΔ
    • MesosʹΑͬͯϫʔΧʔͷ෼഑͕ޮ཰తʹߦ͑Δ༷ʹ
    • ෼ࢄॲཧͰ͖ΔΞϓϦέʔγϣϯͰ͋Ε͹Mesos͕ΑΓྑ͍બ୒ࢶͱͳΔ
    •Ϋϩʔϧͷऴ͕ྃ࣌ؒ໿1/3ʹ୹ॖ
    •τϥϒϧ࣌ͷλεΫ࠶഑ஔ͕ࣗಈͰߦΘΕΔͨΊγεςϜͷՄ༻ੑ޲্
    • ͜ΕΛར༻ͯ͠AWS SpotInstanceͳͲΛ૊Έ߹ΘͤΔͱίετ΋࡟ݮՄೳ
    •ඞͣ͠΋DC/OS͸ඞཁͰͳ͍ͷͰɺέʔεόΠέʔεͰબ୒ ✖
    

    View Slide

36. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ΋ͬͱ஌Γ͍ͨ
    Docker / Apache Mesos / Marathon ʹΑΔ3ഒ଎͍IQONΫϩʔϥʔͷߏங

    https://techblog.zozo.com/entry/iqon-crawler-by-docker-and-mesos-and-marathon

    Apache Mesos with Amazon EC2 SpotFleet
    https://speakerdeck.com/kotatsu360/apache-mesos-with-amazon-ec2-spotfleet
    

    View Slide

37. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    5IBOLZPV
    

    View Slide