jdk-1812-mugt-session-latter-half-iqon-crawler-on-mesos

 jdk-1812-mugt-session-latter-half-iqon-crawler-on-mesos

Japan Container Days v12.18 - DAY 2 [ 2W4 ]
Hashtags : #containerdaysjp #HallWEST ( #MUGT )

Mesos User Group Tokyo Community Session
IQON Crawler on Mesos

231dbe4867c139c325b61e808b757750?s=128

Tatsuro Mitsuno

December 05, 2018
Tweet

Transcript

  1. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Japan Container

    Days v12.18 - DAY 2 [ 2W4 ] Hashtags : #containerdaysjp #HallWEST ( #MUGT ) ZOZO Technologies, Inc. ։ൃ෦ SRE ςοΫϦʔυ ޫ໺ ୡ࿕ *20/Ϋϩʔϥʔج൫ ʲ.FTPTϢʔβࣄྫʳ
  2. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ϓϩϑΟʔϧ ϠϑʔגࣜձࣾʹͯϠϑΦΫ!ɾ޿ࠂͷ։ൃʹܞΘΔɻ2016೥͔

    Β͸גࣜձࣾVASILYɺ๏ਓ౷߹ʹ൐͍2018೥4݄͔Βݱ৬ɻݱ ࡏ͸SREΤϯδχΞͱͯ͠ɺ೔ʑαʔϏεͷ৴པੑ޲্ʹ౒Ί ΔɻVASILY࣌୅ͷऔΓ૊Έ͔Βԑ͋ͬͯMesos User Group TokyoʹࢀՃɻຊ൪؀ڥͰͷ࣮ફωλ୲౰ɻ΢ΟεΩʔͱ༿ר͕ ޷͖ɻѪͯ͠Δɻ ;0;05FDIOPMPHJFT *OD ։ൃ෦43&ςοΫϦʔυ ޫ໺ୡ࿕ Icon illustrated by YOSHI 
 https://ja-jp.facebook.com/yoshi.yone.7
  3. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ;0;05FDIOPMPHJFT *OD

    ZOZOグループのサービス運用・技術開発部門
  4. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓೔ຊ࠷େڃͷϑΝογϣϯγϣοϐϯάαΠτΞϓϦ ˓

    Ҏ্ͷγϣοϓɺ Ҏ্ͷϒϥϯυͷऔΓѻ͍ʢ ೥݄຤࣌఺ʣ ˓ৗ࣌ສ఺Ҏ্ͷ঎඼ΞΠςϜ਺ͱຖ೔ฏۉ ఺Ҏ্ͷ৽ண঎ ඼Λܝࡌ ˓ଈ೔഑ૹαʔϏεΪϑτϥοϐϯάαʔϏεπέ෷͍ͳͲ IUUQ[P[PKQ
  5. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓೔ຊ࠷େڃͷϑΝογϣϯίʔσΟωʔτΞϓϦ ˓

    ສμ΢ϯϩʔυಥഁɺίʔσΟωʔτ౤ߘ૯਺͸ສ݅Ҏ ্ʢͱ΋ʹ೥݄຤࣌఺ʣ ˓શੈքʢ"QQ4UPSF(PPHMFQMBZ͕ར༻ՄೳͳશͯͷࠃʣͰμ ΢ϯϩʔυ͕Մೳ ˓ສਓҎ্ͷϑΥϩϫʔΛ࣋ͭϢʔβʔʢ8&"3*45"ʣ΋஀ੜ IUUQTXFBSKQ
  6. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓౰͕ࣾಠࣗʹ։ൃͨ͠࠾ੇ༻ϘσΟʔεʔπ ˓શମʹࢪ͞ΕͨυοτϚʔΧʔΛεϚʔτϑΥϯΧϝϥͰ౓ࡱ

    Ө͢Δ͜ͱͰɺମܕσʔλΛܭଌ ˓ܭଌͨ͠ମܕσʔλ͸ɺॠ࣌ʹ%ϞσϧԽ͞Εɺ;0;0508/Ξ ϓϦʹอଘɻ%Ϟσϧ͸͋ΒΏΔ֯౓ʹಈ͔͢͜ͱ͕Ͱ͖ɺମܕΛ ౓νΣοΫ͢Δ͜ͱ͕Մೳ IUUQ[P[PKQ[P[PTVJU
  7. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓ʮ;0;046*5ʯͰܭଌͨ͠ମܕσʔλΛ΋ͱʹɺҰਓͻͱΓͷମ ܕʹ߹ͬͨʮ͋ͳͨαΠζʯͷΞΠςϜ

    ˓ʮڀۃͷϑΟοτײʯΛ࣮ݱͨ͠ϕʔγοΫΞΠςϜΛఏڙ άϩʔόϧαΠτʮ;0;0DPNʯͰւ֎ల։ ˓ΞΠςϜ̩γϟπɺσχϜύϯπɺγϟπɺϏδωεεʔπɺ ωΫλΠɺϘʔμʔ̩γϟπɺ௕କΫϧʔωοΫ̩γϟπͳͲ IUUQ[P[PKQQC
  8. Copyright © ZOZO Technologies, Inc. All Rights Reserved.

  9. Ҏ্ͷϑΝογϣϯ&$αΠτ͔Βͷ΂ສ఺Λ௒͑Δ঎඼Λܝࡌ ݄ؒສਓҎ্͕ར༻͢Δ೔ຊ࠷େڃͷϑΝογϣϯαΠτ

  10. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ✖

  11. Copyright © ZOZO Technologies, Inc. All Rights Reserved. *20/ͷதͰಈ͍͍ͯΔ.FTPT ΫϩʔϥʔγεςϜͷج൫ͱͯ͠

  12. Copyright © ZOZO Technologies, Inc. All Rights Reserved. *20/ͷϏδωε EC

    sites 1. ঎඼৘ใ ू໿ 3. ߪೖՄೳͳECαΠτ΁ૹ٬ 4. ܦ༝ച্ͷҰ෦ 2. Ӿཡ
  13. Copyright © ZOZO Technologies, Inc. All Rights Reserved. *20/ͷϏδωε EC

    sites ঎඼৘ใ ू໿ ॏཁ
  14. Copyright © ZOZO Technologies, Inc. All Rights Reserved. *20/ͷϏδωε EC

    sites ॏཁ IQON Ϋϩʔϥ
  15. Copyright © ZOZO Technologies, Inc. All Rights Reserved. εέʔϧײʢ೥౰࣌ʣ online-shot:

    Icons made by Vectors Market from www.flaticon.com is licensed by CC 3.0 BY dress: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY more than 300 EC sites per day more than 1,000,000 fashion items per day
  16. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Application Infrastructure

    RubyͰ࣮૷ ෼ࢄฒྻ γεςϜԽ EC2 Ϋϩʔϥʔͷਐา ߋͳΔଟஈԽ XPSLFS # XPSLFS " &$*OTUBODF XPSLFS " XPSLFS $ XPSLFS % XPSLFS "
  17. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Application Infrastructure

    Ϋϩʔϥʔͷਐา ߋͳΔଟஈԽ XPSLFS # XPSLFS " &$*OTUBODF XPSLFS " XPSLFS $ XPSLFS % XPSLFS " ࣍ͳΔεςʔδ ʢ2016೥ʣ EC2
  18. Copyright © ZOZO Technologies, Inc. All Rights Reserved. XPSLFS #

    XPSLFS " XPSLFS $ XPSLFS % XPSLFS " XPSLFS " XPSLFS # XPSLFS " XPSLFS $ XPSLFS % XPSLFS % XPSLFS % XPSLFS # XPSLFS $ XPSLFS % XPSLFS % TIME Ϋϩʔϥʔͷཧ૝
  19. Copyright © ZOZO Technologies, Inc. All Rights Reserved. XPSLFS #

    XPSLFS " XPSLFS $ XPSLFS % XPSLFS " XPSLFS " XPSLFS # XPSLFS " XPSLFS $ XPSLFS % XPSLFS % XPSLFS % XPSLFS # XPSLFS $ XPSLFS % XPSLFS % TIME ՝୊ • ϫʔΧʔຖʹCPU, ϝϞϦͱ͍ͬͨඞཁͳϦιʔε͕ҟͳΔ • ͦͷ࣌ʑͰݸʑͷΠϯελϯεϦιʔεΛߟྀ͢Δඞཁ͕͋Δ
 ʢͨ͘͠ͳ͍ʣ • ϫʔΧʔͷϓϩηεΛίϯτϩʔϧϓϨʔϯ͔Βૢ࡞Մೳʹ͢Δඞཁ͕͋Δ
 ʢͨ͘͠ͳ͍ʣ
  20. Copyright © ZOZO Technologies, Inc. All Rights Reserved. XPSLFS #

    XPSLFS " XPSLFS $ XPSLFS % XPSLFS " XPSLFS " XPSLFS # XPSLFS " XPSLFS $ XPSLFS % XPSLFS % XPSLFS % XPSLFS # XPSLFS $ XPSLFS % XPSLFS % TIME Ϋϩʔϥʔͷཧ૝
  21. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ىಈɾഇغ͕ ༰қͳശ

    ىಈɾഇغ͕ ༰қͳശ ىಈɾഇغ͕ ༰қͳശ ىಈɾഇغ͕ ༰қͳശ $PNQVUFS $PNQVUFS Կ͔ૉఢͳந৅Խ૚ XPSLFS " XPSLFS # $PNQVUFS XPSLFS $ XPSLFS % $PNQVUFS $PNQVUFS $PNQVUFS
  22. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ىಈɾഇغ͕ ༰қͳശ

    ىಈɾഇغ͕ ༰қͳശ ىಈɾഇغ͕ ༰қͳശ ىಈɾഇغ͕ ༰қͳശ $PNQVUFS $PNQVUFS ίϯςφΦʔέετϨʔγϣϯ૚ XPSLFS " XPSLFS # $PNQVUFS XPSLFS $ XPSLFS % $PNQVUFS $PNQVUFS $PNQVUFS
  23. Copyright © ZOZO Technologies, Inc. All Rights Reserved.

  24. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ΫϩʔϥʔPO.FTPT ܭࢉࢿݯ

    ܭࢉࢿݯͷ؅ཧɾ ू໿ɾந৅Խ ίϯςφͷঢ়گ؂ࢹ ΞϓϦέʔγϣϯ
  25. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ΫϩʔϥʔPO.FTPT ෼ࢄγεςϜ༻

    λεΫQueue Queue ؂ࢹ Queue؂ࢹ APIʹΑΔίϯςφૢ࡞
  26. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ΫϩʔϥʔPO.FTPT ϋʔυ΢ΣΞো֐Ͱ

    λεΫ͕ݮগͨ͠৔߹
  27. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ΫϩʔϥʔPO.FTPT Marathon͕λεΫ਺Λҡ࣋

    Mesos͕λεΫΛ࠶෼഑͢Δ
  28. Copyright © ZOZO Technologies, Inc. All Rights Reserved. racing: Icons

    made by Freepik from www.flaticon.com is licensed by CC 3.0 BY crawler operating time 67% down
  29. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ͳͥ.FTPTʁ shakehand,

    home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY
  30. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ϙΠϯτͦͷɿ୳͍ͯͨ͠΋ͷͱࢥ૝͕ϕετϚον shakehand,

    home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY • Mesosͷࢥ૝ɹɿ෼ࢄ؀ڥΛڊେͳܭࢉࢿݯͱͯ͠Έͳ͢ • ୳͍ͯͨ͠΋ͷ ɿϫʔΧʔΛ౎߹Α͘഑ஔ͢ΔͨΊͷܭࢉࢿݯ
 • ϫʔΧʔͷಛੑɿ෼ࢄɾඇಉظॲཧ
 ʹʼ CPU΍ϝϞϦͱ͍ͬͨܭࢉࢿݯʹͷΈґଘ • Mesosͷࢥ૝͸γϯϓϧͰɺ͔ͭϕετϚονͩͬͨ
  31. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ϙΠϯτͦͷɿγϯϓϧͳΫϥελ shakehand,

    home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY Mesos Master Zookeeper Mesos Slave Framework (Optional) • ࠷খ3ཁૉ • ϑϨʔϜϫʔΫ͸޷͖ͳ͚ͩ • LinuxσΟετϦʹΑͬͯ͸
 apt/yumͰΠϯετʔϧՄೳ
  32. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ϙΠϯτͦͷɿ๛෋ͳυΩϡϝϯτ shakehand,

    home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY • ৄࡉͳެࣜυΩϡϝϯτ • ΤʔδΣϯτͷύϥϝʔλ • Ξυόϯευͳ࢖͍ํͱ۩ମతͳྫ • APIͷ࢓༷ • ʮެࣜΛ୳ͤ͹ྑ͍ʯ
 ͱ͍͏҆৺ײ
  33. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ͳͥ.FTPTʁ shakehand,

    home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY
  34. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ͱ͜ΖͰ%$04ʁ ܭࢉࢿݯ

    Mesos Marathon ʢDC/OS͕ࣗ਎Λ؅ཧ͢ΔͨΊʣ DC/OS Marathon ʢΞϓϦέʔγϣϯ༻ʣ ΞϓϦέʔγϣϯ }1.ϑϨʔϜϫʔΫͷ૚͕ް͘ͳΔ 2.DC/OSͷར఺Ͱ͋Δͦͷଞͷ෼ࢄγεςϜΛಈ͔͢༧ఆ͕ͳ͍ 3.DC/OS͕࣋ͭϩʔυόϥϯαɾDNS͕ෆཁ γϯϓϧʹѻ͏ͨΊɺDC/OS͸ෆ࠾༻
  35. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ·ͱΊ •IQONΫϩʔϥʔج൫͸Mesos͕࢖ΘΕ͍ͯΔ

    • MesosʹΑͬͯϫʔΧʔͷ෼഑͕ޮ཰తʹߦ͑Δ༷ʹ • ෼ࢄॲཧͰ͖ΔΞϓϦέʔγϣϯͰ͋Ε͹Mesos͕ΑΓྑ͍બ୒ࢶͱͳΔ •Ϋϩʔϧͷऴ͕ྃ࣌ؒ໿1/3ʹ୹ॖ •τϥϒϧ࣌ͷλεΫ࠶഑ஔ͕ࣗಈͰߦΘΕΔͨΊγεςϜͷՄ༻ੑ޲্ • ͜ΕΛར༻ͯ͠AWS SpotInstanceͳͲΛ૊Έ߹ΘͤΔͱίετ΋࡟ݮՄೳ •ඞͣ͠΋DC/OS͸ඞཁͰͳ͍ͷͰɺέʔεόΠέʔεͰબ୒ ✖
  36. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ΋ͬͱ஌Γ͍ͨ Docker

    / Apache Mesos / Marathon ʹΑΔ3ഒ଎͍IQONΫϩʔϥʔͷߏங
 https://techblog.zozo.com/entry/iqon-crawler-by-docker-and-mesos-and-marathon
 Apache Mesos with Amazon EC2 SpotFleet https://speakerdeck.com/kotatsu360/apache-mesos-with-amazon-ec2-spotfleet
  37. Copyright © ZOZO Technologies, Inc. All Rights Reserved. 5IBOLZPV