goodby-waiting-status-forever

 goodby-waiting-status-forever

231dbe4867c139c325b61e808b757750?s=128

Tatsuro Mitsuno

November 01, 2017
Tweet

Transcript

  1. © 2017 VASILY,Inc. શͯͷ“Waiting”Λ ੜ·ΕΔલʹফ͠ڈΓ͍ͨɻ Mesos Meetup Tokyo #3
 2017/11/01

    Wed. Tatsuro MITSUNO
  2. © 2017 VASILY,Inc. ࣗݾ঺հ ▸ Tatsuro Mitsuno / ޫ໺ ୡ࿕

    ▸ 2012/04 Yahoo Japan Corporation ▸ 2016/04 VASILY, Inc. ▸ Infrastructure Engineer ▸ Twitter GitHub Qiita: @kotatsu360 Icon illustrated by YOSHI 
 https://ja-jp.facebook.com/yoshi.yone.7 Yukihiro "Matz" Matsumoto
 (VASILY technical advisor ) 
  3. © 2017 VASILY,Inc. Mesos Meetup Tokyo #2 ʹͯ https://speakerdeck.com/kotatsu360/apache-mesos-with-amazon-ec2-spotfleet Apache

    Mesos ͱ Amazon EC2 SpotFleetͰߏங͞Εͨ
 ฐࣾͷΫϩʔϥʔج൫ʹ͍ͭͯ͝঺հ͠·ͨ͠ɻ
  4. © 2017 VASILY,Inc. VASILYͷΫϩʔϥʔج൫  scaling request control SpotFleet Instance

    scaling request Lambda CloudWatch / CloudWatch Event update request
  5. © 2017 VASILY,Inc. ຊ೔ͷςʔϚ Marathonͷ“Waiting”Λղܾ͢Δ ͜Ε

  6. © 2017 VASILY,Inc. Waitingͱ͸  ▸ Marathon͕λεΫʹඞཁͳϦιʔεΛ֬อͰ͖ͳ͍ঢ়ଶ ▸ ެࣜυΩϡϝϯτͷτϥϒϧγϡʔςΟϯάͷୈҰ߲໨͸
 An

    app stays in “Waiting” forever ▸ ୭΋͕ૺ۰͢Δ ▸ ࣌ؒܦաͰղܾ͠ͳ͍ ▸ ղܾํ๏ɿෆ଍͍ͯ͠ΔϦιʔεΛಛఆͯ͠ɺ௥Ճɾ։์
  7. © 2017 VASILY,Inc. Waitingͷ೉఺ ▸ ͲͷϦιʔε͕ෆ଍͍ͯ͠Δ͔Θ͔Βͳ͍ɾɾɾ ▸ ϝϞϦɾCPUɾϙʔτɾetc,,, ▸ Mesos͕֬อՄೳͳϦιʔεશ͕ͯݪҼʹͳΓ͏Δ

    ▸ ͔͠͠ɺز͔ͭͷ͋Γ͕ͪͳύλʔϯʹ෼ྨՄೳ ▸ ຊ೔͝঺հ͢Δͷ͸࣮ࡍʹࢲ͕ૺ۰ͨ͠Waitingͱͦͷରࡦ 
  8. © 2017 VASILY,Inc. Waitingͦͷ1ɿΫϥελϦιʔε଍Γͯ·͔͢ʁ

  9. © 2017 VASILY,Inc.  Waitingͦͷ1ɿΫϥελϦιʔε଍Γͯ·͔͢ʁ ݪҼɿλεΫ͕࢖͏Ϧιʔε > MesosΫϥελͷۭ͖Ϧιʔε ֬ೝɿλεΫ͕ඞཁͱ͢ΔϝϞϦ΍CPUΛܭࢉͯ͠ɺMesos UIͰۭ͖Λ֬ೝ

    ରࡦɿMesosΫϥελΛ૿΍͢ɾλεΫ΁ͷׂ౰ྔΛݮΒ͢ λεΫA ϝϞϦ512M ϝϞϦ 2048M ʮ2560M֬อʯ ʮϝϞϦ଍Βͳ͍ʯ λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫAΛ5ݸ ϝϞϦ512M Waiting
  10. © 2017 VASILY,Inc. Waitingͦͷ2ɿλεΫͷ੍໿ຬͨͯ͠·͔͢ʁ

  11. © 2017 VASILY,Inc.  Waitingͦͷ2ɿλεΫͷ੍໿ຬͨͯ͠·͔͢ʁ ݪҼɿ੍໿͖ͭλεΫͷ੍໿͕ຬͨͤͳ͍ ֬ೝɿλεΫͷ੍໿͕ຬͨ͞Ε͍ͯΔ͔ݟ௚͢ʢಛʹϢχʔΫ੍໿ʣ ରࡦɿϊʔυΛ૿΍͢ λεΫA ϝϞϦ512M

    λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫBΛ5ݸ Constraints = hostname, unique Waiting ΤʔδΣϯτ ʮ1λεΫ/1ϗετʯ ʮϗετ଍Γͳ͍ʯ ΤʔδΣϯτ ΤʔδΣϯτ ΤʔδΣϯτ
  12. © 2017 VASILY,Inc.  Waitingͦͷ2ѥछɿϦιʔεยدͬͯແ͍Ͱ͔͢ʁ ݪҼɿ੍໿͖ͭλεΫ͕࢖͏Ϧιʔε>ಛఆϗετͷۭ͖Ϧιʔε ֬ೝɿλεΫ͕ಛఆͷϗετʹยدͬͯͳ͍͔ʢكͳঢ়گʣ
 ରࡦɿఆظతʹλεΫΛϦελʔτ͢Δɾ੍໿͖ͭλεΫΛे෼ʹখ͘͢͞Δ λεΫA ϝϞϦ512M

    λεΫA ϝϞϦ512M λεΫBΛ2ݸ Constraints = hostname, unique Waiting ʮ1λεΫ/1ϗετʯ ʮ1୆ͰϝϞϦ଍Γͳ͍ʯ ΤʔδΣϯτ ϝϞϦ࢖༻཰ 90% ΤʔδΣϯτ ϝϞϦ࢖༻཰ 20%
  13. © 2017 VASILY,Inc. Waitingͦͷ3ɿϦιʔεͷ࢖༻ڐՄ͋Γ·͔͢ʁ

  14. © 2017 VASILY,Inc.  Waitingͦͷ3ɿϦιʔεͷ࢖༻ڐՄ͋Γ·͔͢ʁ ݪҼɿλεΫ͕࢖͏ϙʔτͷ࢖༻ڐՄ͕ͳ͍ ֬ೝɿλεΫ͕ಛผͳϦιʔεΛ࢖Θͳ͍͔֬ೝ ରࡦɿλεΫʹrequirePortsɺagentϓϩηεʹresourcesΦϓγϣϯΛ௥Ճ
 https://mesosphere.github.io/marathon/docs/troubleshooting.html#requiring-a-specific-host-port Waiting

    λεΫCΛ1ͭ ϗετϙʔτ80Λ࢖༻ ʮ80ϙʔτʯ ʮڐՄແ͠ʯ σϑΥϧτઃఆͷ
 mesos-agentϓϩηε
  15. © 2017 VASILY,Inc.  Waitingͦͷ3ѥछɿطଘλεΫ͕ར༻͍ͯ͠·ͤΜ͔ʁ ݪҼɿ৽نλεΫ͕࢖͍͍ͨϙʔτΛطଘλεΫ͕όΠϯυࡁΈ ֬ೝɿൃੜλΠϛϯάɻσϓϩΠ࣌ͳΒϙʔτ͕͔ͪ߹͍ͬͯΔՄೳੑ
 ରࡦɿupgradeStrategyͰɺminimumHealthCapacityͱmaximumOverCapacityʹ0Λࢦఆ
 ʢ৽نλεΫͷσϓϩΠલʹطଘλεΫ͕kill͞ΕΔʣ Waiting

    λεΫCΛ1ͭ ϗετϙʔτ80Λ࢖༻ ʮ80ϙʔτʯ ʮ࢖༻தʯ λεΫC ʢrunningʣ
  16. © 2017 VASILY,Inc. ·ͱΊ ▸ λεΫ͕ඞཁͱ͢Δ֤छϦιʔε͕֬อͰ͖ͳ͍৔߹ɺWaitingʹͳΔ ▸ ୅දతͳWaitingύλʔϯ ▸ ϝϞϦɾCPU͕଍Γͳ͍

    ▸ ੍໿͕ຬͨͤͳ͍ ▸ ಛఆͷϦιʔεΛ࢖͏ݖݶ͕ͳ͍ ▸ ద੾ͳϦιʔε෼഑ͱಉ࣌ʹɺλεΫࣗମͷݟ௚͠΋ ▸ ੍໿΍ϗετϙʔτ͕ඞཁͳ͍λεΫͰ͋Ε͹ࣗવʹ࢖͑Δ 
  17. © 2017 VASILY,Inc. We are Hiring https://www.wantedly.com/companies/vasily