goodby-waiting-status-forever

Mesos Meetup Tokyo #3
https://mesos.connpass.com/event/67877/

ref1. Apache Mesos 
with Amazon EC2 SpotFleet
https://speakerdeck.com/kotatsu360/apache-mesos-with-amazon-ec2-spotfleet

Transcript

1. © 2017 VASILY,Inc.
   શͯͷ“Waiting”Λ
   ੜ·ΕΔલʹফ͠ڈΓ͍ͨɻ
   Mesos Meetup Tokyo #3

   2017/11/01 Wed. Tatsuro MITSUNO
   

   View Slide

2. © 2017 VASILY,Inc.
   ࣗݾ঺հ
   ▸ Tatsuro Mitsuno / ޫ໺ ୡ࿕
   ▸ 2012/04 Yahoo Japan Corporation
   ▸ 2016/04 VASILY, Inc.
   ▸ Infrastructure Engineer
   ▸ Twitter GitHub Qiita: @kotatsu360
   Icon illustrated by YOSHI 

   https://ja-jp.facebook.com/yoshi.yone.7
   Yukihiro "Matz" Matsumoto

   (VASILY technical advisor )
   
   

   View Slide

3. © 2017 VASILY,Inc.
   Mesos Meetup Tokyo #2 ʹͯ
   https://speakerdeck.com/kotatsu360/apache-mesos-with-amazon-ec2-spotfleet
   Apache Mesos ͱ Amazon EC2 SpotFleetͰߏங͞Εͨ

   ฐࣾͷΫϩʔϥʔج൫ʹ͍ͭͯ͝঺հ͠·ͨ͠ɻ
   

   View Slide

4. © 2017 VASILY,Inc.
   VASILYͷΫϩʔϥʔج൫
   
   scaling request
   control
   SpotFleet Instance
   scaling
   request
   Lambda
   CloudWatch /
   CloudWatch Event
   update request
   

   View Slide

5. © 2017 VASILY,Inc.
   ຊ೔ͷςʔϚ
   Marathonͷ“Waiting”Λղܾ͢Δ
   ͜Ε
   

   View Slide

6. © 2017 VASILY,Inc.
   Waitingͱ͸
   
   ▸ Marathon͕λεΫʹඞཁͳϦιʔεΛ֬อͰ͖ͳ͍ঢ়ଶ
   ▸ ެࣜυΩϡϝϯτͷτϥϒϧγϡʔςΟϯάͷୈҰ߲໨͸

   An app stays in “Waiting” forever
   ▸ ୭΋͕ૺ۰͢Δ
   ▸ ࣌ؒܦաͰղܾ͠ͳ͍
   ▸ ղܾํ๏ɿෆ଍͍ͯ͠ΔϦιʔεΛಛఆͯ͠ɺ௥Ճɾ։์
   

   View Slide

7. © 2017 VASILY,Inc.
   Waitingͷ೉఺
   ▸ ͲͷϦιʔε͕ෆ଍͍ͯ͠Δ͔Θ͔Βͳ͍ɾɾɾ
   ▸ ϝϞϦɾCPUɾϙʔτɾetc,,,
   ▸ Mesos͕֬อՄೳͳϦιʔεશ͕ͯݪҼʹͳΓ͏Δ
   ▸ ͔͠͠ɺز͔ͭͷ͋Γ͕ͪͳύλʔϯʹ෼ྨՄೳ
   ▸ ຊ೔͝঺հ͢Δͷ͸࣮ࡍʹࢲ͕ૺ۰ͨ͠Waitingͱͦͷରࡦ
   
   

   View Slide

8. © 2017 VASILY,Inc.
   Waitingͦͷ1ɿΫϥελϦιʔε଍Γͯ·͔͢ʁ
   

   View Slide

9. © 2017 VASILY,Inc.
   Waitingͦͷ1ɿΫϥελϦιʔε଍Γͯ·͔͢ʁ
   ݪҼɿλεΫ͕࢖͏Ϧιʔε > MesosΫϥελͷۭ͖Ϧιʔε
   ֬ೝɿλεΫ͕ඞཁͱ͢ΔϝϞϦ΍CPUΛܭࢉͯ͠ɺMesos UIͰۭ͖Λ֬ೝ
   ରࡦɿMesosΫϥελΛ૿΍͢ɾλεΫ΁ͷׂ౰ྔΛݮΒ͢
   λεΫA
   ϝϞϦ512M
   ϝϞϦ 2048M
   ʮ2560M֬อʯ
   ʮϝϞϦ଍Βͳ͍ʯ
   λεΫA
   ϝϞϦ512M
   λεΫA
   ϝϞϦ512M
   λεΫA
   ϝϞϦ512M
   λεΫA
   ϝϞϦ512M
   λεΫAΛ5ݸ
   ϝϞϦ512M
   Waiting
   

   View Slide

10. © 2017 VASILY,Inc.
    Waitingͦͷ2ɿλεΫͷ੍໿ຬͨͯ͠·͔͢ʁ
    

    View Slide

11. © 2017 VASILY,Inc.
    Waitingͦͷ2ɿλεΫͷ੍໿ຬͨͯ͠·͔͢ʁ
    ݪҼɿ੍໿͖ͭλεΫͷ੍໿͕ຬͨͤͳ͍
    ֬ೝɿλεΫͷ੍໿͕ຬͨ͞Ε͍ͯΔ͔ݟ௚͢ʢಛʹϢχʔΫ੍໿ʣ
    ରࡦɿϊʔυΛ૿΍͢
    λεΫA
    ϝϞϦ512M
    λεΫA
    ϝϞϦ512M
    λεΫA
    ϝϞϦ512M
    λεΫA
    ϝϞϦ512M
    λεΫA
    ϝϞϦ512M
    λεΫBΛ5ݸ
    Constraints =
    hostname, unique
    Waiting
    ΤʔδΣϯτ
    ʮ1λεΫ/1ϗετʯ
    ʮϗετ଍Γͳ͍ʯ
    ΤʔδΣϯτ
    ΤʔδΣϯτ ΤʔδΣϯτ
    

    View Slide

12. © 2017 VASILY,Inc.
    Waitingͦͷ2ѥछɿϦιʔεยدͬͯແ͍Ͱ͔͢ʁ
    ݪҼɿ੍໿͖ͭλεΫ͕࢖͏Ϧιʔε>ಛఆϗετͷۭ͖Ϧιʔε
    ֬ೝɿλεΫ͕ಛఆͷϗετʹยدͬͯͳ͍͔ʢكͳঢ়گʣ

    ରࡦɿఆظతʹλεΫΛϦελʔτ͢Δɾ੍໿͖ͭλεΫΛे෼ʹখ͘͢͞Δ
    λεΫA
    ϝϞϦ512M
    λεΫA
    ϝϞϦ512M
    λεΫBΛ2ݸ
    Constraints =
    hostname, unique
    Waiting
    ʮ1λεΫ/1ϗετʯ
    ʮ1୆ͰϝϞϦ଍Γͳ͍ʯ
    ΤʔδΣϯτ
    ϝϞϦ࢖༻཰
    90%
    ΤʔδΣϯτ
    ϝϞϦ࢖༻཰
    20%
    

    View Slide

13. © 2017 VASILY,Inc.
    Waitingͦͷ3ɿϦιʔεͷ࢖༻ڐՄ͋Γ·͔͢ʁ
    

    View Slide

14. © 2017 VASILY,Inc.
    Waitingͦͷ3ɿϦιʔεͷ࢖༻ڐՄ͋Γ·͔͢ʁ
    ݪҼɿλεΫ͕࢖͏ϙʔτͷ࢖༻ڐՄ͕ͳ͍
    ֬ೝɿλεΫ͕ಛผͳϦιʔεΛ࢖Θͳ͍͔֬ೝ
    ରࡦɿλεΫʹrequirePortsɺagentϓϩηεʹresourcesΦϓγϣϯΛ௥Ճ

    https://mesosphere.github.io/marathon/docs/troubleshooting.html#requiring-a-specific-host-port
    Waiting
    λεΫCΛ1ͭ
    ϗετϙʔτ80Λ࢖༻
    ʮ80ϙʔτʯ
    ʮڐՄແ͠ʯ
    σϑΥϧτઃఆͷ

    mesos-agentϓϩηε
    

    View Slide

15. © 2017 VASILY,Inc.
    Waitingͦͷ3ѥछɿطଘλεΫ͕ར༻͍ͯ͠·ͤΜ͔ʁ
    ݪҼɿ৽نλεΫ͕࢖͍͍ͨϙʔτΛطଘλεΫ͕όΠϯυࡁΈ
    ֬ೝɿൃੜλΠϛϯάɻσϓϩΠ࣌ͳΒϙʔτ͕͔ͪ߹͍ͬͯΔՄೳੑ

    ରࡦɿupgradeStrategyͰɺminimumHealthCapacityͱmaximumOverCapacityʹ0Λࢦఆ

    ʢ৽نλεΫͷσϓϩΠલʹطଘλεΫ͕kill͞ΕΔʣ
    Waiting
    λεΫCΛ1ͭ
    ϗετϙʔτ80Λ࢖༻
    ʮ80ϙʔτʯ
    ʮ࢖༻தʯ
    λεΫC
    ʢrunningʣ
    

    View Slide

16. © 2017 VASILY,Inc.
    ·ͱΊ
    ▸ λεΫ͕ඞཁͱ͢Δ֤छϦιʔε͕֬อͰ͖ͳ͍৔߹ɺWaitingʹͳΔ
    ▸ ୅දతͳWaitingύλʔϯ
    ▸ ϝϞϦɾCPU͕଍Γͳ͍
    ▸ ੍໿͕ຬͨͤͳ͍
    ▸ ಛఆͷϦιʔεΛ࢖͏ݖݶ͕ͳ͍
    ▸ ద੾ͳϦιʔε෼഑ͱಉ࣌ʹɺλεΫࣗମͷݟ௚͠΋
    ▸ ੍໿΍ϗετϙʔτ͕ඞཁͳ͍λεΫͰ͋Ε͹ࣗવʹ࢖͑Δ
    
    

    View Slide

17. © 2017 VASILY,Inc.
    We are Hiring
    https://www.wantedly.com/companies/vasily
    

    View Slide