Upgrade to Pro — share decks privately, control downloads, hide ads and more …

goodby-waiting-status-forever

 goodby-waiting-status-forever

Tatsuro Mitsuno

November 01, 2017
Tweet

More Decks by Tatsuro Mitsuno

Other Decks in Technology

Transcript

  1. © 2017 VASILY,Inc.
    શͯͷ“Waiting”Λ
    ੜ·ΕΔલʹফ͠ڈΓ͍ͨɻ
    Mesos Meetup Tokyo #3

    2017/11/01 Wed. Tatsuro MITSUNO

    View full-size slide

  2. © 2017 VASILY,Inc.
    ࣗݾ঺հ
    ▸ Tatsuro Mitsuno / ޫ໺ ୡ࿕
    ▸ 2012/04 Yahoo Japan Corporation
    ▸ 2016/04 VASILY, Inc.
    ▸ Infrastructure Engineer
    ▸ Twitter GitHub Qiita: @kotatsu360
    Icon illustrated by YOSHI 

    https://ja-jp.facebook.com/yoshi.yone.7
    Yukihiro "Matz" Matsumoto

    (VASILY technical advisor )

    View full-size slide

  3. © 2017 VASILY,Inc.
    Mesos Meetup Tokyo #2 ʹͯ
    https://speakerdeck.com/kotatsu360/apache-mesos-with-amazon-ec2-spotfleet
    Apache Mesos ͱ Amazon EC2 SpotFleetͰߏங͞Εͨ

    ฐࣾͷΫϩʔϥʔج൫ʹ͍ͭͯ͝঺հ͠·ͨ͠ɻ

    View full-size slide

  4. © 2017 VASILY,Inc.
    VASILYͷΫϩʔϥʔج൫

    scaling request
    control
    SpotFleet Instance
    scaling
    request
    Lambda
    CloudWatch /
    CloudWatch Event
    update request

    View full-size slide

  5. © 2017 VASILY,Inc.
    ຊ೔ͷςʔϚ
    Marathonͷ“Waiting”Λղܾ͢Δ
    ͜Ε

    View full-size slide

  6. © 2017 VASILY,Inc.
    Waitingͱ͸

    ▸ Marathon͕λεΫʹඞཁͳϦιʔεΛ֬อͰ͖ͳ͍ঢ়ଶ
    ▸ ެࣜυΩϡϝϯτͷτϥϒϧγϡʔςΟϯάͷୈҰ߲໨͸

    An app stays in “Waiting” forever
    ▸ ୭΋͕ૺ۰͢Δ
    ▸ ࣌ؒܦաͰղܾ͠ͳ͍
    ▸ ղܾํ๏ɿෆ଍͍ͯ͠ΔϦιʔεΛಛఆͯ͠ɺ௥Ճɾ։์

    View full-size slide

  7. © 2017 VASILY,Inc.
    Waitingͷ೉఺
    ▸ ͲͷϦιʔε͕ෆ଍͍ͯ͠Δ͔Θ͔Βͳ͍ɾɾɾ
    ▸ ϝϞϦɾCPUɾϙʔτɾetc,,,
    ▸ Mesos͕֬อՄೳͳϦιʔεશ͕ͯݪҼʹͳΓ͏Δ
    ▸ ͔͠͠ɺز͔ͭͷ͋Γ͕ͪͳύλʔϯʹ෼ྨՄೳ
    ▸ ຊ೔͝঺հ͢Δͷ͸࣮ࡍʹࢲ͕ૺ۰ͨ͠Waitingͱͦͷରࡦ

    View full-size slide

  8. © 2017 VASILY,Inc.
    Waitingͦͷ1ɿΫϥελϦιʔε଍Γͯ·͔͢ʁ

    View full-size slide

  9. © 2017 VASILY,Inc.
    Waitingͦͷ1ɿΫϥελϦιʔε଍Γͯ·͔͢ʁ
    ݪҼɿλεΫ͕࢖͏Ϧιʔε > MesosΫϥελͷۭ͖Ϧιʔε
    ֬ೝɿλεΫ͕ඞཁͱ͢ΔϝϞϦ΍CPUΛܭࢉͯ͠ɺMesos UIͰۭ͖Λ֬ೝ
    ରࡦɿMesosΫϥελΛ૿΍͢ɾλεΫ΁ͷׂ౰ྔΛݮΒ͢
    λεΫA
    ϝϞϦ512M
    ϝϞϦ 2048M
    ʮ2560M֬อʯ
    ʮϝϞϦ଍Βͳ͍ʯ
    λεΫA
    ϝϞϦ512M
    λεΫA
    ϝϞϦ512M
    λεΫA
    ϝϞϦ512M
    λεΫA
    ϝϞϦ512M
    λεΫAΛ5ݸ
    ϝϞϦ512M
    Waiting

    View full-size slide

  10. © 2017 VASILY,Inc.
    Waitingͦͷ2ɿλεΫͷ੍໿ຬͨͯ͠·͔͢ʁ

    View full-size slide

  11. © 2017 VASILY,Inc.
    Waitingͦͷ2ɿλεΫͷ੍໿ຬͨͯ͠·͔͢ʁ
    ݪҼɿ੍໿͖ͭλεΫͷ੍໿͕ຬͨͤͳ͍
    ֬ೝɿλεΫͷ੍໿͕ຬͨ͞Ε͍ͯΔ͔ݟ௚͢ʢಛʹϢχʔΫ੍໿ʣ
    ରࡦɿϊʔυΛ૿΍͢
    λεΫA
    ϝϞϦ512M
    λεΫA
    ϝϞϦ512M
    λεΫA
    ϝϞϦ512M
    λεΫA
    ϝϞϦ512M
    λεΫA
    ϝϞϦ512M
    λεΫBΛ5ݸ
    Constraints =
    hostname, unique
    Waiting
    ΤʔδΣϯτ
    ʮ1λεΫ/1ϗετʯ
    ʮϗετ଍Γͳ͍ʯ
    ΤʔδΣϯτ
    ΤʔδΣϯτ ΤʔδΣϯτ

    View full-size slide

  12. © 2017 VASILY,Inc.
    Waitingͦͷ2ѥछɿϦιʔεยدͬͯແ͍Ͱ͔͢ʁ
    ݪҼɿ੍໿͖ͭλεΫ͕࢖͏Ϧιʔε>ಛఆϗετͷۭ͖Ϧιʔε
    ֬ೝɿλεΫ͕ಛఆͷϗετʹยدͬͯͳ͍͔ʢكͳঢ়گʣ

    ରࡦɿఆظతʹλεΫΛϦελʔτ͢Δɾ੍໿͖ͭλεΫΛे෼ʹখ͘͢͞Δ
    λεΫA
    ϝϞϦ512M
    λεΫA
    ϝϞϦ512M
    λεΫBΛ2ݸ
    Constraints =
    hostname, unique
    Waiting
    ʮ1λεΫ/1ϗετʯ
    ʮ1୆ͰϝϞϦ଍Γͳ͍ʯ
    ΤʔδΣϯτ
    ϝϞϦ࢖༻཰
    90%
    ΤʔδΣϯτ
    ϝϞϦ࢖༻཰
    20%

    View full-size slide

  13. © 2017 VASILY,Inc.
    Waitingͦͷ3ɿϦιʔεͷ࢖༻ڐՄ͋Γ·͔͢ʁ

    View full-size slide

  14. © 2017 VASILY,Inc.
    Waitingͦͷ3ɿϦιʔεͷ࢖༻ڐՄ͋Γ·͔͢ʁ
    ݪҼɿλεΫ͕࢖͏ϙʔτͷ࢖༻ڐՄ͕ͳ͍
    ֬ೝɿλεΫ͕ಛผͳϦιʔεΛ࢖Θͳ͍͔֬ೝ
    ରࡦɿλεΫʹrequirePortsɺagentϓϩηεʹresourcesΦϓγϣϯΛ௥Ճ

    https://mesosphere.github.io/marathon/docs/troubleshooting.html#requiring-a-specific-host-port
    Waiting
    λεΫCΛ1ͭ
    ϗετϙʔτ80Λ࢖༻
    ʮ80ϙʔτʯ
    ʮڐՄແ͠ʯ
    σϑΥϧτઃఆͷ

    mesos-agentϓϩηε

    View full-size slide

  15. © 2017 VASILY,Inc.
    Waitingͦͷ3ѥछɿطଘλεΫ͕ར༻͍ͯ͠·ͤΜ͔ʁ
    ݪҼɿ৽نλεΫ͕࢖͍͍ͨϙʔτΛطଘλεΫ͕όΠϯυࡁΈ
    ֬ೝɿൃੜλΠϛϯάɻσϓϩΠ࣌ͳΒϙʔτ͕͔ͪ߹͍ͬͯΔՄೳੑ

    ରࡦɿupgradeStrategyͰɺminimumHealthCapacityͱmaximumOverCapacityʹ0Λࢦఆ

    ʢ৽نλεΫͷσϓϩΠલʹطଘλεΫ͕kill͞ΕΔʣ
    Waiting
    λεΫCΛ1ͭ
    ϗετϙʔτ80Λ࢖༻
    ʮ80ϙʔτʯ
    ʮ࢖༻தʯ
    λεΫC
    ʢrunningʣ

    View full-size slide

  16. © 2017 VASILY,Inc.
    ·ͱΊ
    ▸ λεΫ͕ඞཁͱ͢Δ֤छϦιʔε͕֬อͰ͖ͳ͍৔߹ɺWaitingʹͳΔ
    ▸ ୅දతͳWaitingύλʔϯ
    ▸ ϝϞϦɾCPU͕଍Γͳ͍
    ▸ ੍໿͕ຬͨͤͳ͍
    ▸ ಛఆͷϦιʔεΛ࢖͏ݖݶ͕ͳ͍
    ▸ ద੾ͳϦιʔε෼഑ͱಉ࣌ʹɺλεΫࣗମͷݟ௚͠΋
    ▸ ੍໿΍ϗετϙʔτ͕ඞཁͳ͍λεΫͰ͋Ε͹ࣗવʹ࢖͑Δ

    View full-size slide

  17. © 2017 VASILY,Inc.
    We are Hiring
    https://www.wantedly.com/companies/vasily

    View full-size slide