Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes (EKS) で構築する
スケーラブルなジョブ実行基盤

Kubernetes (EKS) で構築する
スケーラブルなジョブ実行基盤

Kenta Suzuki

August 28, 2019
Tweet

More Decks by Kenta Suzuki

Other Decks in Technology

Transcript

  1. ࣗݾ঺հ • Kenta Suzuki • twitter: @kekekenta • CTO, primeNumber

    inc.
 εϥΠυͷ࠷ޙʹձࣾ঺հࡌͤ·ͨ͠ͷͰɺ͝ڵຯ͋Γ·ͨ͠Βޚཡ͍ͩ͘͞ •෼ੳج൫ߏஙɾӡ༻ࢧԉSaaSʮtrocco®ʯ
  2. troccoͷσʔλసૹδϣϒ • σʔλιʔε • MySQLɺPostgreSQLɺSQLServerͳͲ • εέδϡʔϧ࣮ߦ • ࠷୹1෼ִؒ ೚ҙͷσʔλιʔεͷςʔϒϧ౳ͷ಺༰ΛDWHʹసૹ

    troccoͷσʔλసૹ෦෼͸OSSͷembulkΛଟ༻͍ͯ͠·͢ɻ
 ͜ͷ৔ΛआΓͯɺίϛολʔͷํʑɺembulkίϛϡχςΟͷํʑʹײँਃ্͛͠·͢ɻ ࠓޙ͸embulkͷpluginެ։౳Ͱߩݙ͍ͯ͘͠༧ఆͰ͢ʂ
  3. ϗετͷεέʔϧΠϯ΋೉͍͠ • Auto Scaling Group ͷ Life cycle hook Λ࢖ͬͯɺΠϯ

    ελϯεఀࢭલʹDRAININGΛ࣮ࢪ • ىಈதͷλεΫ͕ͳΔ·Ͱ؂ࢹ͠ɺͳ͘ͳͬͨΒΠϯε λϯεఀࢭΛ͢͢ΊΔ • લड़ͷ௨ΓɺىಈதͷλεΫҰཡ͔Βফ͑ΔͨΊ͝ʹΐ ͝ʹΐ΍ͬͯͨ
  4. ݱࡏͷΞʔΩςΫνϟ SQS εέδϡʔϥʔ 8PSLFS 1PE +PC %JTQBUDIFS Kubernetes δϣϒ create

    pod schuedule pod &,4/0%& 8PSLFS 1PE $MVTUFS "VUPTDBMFS Scale in/out
  5. Job Dispatcher SQS εέδϡʔϥʔ 8PSLFS 1PE Kubernetes δϣϒ create pod

    schuedule pod &,4/0%& 8PSLFS 1PE $MVTUFS "VUPTDBMFS Scale in/out +PC %JTQBUDIFS
  6. Job Dispatcher • SQS Λ polling • k8s API ʹϦΫΤετΛ౤͛ɺ

    pod Λ࡞੒ ˞ίʔυ͸ΠϝʔδͰ͢ $16ϝϞϦࢦఆ
  7. Worker pod SQS εέδϡʔϥʔ +PC %JTQBUDIFS Kubernetes δϣϒ create pod

    schuedule pod &,4/0%& $MVTUFS "VUPTDBMFS Scale in/out 8PSLFS 1PE 8PSLFS 1PE
  8. Cluster autoscaler SQS εέδϡʔϥʔ +PC %JTQBUDIFS Kubernetes δϣϒ create pod

    schuedule pod &,4/0%& Scale in/out 8PSLFS 1PE 8PSLFS 1PE $MVTUFS "VUPTDBMFS
  9. ·ͱΊ • γϯϓϧ • k8sͷݞʹ৐Ε͹ɺεέʔϥϒϧͳδϣϒج൫ߏங΋γϯϓ ϧʹͳΔ • ؂ࢹ • ӡ༻࣮੷͕ແ͍ͷͰɺͱΓ͋͑ͣऔΕΔσʔλ͸ͱ͓ͬͯ͘

    • k8s΁ͷҠߦΛৼΓฦͬͯ • όον͔Βk8sಋೖ͢Δͷ͸ׂͱ΍Γ΍͍͢ͷͰ͓͢͢Ί • কདྷతʹ • কདྷతʹ͸ɺϗετͷ؅ཧͱ͔΍Βͳͯ͘ྑ͘ͳΔͱخ͍͠
  10. גࣜձࣾ primeNumber • 2015೥11݄ ઃཱ • ࣾһ9໊ʢ͏ͪΤϯδχΞ8໊ʣ
 ύʔτφʔΤϯδχΞ30໊Ҏ্ • 2018೥11݄

    trocco®ϦϦʔε • 2019೥7݄ γϦʔζAࢿۚௐୡʢ໿1.3ԯԁʣ࣮ࢪ https://primenumber.co.jp