Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes (EKS) で構築する
スケーラブルなジョブ実行基盤

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers.

Kubernetes (EKS) で構築する
スケーラブルなジョブ実行基盤

Avatar for Kenta Suzuki

Kenta Suzuki

August 28, 2019
Tweet

More Decks by Kenta Suzuki

Other Decks in Technology

Transcript

  1. ࣗݾ঺հ • Kenta Suzuki • twitter: @kekekenta • CTO, primeNumber

    inc.
 εϥΠυͷ࠷ޙʹձࣾ঺հࡌͤ·ͨ͠ͷͰɺ͝ڵຯ͋Γ·ͨ͠Βޚཡ͍ͩ͘͞ •෼ੳج൫ߏஙɾӡ༻ࢧԉSaaSʮtrocco®ʯ
  2. troccoͷσʔλసૹδϣϒ • σʔλιʔε • MySQLɺPostgreSQLɺSQLServerͳͲ • εέδϡʔϧ࣮ߦ • ࠷୹1෼ִؒ ೚ҙͷσʔλιʔεͷςʔϒϧ౳ͷ಺༰ΛDWHʹసૹ

    troccoͷσʔλసૹ෦෼͸OSSͷembulkΛଟ༻͍ͯ͠·͢ɻ
 ͜ͷ৔ΛआΓͯɺίϛολʔͷํʑɺembulkίϛϡχςΟͷํʑʹײँਃ্͛͠·͢ɻ ࠓޙ͸embulkͷpluginެ։౳Ͱߩݙ͍ͯ͘͠༧ఆͰ͢ʂ
  3. ϗετͷεέʔϧΠϯ΋೉͍͠ • Auto Scaling Group ͷ Life cycle hook Λ࢖ͬͯɺΠϯ

    ελϯεఀࢭલʹDRAININGΛ࣮ࢪ • ىಈதͷλεΫ͕ͳΔ·Ͱ؂ࢹ͠ɺͳ͘ͳͬͨΒΠϯε λϯεఀࢭΛ͢͢ΊΔ • લड़ͷ௨ΓɺىಈதͷλεΫҰཡ͔Βফ͑ΔͨΊ͝ʹΐ ͝ʹΐ΍ͬͯͨ
  4. ݱࡏͷΞʔΩςΫνϟ SQS εέδϡʔϥʔ 8PSLFS 1PE +PC %JTQBUDIFS Kubernetes δϣϒ create

    pod schuedule pod &,4/0%& 8PSLFS 1PE $MVTUFS "VUPTDBMFS Scale in/out
  5. Job Dispatcher SQS εέδϡʔϥʔ 8PSLFS 1PE Kubernetes δϣϒ create pod

    schuedule pod &,4/0%& 8PSLFS 1PE $MVTUFS "VUPTDBMFS Scale in/out +PC %JTQBUDIFS
  6. Job Dispatcher • SQS Λ polling • k8s API ʹϦΫΤετΛ౤͛ɺ

    pod Λ࡞੒ ˞ίʔυ͸ΠϝʔδͰ͢ $16ϝϞϦࢦఆ
  7. Worker pod SQS εέδϡʔϥʔ +PC %JTQBUDIFS Kubernetes δϣϒ create pod

    schuedule pod &,4/0%& $MVTUFS "VUPTDBMFS Scale in/out 8PSLFS 1PE 8PSLFS 1PE
  8. Cluster autoscaler SQS εέδϡʔϥʔ +PC %JTQBUDIFS Kubernetes δϣϒ create pod

    schuedule pod &,4/0%& Scale in/out 8PSLFS 1PE 8PSLFS 1PE $MVTUFS "VUPTDBMFS
  9. ·ͱΊ • γϯϓϧ • k8sͷݞʹ৐Ε͹ɺεέʔϥϒϧͳδϣϒج൫ߏங΋γϯϓ ϧʹͳΔ • ؂ࢹ • ӡ༻࣮੷͕ແ͍ͷͰɺͱΓ͋͑ͣऔΕΔσʔλ͸ͱ͓ͬͯ͘

    • k8s΁ͷҠߦΛৼΓฦͬͯ • όον͔Βk8sಋೖ͢Δͷ͸ׂͱ΍Γ΍͍͢ͷͰ͓͢͢Ί • কདྷతʹ • কདྷతʹ͸ɺϗετͷ؅ཧͱ͔΍Βͳͯ͘ྑ͘ͳΔͱخ͍͠
  10. גࣜձࣾ primeNumber • 2015೥11݄ ઃཱ • ࣾһ9໊ʢ͏ͪΤϯδχΞ8໊ʣ
 ύʔτφʔΤϯδχΞ30໊Ҏ্ • 2018೥11݄

    trocco®ϦϦʔε • 2019೥7݄ γϦʔζAࢿۚௐୡʢ໿1.3ԯԁʣ࣮ࢪ https://primenumber.co.jp