Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kubernetes (EKS) で構築する スケーラブルなジョブ実行基盤
Search
Kenta Suzuki
August 28, 2019
Technology
7
6.2k
Kubernetes (EKS) で構築する スケーラブルなジョブ実行基盤
Kenta Suzuki
August 28, 2019
Tweet
Share
More Decks by Kenta Suzuki
See All by Kenta Suzuki
trocco®での インシデント重大度表(SEVレベル表)運用
kekekenta
0
2.7k
Ruby はじめての型 Gotanda.rb 2023/04/20
kekekenta
0
260
trocco® の品質を守る、とても普通な取り組み
kekekenta
0
4.8k
Embulkを利用したデータ統合SaaSの構築と運用
kekekenta
2
4.7k
Other Decks in Technology
See All in Technology
IAMのマニアックな話2025
nrinetcom
PRO
3
440
生成AI×財務経理:PoCで挑むSlack AI Bot開発と現場巻き込みのリアル
pohdccoe
1
700
クラウド食堂とは?
hiyanger
0
110
Change Managerを活用して本番環境へのセキュアなGUIアクセスを統制する / Control Secure GUI Access to the Production Environment with Change Manager
yuj1osm
0
100
Cracking the Coding Interview 6th Edition
gdplabs
14
28k
What's new in Go 1.24?
ciarana
1
110
AIエージェント時代のエンジニアになろう #jawsug #jawsdays2025 / 20250301 Agentic AI Engineering
yoshidashingo
8
3.7k
Active Directory攻防
cryptopeg
PRO
8
5.5k
クラウド関連のインシデントケースを収集して見えてきたもの
lhazy
7
940
Pwned Labsのすゝめ
ken5scal
2
430
Windows の新しい管理者保護モード
murachiakira
0
200
DevinでAI AWSエンジニア製造計画 序章 〜CDKを添えて〜/devin-load-to-aws-engineer
tomoki10
0
120
Featured
See All Featured
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
Making the Leap to Tech Lead
cromwellryan
133
9.1k
Why You Should Never Use an ORM
jnunemaker
PRO
55
9.2k
The Pragmatic Product Professional
lauravandoore
32
6.4k
Optimizing for Happiness
mojombo
376
70k
Six Lessons from altMBA
skipperchong
27
3.6k
The Art of Programming - Codeland 2020
erikaheidi
53
13k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
21
2.5k
Fireside Chat
paigeccino
34
3.2k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Adopting Sorbet at Scale
ufuk
74
9.2k
The World Runs on Bad Software
bkeepers
PRO
67
11k
Transcript
Kubernetes (EKS) Ͱߏங͢Δ εέʔϥϒϧͳδϣϒ࣮ߦج൫ Kenta Suzuki (twitter: @kekekenta) CTO, primeNumber
inc 2019/8/28 Kubernetes Meetup Tokyo #22
ࣗݾհ • Kenta Suzuki • twitter: @kekekenta • CTO, primeNumber
inc. εϥΠυͷ࠷ޙʹձࣾհࡌͤ·ͨ͠ͷͰɺ͝ڵຯ͋Γ·ͨ͠Βޚཡ͍ͩ͘͞ •ੳج൫ߏஙɾӡ༻ࢧԉSaaSʮtrocco®ʯ
.Z42- 4 4BMFTGPSDF #JH2VFSZ ੳج൫ߏஙɾӡ༻ σʔλసૹ αϚϦʔςʔϒϧ࡞ σʔλࢹ ֎෦࿈ܞ
.Z42- 4 4BMFTGPSDF #JH2VFSZ σʔλసૹ αϚϦʔςʔϒϧ࡞ σʔλࢹ ֎෦࿈ܞ ੳج൫ͷߏஙɾӡ༻Λ·ͱΊͯΧόʔ
troccoͷσʔλసૹδϣϒ • σʔλιʔε • MySQLɺPostgreSQLɺSQLServerͳͲ • εέδϡʔϧ࣮ߦ • ࠷1ִؒ ҙͷσʔλιʔεͷςʔϒϧͷ༰ΛDWHʹసૹ
troccoͷσʔλసૹ෦OSSͷembulkΛଟ༻͍ͯ͠·͢ɻ ͜ͷΛआΓͯɺίϛολʔͷํʑɺembulkίϛϡχςΟͷํʑʹײँਃ্͛͠·͢ɻ ࠓޙembulkͷpluginެ։Ͱߩݙ͍ͯ͘͠༧ఆͰ͢ʂ
troccoͷసૹδϣϒ • δϣϒ࣮ߦ • ʙ10,000/ • δϣϒ͝ͱͷ࣮ߦ࣌ؒ • min: 5s
ʙ max: 5h • ࣮ߦ࣌ؒଳ • ࣌ؒʹ1ͷεύΠΫ
Fan out on ECS چΞʔΩςΫνϟ
Fan out on ECSαʔϏε SQS εέδϡʔϥʔ 8PSLFS &$4UBTL 8PSLFS &$4UBTL
8PSLFS &$4UBTL ECS ECS NODE δϣϒ polling
λεΫεέʔϧΞτͷλΠϛϯά͍͠ • SQSͷΩϡʔͷαΠζΛࢹ • CloudwatchϝτϦΫε͕࠷1͝ͱ • εέʔϧ·Ͱɺ1ͨͳ͍ͱߦ͚ͳ͍
ϗετεέʔϧΞτͷλΠϛϯά͍͠ • λεΫ͕͏ՃͰ্ཱ͕ͪΔϦιʔε͕ ͳ͘ͳͬͨΒεέʔϧ • ىಈͪͷλεΫ͕ग़དྷͨλΠϛϯάͰϗ ετεέʔϧͯ͠΄͍͠ɻɻ
λεΫεέʔϧΠϯ͍͠ • ECSλεΫΛ҆શʹऴྃͤ͞Δඞཁ͕͋Δ • సૹδϣϒ5hʹٴͿ߹͋Δ • TERMγάφϧͷτϥοϓͯ͠graceful shutdownΛ ࣮ •
ͨͩ͠ɺECSͷίϯιʔϧ্ͷ࣮ߦதλεΫҰཡ͔ΒɺTERMΛૹͬͯ͠Β͘͢Δ ͱফ͑ͯ͠·͏
ϗετͷεέʔϧΠϯ͍͠ • Auto Scaling Group ͷ Life cycle hook ΛͬͯɺΠϯ
ελϯεఀࢭલʹDRAININGΛ࣮ࢪ • ىಈதͷλεΫ͕ͳΔ·Ͱࢹ͠ɺͳ͘ͳͬͨΒΠϯε λϯεఀࢭΛ͢͢ΊΔ • લड़ͷ௨ΓɺىಈதͷλεΫҰཡ͔Βফ͑ΔͨΊ͝ʹΐ ͝ʹΐͬͯͨ
Ϧιʔεׂ • సૹߴԽΦϓγϣϯͱͯ͠ɺCPU/ϝϞϦ্ݶΛઃఆ Ͱ͖ΔΑ͏ʹ͍ͨ͠ • ECSͰλεΫ͝ͱʹϦιʔε্ݶΛઃఆͰ͖Δ͕ɺى ಈ࣌ͷΈ • Ϧιʔε্ݶ͝ͱʹαʔϏεΛ͚Δඞཁ͕͋Γ
͍͠
ͦ͜Ͱɺk8s
k8sͷpodεέδϡʔϥʔʹͭ ͍ͯ
k8sͷpodεέδϡʔϥʔʹͭ ͍ͯ
k8sͷpodεέδϡʔϥʔʹͭ ͍ͯ statusͷભҠ "Pending" -> "Running" -> "Succeeded"
k8sͷpodεέδϡʔϥʔ • nodeϦιʔεͷۭ͖ঢ়گʹԠͯ͡ɺεέδϡʔϥʔ͕ॱ࣍podΛ nodeʹׂΓͯͯ͘ΕΔ • podΛ࡞͓͚ͯ͠ɺ͋ͱ࣮ߦ͞ΕΔͷΛ͚ͭͩ ,VCFSOFUFT QPE /0%& QPE
QPE pod্ཱͪ͛ ࣮ߦͪpod create pod
ͱΓ͋͑ͣࡶʹpodΛcreate͢Ε ɺ͋ͱk8s͕࣮ߦ·Ͱ໘ݟ ͯ͘ΕΔ
ϊʔυͷεέʔϧΞτ/Πϯ • EKSͷ߹ɺcluster autoscalerΛ͏ͷ͕σϑΝΫτʢͱ ࢥΘΕΔʣ https://github.com/kubernetes/autoscaler/tree/master/ cluster-autoscaler/cloudprovider/aws • podεέδϡʔϦϯάͷঢ়گʹԠͯ͡ɺEC2ϊʔυͷASG ܦ༝ͰεέʔϧΠϯɾΞτΛͬͯ͘ΕΔ
ϊʔυͷεέʔϧΞτ QPE &$/0%& QPE QPE 1FOEJOH ࠓͷnodeϦιʔεͰɺ podΛ্ཱͪ͛Δ͜ͱ͕ग़དྷͳ͍
ϊʔυͷεέʔϧΞτ QPE &$/0%& QPE &$/0%& QPE ClusterAutoscaler͕ASGܦ༝ͰnodeΛεέʔϧΞτ
ϊʔυεέʔϧΠϯ QPE &$/0%& QPE &$/0%& QPE
ϊʔυεέʔϧΠϯ QPE &$/0%& QPE &$/0%& Ϧιʔεʹ༨༟͕͋Δ
ϊʔυεέʔϧΠϯ QPE &$4/0%& QPE εέʔϧΠϯ
1. create pod ࣮ͯ͠ߦΛͭ 2. ඞཁʹԠͯ͡ɺnode͕εέʔϧΞ τ͢ΔͷΛͭ εέʔϥϒϧδϣϒج൫ര
k8sʹΑΔδϣϒج൫
ݱࡏͷΞʔΩςΫνϟ SQS εέδϡʔϥʔ 8PSLFS 1PE +PC %JTQBUDIFS Kubernetes δϣϒ create
pod schuedule pod &,4/0%& 8PSLFS 1PE $MVTUFS "VUPTDBMFS Scale in/out
Job Dispatcher SQS εέδϡʔϥʔ 8PSLFS 1PE Kubernetes δϣϒ create pod
schuedule pod &,4/0%& 8PSLFS 1PE $MVTUFS "VUPTDBMFS Scale in/out +PC %JTQBUDIFS
Job Dispatcher ˞ίʔυΠϝʔδͰ͢ • SQS Λ polling • k8s API
ʹϦΫΤετΛ͛ɺ pod Λ࡞
Job Dispatcher ˞ίʔυΠϝʔδͰ͢ LT"1*ʹϦΫΤετ • SQS Λ polling • k8s
API ʹϦΫΤετΛ͛ɺ pod Λ࡞
Job Dispatcher ˞ίʔυΠϝʔδͰ͢ EPDLFSΠϝʔ δͷλάࢦఆ • SQS Λ polling •
k8s API ʹϦΫΤετΛ͛ɺ pod Λ࡞
Job Dispatcher ˞ίʔυΠϝʔδͰ͢ QPEͰ࣮ߦ͢Δ ίϚϯυΛࢦఆ • SQS Λ polling •
k8s API ʹϦΫΤετΛ͛ɺ pod Λ࡞
Job Dispatcher • SQS Λ polling • k8s API ʹϦΫΤετΛ͛ɺ
pod Λ࡞ ˞ίʔυΠϝʔδͰ͢ $16ϝϞϦࢦఆ
Worker pod SQS εέδϡʔϥʔ +PC %JTQBUDIFS Kubernetes δϣϒ create pod
schuedule pod &,4/0%& $MVTUFS "VUPTDBMFS Scale in/out 8PSLFS 1PE 8PSLFS 1PE
Worker pod • ࣮ࡍͷసૹΛ࣮ߦ • సૹ։࢝Ͱ্ཱ͕ͪΓɺసૹ͕ऴྃͨ͠Βpodऴྃ ˞ίʔυΠϝʔδͰ͢
Cluster autoscaler SQS εέδϡʔϥʔ +PC %JTQBUDIFS Kubernetes δϣϒ create pod
schuedule pod &,4/0%& Scale in/out 8PSLFS 1PE 8PSLFS 1PE $MVTUFS "VUPTDBMFS
Cluster autoscaler ˞ίʔυΠϝʔδͰ͢
k8sδϣϒج൫ಋೖͷՌ • ѹతίετݮʹޭ • εέʔϧͷλΠϛϯά͕͍͕͍͍͠ ײ͡ʹղܾͰ͖ͨ • cluster autoscaler༷༷ •
։ൃڥͱຊ൪ڥͷಉҰԽ
ࢹʹ͍ͭͯ • create podͰ͖ͳ͍ • pod͕pendingͷ··্ཱ͕ͪΒͳ͍ • pod͕runningதʹಥવࢮ ຊ൪ӡ༻࣮͕ແ͍ͷͰɺຊʹظ௨Γʹ ಈ͔͘ͱ͍͏ҙຯͰࢹΛ࣮ࢪ
troccoͷδϣϒεςʔλεભҠ • DB্ʹδϣϒ͝ͱͷεςʔλεΛอ࣋ • dispatcherpodͳͲ͕ͦΕͧΕߋ৽͢Δ RVFVFE TFUUJOH@VQ FYFDVUJOH TVDDFFEFE SQSʹΩϡʔΠϯά
create pod ྃ pod্ཱ͕ͪΓ·Γ pod͕࣮ߦத సૹྃ
redashͰͷࢹ RVFVFE TFUUJOH@VQ FYFDVUJOH TVDDFFEFE ֤εςʔλεͰཹ͍ͯͨ͠ΒΞϥʔτ
·ͱΊ • γϯϓϧ • k8sͷݞʹΕɺεέʔϥϒϧͳδϣϒج൫ߏஙγϯϓ ϧʹͳΔ • ࢹ • ӡ༻࣮͕ແ͍ͷͰɺͱΓ͋͑ͣऔΕΔσʔλͱ͓ͬͯ͘
• k8sͷҠߦΛৼΓฦͬͯ • όον͔Βk8sಋೖ͢ΔͷׂͱΓ͍͢ͷͰ͓͢͢Ί • কདྷతʹ • কདྷతʹɺϗετͷཧͱ͔Βͳͯ͘ྑ͘ͳΔͱخ͍͠
͝ਗ਼ௌ ͋Γ͕ͱ͏͍͟͝·ͨ͠ :bow Α͚Εձࣾհޚཡ͍ͩ͘͞
גࣜձࣾ primeNumber • 201511݄ ઃཱ • ࣾһ9໊ʢ͏ͪΤϯδχΞ8໊ʣ ύʔτφʔΤϯδχΞ30໊Ҏ্ • 201811݄
trocco®ϦϦʔε • 20197݄ γϦʔζAࢿۚௐୡʢ1.3ԯԁʣ࣮ࢪ https://primenumber.co.jp
ಇ͘ڥ • @தࠇͷԊ͍ • ίΞλΠϜ੍ • 11ʙ15࣌ • िதɺ2·ͰϦϞʔτՄ ※ࢼ༻த
• δϜશֹิॿ
We are hiring • ιϑτΣΞΤϯδχΞ • ΠϯϑϥΤϯδχΞ • ͳͲͳͲɺଟํ໘ઈࢍืूத