Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kubernetes で解決したこと新たに出てきた課題 /#jtf2017
Search
Kodai Sakabe
August 27, 2017
Technology
7
7.7k
Kubernetes で解決したこと新たに出てきた課題 /#jtf2017
1年を通して Kubernetes で解決したこと新たに出てきた課題をどう立ち向かったを発表しました
at July Tech Festa 2017
Kodai Sakabe
August 27, 2017
Tweet
Share
More Decks by Kodai Sakabe
See All by Kodai Sakabe
無理しない AI 活用サービス / #jazug
koudaiii
1
140
Terraform のための AVM を使った IaC のモジュール戦略 / Azure Verified Modules for Terraform
koudaiii
0
160
Well-Architected Framework オペレーションエクセレンス
koudaiii
0
80
AIアプリを支えるチームの最適な開発環境を整備する / / Optimizing AI Development Environment with Platform/DevOps Engineers
koudaiii
0
200
第4回:Boundaryを使用して動的に作成されたAzureリソースにアクセスする / Using Boundary to access dynamically created azure
koudaiii
0
160
[INF-4] 改めて学ぶ Immutable Infrastructure Golden Image編
koudaiii
0
69
Microsoft Entra External ID / FUJIYAMA Tech Showcase #fujishow
koudaiii
1
330
第3回PackerでAzureにおけるイメージコンプライアンスを実現する / HCP Packer + GitHub Actions on Azure
koudaiii
1
240
Azure OpenAI Service リファレンスアーキテクチャからみる本番システムレベルの LLM アプリに必要な検討項目の解説 / From Azure OpenAI Reference Architecture to Production-Ready LLM Apps #serverlessdays #serverlesstokyo
koudaiii
11
4.3k
Other Decks in Technology
See All in Technology
FAST導入1年間のふりかえり〜現実を直視し、さらなる進化を求めて〜 / Review of the first year of FAST implementation
wooootack
1
180
AI工学特論: MLOps・継続的評価
asei
10
2k
Expertise as a Service via MCP
yodakeisuke
1
160
【CEDEC2025】大規模言語モデルを活用したゲーム内会話パートのスクリプト作成支援への取り組み
cygames
PRO
1
320
AWS表彰プログラムとキャリアについて
naoki_0531
1
140
複数のGemini CLIが同時開発する狂気 - Jujutsuが実現するAIエージェント協調の新世界
gunta
13
3.8k
東京海上日動におけるセキュアな開発プロセスの取り組み
miyabit
0
200
PdM業務における使い分け
shinshiro
0
670
Kiro Hookを Terraformで検証
ao_inoue
0
140
P2P ではじめる WebRTC のつまづきどころ
tnoho
1
270
大規模イベントを支える ABEMA の アーキテクチャ 変遷 2025
nagapad
5
510
Snowflake のアーキテクチャは本当に筋がよかったのか / Data Engineering Study #30
indigo13love
0
280
Featured
See All Featured
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
47
9.6k
Mobile First: as difficult as doing things right
swwweet
223
9.7k
Adopting Sorbet at Scale
ufuk
77
9.5k
Reflections from 52 weeks, 52 projects
jeffersonlam
351
21k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
760
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
8
720
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.3k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
5.9k
It's Worth the Effort
3n
185
28k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
110
19k
Transcript
,PEBJ4BLBCF!LPVEBJJJ8BOUFEMZ *OD Kubernetes Ͱղܾͨ͜͠ͱ ৽ͨʹग़͖ͯͨ՝ July Tech Festa 2017
ABOUT ME — Software Engineer — 2010/4- TIS ࡔ෦ େ
(KODAI SAKABE) 2015/8- Wantedly @koudaiii Site: https://koudaiii.com
None
Kubernetes ෳͷϗετؒͰίϯςφԽ͞ΕͨΞϓϦέʔγϣϯΛཧ͢ΔͨΊͷOSS ΞϓϦέʔγϣϯͷdeployment, maintenance, scalingͷͨΊͷجຊతͳϝΧχζϜΛఏڙ https://github.com/kubernetes/kubernetes
Kubernetes Λར༻ͨ݁͠Ռ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞ • ՝ʹͳͬͨ͜ͱ
• खಈͰ͖ͬͯͨ͜ͱͷݶք • Kubernetes ͷ Master ͕ SPOF • ϞχλϦϯάͷෳࡶ͞
Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
ΞδΣϯμ ڈ·ͰͷৼΓฦΓ ࣮ફ Kubernetes ղܾͨ͜͠ͱ / ՝ͷऔΓΈ τΠϧ໓ӡಈ The UNIX
Philosophy ϚϧνϚελʔ ࣗಈϞχλϦϯά
ڈ·ͰͷৼΓฦΓ
͜Ε·ͰͷऔΓΈ • Docker / Chef / Packer • Blue-Green Deployment
• Terraform • CoreOS • cell
Docker / Chef / Packer • Docker image ͷதʹ Chef
ΛͬͯΠϯϑϥΛߏங • Packer Ͱ AWS ͷ instance ͷ AMI Λ࡞Δ • Base ͱͳΔ template Λ༻ҙͯ͠ɺ৽͍͠αʔϏε͕ग़͖ͯͯྲྀ༻Ͱ ͖ΔΑ͏ʹͨ͠ • ΞϓϦέʔγϣϯͷߏங෦Λ Code Խ
Blue-Green Deployment • /healthcheck ͰϨεϙϯεΛฦͨ͠ΒΓସ͑Δ UBHT OBNFBQQ BQQSBJMT DPMPSHSFFO UBHT
OBNFBQQ BQQSBJMT DPMPSCMVF DBQEFQMPZTXJUDI@DPMPSCMVF QPSU QPSU
Terraform • AWS(S3/RDS/ELB etc) / dnsimple Λૢ࡞ IUUQTTQFBLFSEFDLDPNEUBOUFSSBGPSNBUXBOUFEMZUFDIDJSDMFOVNCFS
TAG ͱ AutoScaleGroup(ASG) • instance ͷׂΛɺ EC2 ͷ TAG Λͬͯཧ
• Կಈ͔͍͔ͨ͠ʁ => ASGͰઃఆ • ΠϯϑϥπʔϧΛ LaunchConfig ʹఆٛ • instance ্ཱ͕͕ͪΔͱ࣌ʹࣗʹ͚ΒΕͨ TAG Λ؍ͯɺඞཁͳ service Λ systemd ʹө͠ɺαʔϏεΛىಈ(cell)
cell • AWS TAG Λݩʹ systemd ʹαʔϏεΛొ • αʔϏεͷ status
͕ ok ʹͳͬͨΒ DNS ࣗΛՃ
ٕज़ελοΫ • ৽نαʔϏεΛ্ཱͪ͛Δࡍ͜ͷٕज़ ελοΫΛશ෦༻ҙ͢Δ • Πϯϑϥͷ҆ఆԽͷͨΊͷ༷ʑπʔϧͱ ΈΛ༻ҙ • αʔόʔΛߏங͢Δͱ͜Ζ͔ΒσϓϩΠɺ cell
·ͰΛίʔυԽ Web Application Monitoring Logging Auto Scale Load Balancer Internal DNS
ղͳखॱ
࣮ફ Kubernetes
Kubernetes ಋೖ αʔϏε͕Ͱ͖ΔʹҰݸͣͭαʔόʔΛ༻ҙͯ͠ߏங͢Δํ๏ ෳͷαʔόʔΛҰͭͷେ͖ͳαʔόʔʹݟཱͯͯར༻͢Δ ಉ͡Α͏ͳΠϯϑϥ࡞ۀΛݮΒ͠ɺϦιʔεΛޮతʹར༻͢Δ ࢀߟ: CodeZineʮKubernetesΛͬͨมԽʹڧ͍Πϯϑϥ――WantedlyͷΠϯϑϥνʔϜ͕େʹ͍ͯ͠Δ͜ ͱʯhttps://codezine.jp/article/detail/10357
The Twelve Factor App •12 ݸͷΞϓϦέʔγϣϯͷํ๏ https://12factor.net/ja/ •Herokuࣄଶίϯςφٕज़ɻDocker Λӡ༻͍ͯ͘͠࠷దͳํ๏ •σʔλετΞΛίϯςφʹؚΊͳ͍
•։ൃڥͱຊ൪ڥͷ ࠩҟΛڥมͰઃఆ •ܧଓతσϓϩΠ
ͦͷଞՃϧʔϧ Docker ͷ image ܰ͘ CI Λ௨ͯ͠ɺimage ࡞ɺdeploy Λߦͳ͏ ϦϙδτϦͱ
namespace ͕ 1ର1 ʹͳΔΑ͏ʹ
docker image ܰ͘ https://github.com/koudaiii/sltd/blob/master/Dockerfile Golang Ͱ͋ΕόΠφϦΛΆΜى͖ͩͱখ͍͞ 1 FROM alpine:3.6 2
3 RUN apk add --no-cache --update ca-certificates 4 5 COPY bin/sltd /sltd 6 7 ENTRYPOINT ["/sltd"] 8 CMD ["help"]
CI/CD .travis.yml ΑΓൈਮ 58 script: 59 - ./script/ci-test # Test
60 - ./script/ci-build # docker build 61 - ./script/ci-push # docker push 62 ɾɾɾɾɾɾɾɾɾɾɾ 69 deploy: 70 skip_cleanup: true 71 provider: script 72 script: ./script/ci-deploy # Release 73 on: 74 all_branches: true
None
ϦϙδτϦ = namespace ֤ϦϙδτϦʹ /kubernetes Λஔ $ tree kubernetes/ kubernetes/
ᵓᴷᴷ jobs ᴹ ᵓᴷᴷ create-blog-job.yaml ᴹ ᵓᴷᴷ destroy-blog-job.yaml ᴹ ᵋᴷᴷ update-monthly-ranking-job.yaml ᵓᴷᴷ namespace.yaml ᵓᴷᴷ rails-docker-sample-run.yaml ᵓᴷᴷ rails-docker-sample-hpa.yaml ᵋᴷᴷ rails-docker-sample-svc.yaml
ϚχϑΣετϑΝΠϧ Namespace ઐ༻ͷ໊લۭؒ Deployment ΞϓϦέʔγϣϯͷ࣮ߦ HorizontalPodAutoscaler cpu/memory ͔Β pod Φʔτεέʔϧ
Pod one-off ίϯςφ༻(rails c Λߦͳ͏༻) Service SSL͖ELB Job oneshot Ͱ࣮ߦ༻(rake db:migrate ) CronJob Cron ܗࣜͰλεΫΛ࣮ߦ(rake task )
Namespace ઐ༻ͷ໊લۭؒΛ࡞ 1 apiVersion: v1 2 kind: Namespace 3 metadata:
4 name: hoge
Deployment 1 apiVersion: extensions/v1beta1 2 kind: Deployment ɾɾɾɾɾɾɾɾɾɾɾ 9 spec:
11 strategy: 12 type: RollingUpdate 13 rollingUpdate: 14 maxSurge: 50% # ҰೖΕସ͑ 15 maxUnavailable: 0 # replicas Λҡ࣋ ɾɾɾɾɾɾɾɾɾɾɾ 23 spec: 24 containers: 25 - image: wantedly/python:latest 26 name: python 27 ports: 28 - containerPort: 8000 29 readinessProbe: # ready ͷఆٛ 30 httpGet: 31 path: /ping 32 port: 8000 33 initialDelaySeconds: 10 34 timeoutSeconds: 1 35 envFrom: 36 - secretRef: 37 name: dotenv 38 command: ["script/server"]
HorizontalPodAutoscaler Pod ͷ autoscale 1 apiVersion: autoscaling/v1 2 kind: HorizontalPodAutoscaler
3 metadata: 4 name: hoge 5 namespace: hoge 6 spec: 7 maxReplicas: 30 8 minReplicas: 1 9 scaleTargetRef: 10 apiVersion: extensions/v1beta1 11 kind: Deployment 12 name: python 13 targetCPUUtilizationPercentage: 50
Pod one-off Container(ྫ: rails c) 1 apiVersion: v1 2 kind:
Pod 3 metadata: 4 name: {USER} 5 namespace: hoge 6 labels: 7 role: console 8 spec: ɾɾɾɾɾɾɾɾɾɾɾ 11 containers: 12 - image: wantedly/bash:latest 13 imagePullPolicy: Always 14 name: {USER} 15 command: 16 - bash 17 stdin: true 18 stdinOnce: true 19 terminationMessagePath: /dev/termination-log 20 tty: true 21 envFrom: 22 - secretRef: 23 name: dotenv
Service SSLূ໌ॻ͖ ELB 1 apiVersion: v1 2 kind: Service 3
metadata: 4 name: hoge 6 annotations: 7 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:reagion:x:certificate/x 8 service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http 9 labels: 10 name: hoge 11 role: web 12 spec: 13 ports: 14 - port: 443 15 protocol: TCP 16 targetPort: 8000 17 selector: 18 name: hoge 19 role: web 20 type: LoadBalancer
Job (ྫ rake db:migrate) 1 apiVersion: batch/v1 2 kind: Job
ɾɾɾɾɾɾɾɾɾɾɾ 16 spec: 17 restartPolicy: Never 18 containers: 19 - name: db-migrate-[REPLACE_WITH_DATETIME] 20 image: wantedly/rails:[REPLACE_WITH_TAG] 21 command: ["bundle", "exec", "rake", "db:migrate"] 22 envFrom: 23 - secretRef: 24 name: dotenv
CronJob (ྫ rake task) concurrencyPolicy Ͱલ࣮ߦ͕ͬͨ߹ͷϋϯυϦϯά͕ग़དྷΔ 1 apiVersion: batch/v2alpha1 2
kind: CronJob 3 metadata: 4 namespace: hoge 5 name: create-post 6 labels: 7 job: create-post 9 spec: 10 schedule: "*/30 * * * *" 11 concurrencyPolicy: “Replace" # “Allow” or “Forbid” 12 successfulJobsHistoryLimit: 5 13 failedJobsHistoryLimit: 5 ɾɾɾɾɾɾɾɾɾɾɾ
ղܾͨ͜͠ͱ / ՝ͷऔΓΈ
ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞
৽نαʔϏεͷग़͢͠͞ ͜Ε·Ͱ ݱࡏ
εέʔϧͷ͢͠͞ •͜Ε·Ͱ instance ϨϕϧͰɺεέʔϧΛߦͳ͏ΈΛ࡞Δඞཁ͕͋ͬͨ •ϔϧενΣοΫ •αʔϏεσΟεΧόϦ •DNSొ / Load Balancer
•ࣗݾम෮ •Kubernetes ͔ΒɺԼهͷ͔Β༰қʹͳͬͨ •αʔϏεσΟεΧόϦͱDNSɺLoad Balancer ྑ͠ͳʹ͞Ε͍ͯΔ •ίϯςφϨϕϧͷεέʔϧग़དྷΔ $ kubectl autoscale deployment foo --min=2 —-max=10 —cpu-percent=50 # શମͷCPUUtilization 50% ᮢͰ૿ݮ $ kubectl scale --replicas=3 deployment foo # ͦͷͰ3ͭʹ͢Δ
՝ʹͳͬͨ͜ͱ 1. खಈͰ͖ͬͯͨ͜ͱͷݶք 2. Kubernetes ͷ Master ͕ SPOF 3.
ϞχλϦϯάͷෳࡶ͞
՝ͷऔΓΈ 1. τΠϧ໓ӡಈ 2. ϚϧνϚελʔ 3. ࣗಈϞχλϦϯά
“τΠϧͱɺϓϩμΫγϣϯαʔϏεΛಈ࡞ͤ͞Δ͜ͱʹ ؔ͢Δ࡞ۀͰɺख࡞ۀͰ܁Γฦ͠ߦΘΕɺࣗಈԽ͢Δ͜ͱ ͕ՄೳͰ͋Γɺઓज़తͰظతͳՁΛ࣋ͨͣɺ࡞ۀྔ͕αʔ Ϗεͷʹൺྫ͢Δͱ͍ͬͨΛ࣋ͭͷͰ͢ɻ” ൈਮɿ: Betsy Beyer “SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά”
ҰͭҎ্ͯ·ΔͱτΠϧͷՄೳੑ༗ • ख࡞ۀͰ͋Δ͜ͱ • ܁Γฦ͞ΕΔ͜ͱ • ࣗಈԽग़དྷΔ͜ͱ • ઓज़తͰ͋Δ͜ͱ(ઓུతͰ͋ͬͨΓ༧ଌʹجͮ͘ͷͰͳ͘ɺׂΓࠐΈͰ࢝·ΓɺͳͲ ͕ੜͨ͜͡ͱͷରԠ)
• ظతͳՁΛ࣋ͨͳ͍͜ͱ • αʔϏεͷʹରͯ͠O(n)Ͱ͋Δ͜ͱ
τΠϧ໓ӡಈ •test-build-push-release => CI ͷ template Խ •autoscale => HorizontalPodAutoscaler
ϚχϑΣετΛॻ͘ •migrate => Job ϚχϑΣετΛॻ͘ •one-off => ઐ༻ͷ Pod ϚχϑΣετΛॻ͘ •SSL ͷखܰ͞ => Service ϚχϑΣετʹ annotation Λه͢Δͱઃఆ͞ΕΔ •deploy & rollback => The UNIX Philosophy ଇͬͨπʔϧ࡞ ࣮΄΅,VCFSOFUFTͰ࣮ݱ
The UNIX Philosophy
The UNIX Philosophy ൈਮ • Small is beautiful. খ͍͞ͷඒ͍͠ •
Make each program do one thing well. 1ͭͷϓϩάϥϜʹ1ͭͷ͜ͱΛ͏·͘ ΒͤΔ • Choose portability over efficiency. ޮΑΓҠ২ੑΛ༏ઌ͢Δ • Use software leverage to your advantage. ιϑτΣΞΛᑏࢠ(ͯ͜)ͱͯ͠͏ • Avoid captive user interfaces. աͷରతΠϯλʔϑΣʔεΛආ͚Δ
Server Tools ͱ Client Tool • Ұͭͷ͜ͱΛ্ख͘Δπʔϧ࡞Γɺᑏࢠͱͯ͠ར༻ => ͍ͭͰަՄೳ •
ΫϥΠΞϯτଆ׳Ε͠Μͩύοέʔδཧ => homebrew Ͱ৴ • ΫϥΠΞϯτଆπʔϧΛݺͼग़ͨ͢Ίͷϥούʔʹઐ೦ • ΫϥΠΞϯτଆʹग़དྷΔݶΓϩδοΫΛೖΕͳ͍ => ॲཧ server ্ • ωοτஅΛఆ͢Δ => server ͱ container ҡ࣋
homebrew Golang πʔϧΛ homebrew Ͱ binary Λ৴ ΫϩείϯύΠϥͰOSґଘ͕ۃྗͳ͘πʔϧ͕͑ΔΑ͏ʹ
kube ࣾπʔϧ
खಈσϓϩΠ ϚχϑΣετϑΝΠϧʹ annotations Λه͢Δ͜ͱͰखಈσϓϩΠ͕Ͱ͖Δ k8ship deploy [BRANCH|COMMIT_SHA1] 1 apiVersion: extensions/v1beta1
2 kind: Deployment 3 metadata: 4 name: rails 5 namespace: rails 6 labels: 7 name: rails 8 role: web 9 annotations: 10 wantedly.com/deploy-target: “true" # ه 11 wantedly.com/deploy-target-container: rails # ه 12 wantedly.com/github: rails=wantedly/rails # ه https://github.com/dtan4/k8ship
kube deploy ίϚϯυ -dry-run ΦϓγϣϯͰ diff Λग़͢ merge ޙҙͷλΠϛϯάͰ deploy
͍ͨ͠߹ʹར༻ $ kube qa deploy master --dry-run [dry-run] deploy to (deployment: "rails", container: "rails") [dry-run] before: wantedly/rails:d1cb608ee61cb18f8c397c2e27576573879b2fcf [dry-run] after: wantedly/rails:cb19269526f946222110fbdfb3d107c4b5a18fe7
ϚϧνϚελʔ
ϚϧνϚελʔ •Kubernetes ͷ version up Ͳ͏͢Δͷʁ •Ϛελʔ͕ࢭ·ΔͱͲ͏ͳΔͷʁ •Ϋϥελͷఀࢭͳ͘ kops Λͬͯ
Rolling upgrade Ͱ͖ΔΑ͏ʹมߋ •ϚελʔΛࡾ༻ҙ͠ɺϚϧνϚελʔߏʹ͔͑ͯ SPOF վળ
ࣗಈϞχλϦϯά
ࣗಈϞχλϦϯά • 1ͭ1ͭઃఆ͢ΔͷͰͳ͘ɺ൚༻తʹద༻͞ΕΔΑ͏ʹ͢Δ • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}}
• request ͱ status code औಘ • μογϡϘʔυʹө • Deployment Ͱఆٛͨ͠
ࣗಈͰಉ࣭͡ͷϞχλϦϯάΛࢦ͢ https://www.datadoghq.com/blog/monitoring-101-collecting-data/ http://qiita.com/koudaiii/items/bc89368e1279649f2498
Dashboard
ϞχλϦϯάͰ༻͍ͯ͠Δͷ • kelseyhightower/konfd => PostgresqlͳͲͷใΛϦϙδτϦʹؚΊͳ͍࣌ʹར༻ • kubernetes/kube-state-metrics => deployment ͷ
available Λऔಘ • datadog/docker-dd-agent => docker ༻ɺ web ༻ɺ db ༻ͱׂͯ͠ผͰඋ • koudaiii/sltd => Service ͷ metadata Λݩʹ ELB ͷ tag ʹՃ • logentries => ΫϥελશͯͷϩάΛૹΔ
ਪҠ
αʔϏεਪҠ (2017/04Ἤ) Service 9 => 60 Container 700+ => 1,500
- 2,500+
·ͱΊ • kubernetes ɺΠϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬʹͳͬͨ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞
• ՝͔ΒऔΓΜͩ͜ͱ • खಈͰ͖ͬͯͨ͜ͱͷݶք => τΠϧ໓ӡಈ • Ϛελʔ͕ SPOF => ϚϧνϚελʔԽ • ϞχλϦϯάͷෳࡶ͞ => ࣗಈϞχλϦϯά
Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
IUUQTXXXXBOUFEMZDPNQSPKFDUT