Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kubernetes で解決したこと新たに出てきた課題 /#jtf2017
Search
Kodai Sakabe
August 27, 2017
Technology
7
7.6k
Kubernetes で解決したこと新たに出てきた課題 /#jtf2017
1年を通して Kubernetes で解決したこと新たに出てきた課題をどう立ち向かったを発表しました
at July Tech Festa 2017
Kodai Sakabe
August 27, 2017
Tweet
Share
More Decks by Kodai Sakabe
See All by Kodai Sakabe
Terraform のための AVM を使った IaC のモジュール戦略 / Azure Verified Modules for Terraform
koudaiii
0
120
Well-Architected Framework オペレーションエクセレンス
koudaiii
0
70
AIアプリを支えるチームの最適な開発環境を整備する / / Optimizing AI Development Environment with Platform/DevOps Engineers
koudaiii
0
170
第4回:Boundaryを使用して動的に作成されたAzureリソースにアクセスする / Using Boundary to access dynamically created azure
koudaiii
0
120
[INF-4] 改めて学ぶ Immutable Infrastructure Golden Image編
koudaiii
0
55
Microsoft Entra External ID / FUJIYAMA Tech Showcase #fujishow
koudaiii
1
270
第3回PackerでAzureにおけるイメージコンプライアンスを実現する / HCP Packer + GitHub Actions on Azure
koudaiii
1
200
Azure OpenAI Service リファレンスアーキテクチャからみる本番システムレベルの LLM アプリに必要な検討項目の解説 / From Azure OpenAI Reference Architecture to Production-Ready LLM Apps #serverlessdays #serverlesstokyo
koudaiii
11
4.1k
about Azure SREインフラ分科会 / Azure SRE and Infrastructure SIG
koudaiii
0
110
Other Decks in Technology
See All in Technology
ソフトウェアエンジニアと仕事するときに知っておいたほうが良いこと / Key points for working with software engineers
pinkumohikan
1
140
わたしのOSS活動
kazupon
2
340
AIエージェント入門
minorun365
PRO
30
16k
Raycast AI APIを使ってちょっと便利な拡張機能を作ってみた / created-a-handy-extension-using-the-raycast-ai-api
kawamataryo
0
210
Pwned Labsのすゝめ
ken5scal
1
310
JEDAI Meetup! Databricks AI/BI概要
databricksjapan
0
310
OSS構成管理ツールCMDBuildを使ったAWSリソース管理の自動化
satorufunai
0
570
短縮URLをお手軽に導入しよう
nakasho
0
140
Oracle Database Technology Night #87-1 : Exadata Database Service on Exascale Infrastructure(ExaDB-XS)サービス詳細
oracle4engineer
PRO
1
120
php-conference-nagoya-2025
fuwasegu
0
140
遷移の高速化 ヤフートップの試行錯誤
narirou
6
1k
Raycast Favorites × Script Command で実現するお手軽情報チェック
smasato
1
140
Featured
See All Featured
Visualization
eitanlees
146
15k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Rebuilding a faster, lazier Slack
samanthasiow
80
8.9k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.1k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
7
640
Building an army of robots
kneath
303
45k
Designing for Performance
lara
604
68k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
YesSQL, Process and Tooling at Scale
rocio
172
14k
It's Worth the Effort
3n
184
28k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.3k
[RailsConf 2023] Rails as a piece of cake
palkan
53
5.3k
Transcript
,PEBJ4BLBCF!LPVEBJJJ8BOUFEMZ *OD Kubernetes Ͱղܾͨ͜͠ͱ ৽ͨʹग़͖ͯͨ՝ July Tech Festa 2017
ABOUT ME — Software Engineer — 2010/4- TIS ࡔ෦ େ
(KODAI SAKABE) 2015/8- Wantedly @koudaiii Site: https://koudaiii.com
None
Kubernetes ෳͷϗετؒͰίϯςφԽ͞ΕͨΞϓϦέʔγϣϯΛཧ͢ΔͨΊͷOSS ΞϓϦέʔγϣϯͷdeployment, maintenance, scalingͷͨΊͷجຊతͳϝΧχζϜΛఏڙ https://github.com/kubernetes/kubernetes
Kubernetes Λར༻ͨ݁͠Ռ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞ • ՝ʹͳͬͨ͜ͱ
• खಈͰ͖ͬͯͨ͜ͱͷݶք • Kubernetes ͷ Master ͕ SPOF • ϞχλϦϯάͷෳࡶ͞
Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
ΞδΣϯμ ڈ·ͰͷৼΓฦΓ ࣮ફ Kubernetes ղܾͨ͜͠ͱ / ՝ͷऔΓΈ τΠϧ໓ӡಈ The UNIX
Philosophy ϚϧνϚελʔ ࣗಈϞχλϦϯά
ڈ·ͰͷৼΓฦΓ
͜Ε·ͰͷऔΓΈ • Docker / Chef / Packer • Blue-Green Deployment
• Terraform • CoreOS • cell
Docker / Chef / Packer • Docker image ͷதʹ Chef
ΛͬͯΠϯϑϥΛߏங • Packer Ͱ AWS ͷ instance ͷ AMI Λ࡞Δ • Base ͱͳΔ template Λ༻ҙͯ͠ɺ৽͍͠αʔϏε͕ग़͖ͯͯྲྀ༻Ͱ ͖ΔΑ͏ʹͨ͠ • ΞϓϦέʔγϣϯͷߏங෦Λ Code Խ
Blue-Green Deployment • /healthcheck ͰϨεϙϯεΛฦͨ͠ΒΓସ͑Δ UBHT OBNFBQQ BQQSBJMT DPMPSHSFFO UBHT
OBNFBQQ BQQSBJMT DPMPSCMVF DBQEFQMPZTXJUDI@DPMPSCMVF QPSU QPSU
Terraform • AWS(S3/RDS/ELB etc) / dnsimple Λૢ࡞ IUUQTTQFBLFSEFDLDPNEUBOUFSSBGPSNBUXBOUFEMZUFDIDJSDMFOVNCFS
TAG ͱ AutoScaleGroup(ASG) • instance ͷׂΛɺ EC2 ͷ TAG Λͬͯཧ
• Կಈ͔͍͔ͨ͠ʁ => ASGͰઃఆ • ΠϯϑϥπʔϧΛ LaunchConfig ʹఆٛ • instance ্ཱ͕͕ͪΔͱ࣌ʹࣗʹ͚ΒΕͨ TAG Λ؍ͯɺඞཁͳ service Λ systemd ʹө͠ɺαʔϏεΛىಈ(cell)
cell • AWS TAG Λݩʹ systemd ʹαʔϏεΛొ • αʔϏεͷ status
͕ ok ʹͳͬͨΒ DNS ࣗΛՃ
ٕज़ελοΫ • ৽نαʔϏεΛ্ཱͪ͛Δࡍ͜ͷٕज़ ελοΫΛશ෦༻ҙ͢Δ • Πϯϑϥͷ҆ఆԽͷͨΊͷ༷ʑπʔϧͱ ΈΛ༻ҙ • αʔόʔΛߏங͢Δͱ͜Ζ͔ΒσϓϩΠɺ cell
·ͰΛίʔυԽ Web Application Monitoring Logging Auto Scale Load Balancer Internal DNS
ղͳखॱ
࣮ફ Kubernetes
Kubernetes ಋೖ αʔϏε͕Ͱ͖ΔʹҰݸͣͭαʔόʔΛ༻ҙͯ͠ߏங͢Δํ๏ ෳͷαʔόʔΛҰͭͷେ͖ͳαʔόʔʹݟཱͯͯར༻͢Δ ಉ͡Α͏ͳΠϯϑϥ࡞ۀΛݮΒ͠ɺϦιʔεΛޮతʹར༻͢Δ ࢀߟ: CodeZineʮKubernetesΛͬͨมԽʹڧ͍Πϯϑϥ――WantedlyͷΠϯϑϥνʔϜ͕େʹ͍ͯ͠Δ͜ ͱʯhttps://codezine.jp/article/detail/10357
The Twelve Factor App •12 ݸͷΞϓϦέʔγϣϯͷํ๏ https://12factor.net/ja/ •Herokuࣄଶίϯςφٕज़ɻDocker Λӡ༻͍ͯ͘͠࠷దͳํ๏ •σʔλετΞΛίϯςφʹؚΊͳ͍
•։ൃڥͱຊ൪ڥͷ ࠩҟΛڥมͰઃఆ •ܧଓతσϓϩΠ
ͦͷଞՃϧʔϧ Docker ͷ image ܰ͘ CI Λ௨ͯ͠ɺimage ࡞ɺdeploy Λߦͳ͏ ϦϙδτϦͱ
namespace ͕ 1ର1 ʹͳΔΑ͏ʹ
docker image ܰ͘ https://github.com/koudaiii/sltd/blob/master/Dockerfile Golang Ͱ͋ΕόΠφϦΛΆΜى͖ͩͱখ͍͞ 1 FROM alpine:3.6 2
3 RUN apk add --no-cache --update ca-certificates 4 5 COPY bin/sltd /sltd 6 7 ENTRYPOINT ["/sltd"] 8 CMD ["help"]
CI/CD .travis.yml ΑΓൈਮ 58 script: 59 - ./script/ci-test # Test
60 - ./script/ci-build # docker build 61 - ./script/ci-push # docker push 62 ɾɾɾɾɾɾɾɾɾɾɾ 69 deploy: 70 skip_cleanup: true 71 provider: script 72 script: ./script/ci-deploy # Release 73 on: 74 all_branches: true
None
ϦϙδτϦ = namespace ֤ϦϙδτϦʹ /kubernetes Λஔ $ tree kubernetes/ kubernetes/
ᵓᴷᴷ jobs ᴹ ᵓᴷᴷ create-blog-job.yaml ᴹ ᵓᴷᴷ destroy-blog-job.yaml ᴹ ᵋᴷᴷ update-monthly-ranking-job.yaml ᵓᴷᴷ namespace.yaml ᵓᴷᴷ rails-docker-sample-run.yaml ᵓᴷᴷ rails-docker-sample-hpa.yaml ᵋᴷᴷ rails-docker-sample-svc.yaml
ϚχϑΣετϑΝΠϧ Namespace ઐ༻ͷ໊લۭؒ Deployment ΞϓϦέʔγϣϯͷ࣮ߦ HorizontalPodAutoscaler cpu/memory ͔Β pod Φʔτεέʔϧ
Pod one-off ίϯςφ༻(rails c Λߦͳ͏༻) Service SSL͖ELB Job oneshot Ͱ࣮ߦ༻(rake db:migrate ) CronJob Cron ܗࣜͰλεΫΛ࣮ߦ(rake task )
Namespace ઐ༻ͷ໊લۭؒΛ࡞ 1 apiVersion: v1 2 kind: Namespace 3 metadata:
4 name: hoge
Deployment 1 apiVersion: extensions/v1beta1 2 kind: Deployment ɾɾɾɾɾɾɾɾɾɾɾ 9 spec:
11 strategy: 12 type: RollingUpdate 13 rollingUpdate: 14 maxSurge: 50% # ҰೖΕସ͑ 15 maxUnavailable: 0 # replicas Λҡ࣋ ɾɾɾɾɾɾɾɾɾɾɾ 23 spec: 24 containers: 25 - image: wantedly/python:latest 26 name: python 27 ports: 28 - containerPort: 8000 29 readinessProbe: # ready ͷఆٛ 30 httpGet: 31 path: /ping 32 port: 8000 33 initialDelaySeconds: 10 34 timeoutSeconds: 1 35 envFrom: 36 - secretRef: 37 name: dotenv 38 command: ["script/server"]
HorizontalPodAutoscaler Pod ͷ autoscale 1 apiVersion: autoscaling/v1 2 kind: HorizontalPodAutoscaler
3 metadata: 4 name: hoge 5 namespace: hoge 6 spec: 7 maxReplicas: 30 8 minReplicas: 1 9 scaleTargetRef: 10 apiVersion: extensions/v1beta1 11 kind: Deployment 12 name: python 13 targetCPUUtilizationPercentage: 50
Pod one-off Container(ྫ: rails c) 1 apiVersion: v1 2 kind:
Pod 3 metadata: 4 name: {USER} 5 namespace: hoge 6 labels: 7 role: console 8 spec: ɾɾɾɾɾɾɾɾɾɾɾ 11 containers: 12 - image: wantedly/bash:latest 13 imagePullPolicy: Always 14 name: {USER} 15 command: 16 - bash 17 stdin: true 18 stdinOnce: true 19 terminationMessagePath: /dev/termination-log 20 tty: true 21 envFrom: 22 - secretRef: 23 name: dotenv
Service SSLূ໌ॻ͖ ELB 1 apiVersion: v1 2 kind: Service 3
metadata: 4 name: hoge 6 annotations: 7 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:reagion:x:certificate/x 8 service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http 9 labels: 10 name: hoge 11 role: web 12 spec: 13 ports: 14 - port: 443 15 protocol: TCP 16 targetPort: 8000 17 selector: 18 name: hoge 19 role: web 20 type: LoadBalancer
Job (ྫ rake db:migrate) 1 apiVersion: batch/v1 2 kind: Job
ɾɾɾɾɾɾɾɾɾɾɾ 16 spec: 17 restartPolicy: Never 18 containers: 19 - name: db-migrate-[REPLACE_WITH_DATETIME] 20 image: wantedly/rails:[REPLACE_WITH_TAG] 21 command: ["bundle", "exec", "rake", "db:migrate"] 22 envFrom: 23 - secretRef: 24 name: dotenv
CronJob (ྫ rake task) concurrencyPolicy Ͱલ࣮ߦ͕ͬͨ߹ͷϋϯυϦϯά͕ग़དྷΔ 1 apiVersion: batch/v2alpha1 2
kind: CronJob 3 metadata: 4 namespace: hoge 5 name: create-post 6 labels: 7 job: create-post 9 spec: 10 schedule: "*/30 * * * *" 11 concurrencyPolicy: “Replace" # “Allow” or “Forbid” 12 successfulJobsHistoryLimit: 5 13 failedJobsHistoryLimit: 5 ɾɾɾɾɾɾɾɾɾɾɾ
ղܾͨ͜͠ͱ / ՝ͷऔΓΈ
ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞
৽نαʔϏεͷग़͢͠͞ ͜Ε·Ͱ ݱࡏ
εέʔϧͷ͢͠͞ •͜Ε·Ͱ instance ϨϕϧͰɺεέʔϧΛߦͳ͏ΈΛ࡞Δඞཁ͕͋ͬͨ •ϔϧενΣοΫ •αʔϏεσΟεΧόϦ •DNSొ / Load Balancer
•ࣗݾम෮ •Kubernetes ͔ΒɺԼهͷ͔Β༰қʹͳͬͨ •αʔϏεσΟεΧόϦͱDNSɺLoad Balancer ྑ͠ͳʹ͞Ε͍ͯΔ •ίϯςφϨϕϧͷεέʔϧग़དྷΔ $ kubectl autoscale deployment foo --min=2 —-max=10 —cpu-percent=50 # શମͷCPUUtilization 50% ᮢͰ૿ݮ $ kubectl scale --replicas=3 deployment foo # ͦͷͰ3ͭʹ͢Δ
՝ʹͳͬͨ͜ͱ 1. खಈͰ͖ͬͯͨ͜ͱͷݶք 2. Kubernetes ͷ Master ͕ SPOF 3.
ϞχλϦϯάͷෳࡶ͞
՝ͷऔΓΈ 1. τΠϧ໓ӡಈ 2. ϚϧνϚελʔ 3. ࣗಈϞχλϦϯά
“τΠϧͱɺϓϩμΫγϣϯαʔϏεΛಈ࡞ͤ͞Δ͜ͱʹ ؔ͢Δ࡞ۀͰɺख࡞ۀͰ܁Γฦ͠ߦΘΕɺࣗಈԽ͢Δ͜ͱ ͕ՄೳͰ͋Γɺઓज़తͰظతͳՁΛ࣋ͨͣɺ࡞ۀྔ͕αʔ Ϗεͷʹൺྫ͢Δͱ͍ͬͨΛ࣋ͭͷͰ͢ɻ” ൈਮɿ: Betsy Beyer “SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά”
ҰͭҎ্ͯ·ΔͱτΠϧͷՄೳੑ༗ • ख࡞ۀͰ͋Δ͜ͱ • ܁Γฦ͞ΕΔ͜ͱ • ࣗಈԽग़དྷΔ͜ͱ • ઓज़తͰ͋Δ͜ͱ(ઓུతͰ͋ͬͨΓ༧ଌʹجͮ͘ͷͰͳ͘ɺׂΓࠐΈͰ࢝·ΓɺͳͲ ͕ੜͨ͜͡ͱͷରԠ)
• ظతͳՁΛ࣋ͨͳ͍͜ͱ • αʔϏεͷʹରͯ͠O(n)Ͱ͋Δ͜ͱ
τΠϧ໓ӡಈ •test-build-push-release => CI ͷ template Խ •autoscale => HorizontalPodAutoscaler
ϚχϑΣετΛॻ͘ •migrate => Job ϚχϑΣετΛॻ͘ •one-off => ઐ༻ͷ Pod ϚχϑΣετΛॻ͘ •SSL ͷखܰ͞ => Service ϚχϑΣετʹ annotation Λه͢Δͱઃఆ͞ΕΔ •deploy & rollback => The UNIX Philosophy ଇͬͨπʔϧ࡞ ࣮΄΅,VCFSOFUFTͰ࣮ݱ
The UNIX Philosophy
The UNIX Philosophy ൈਮ • Small is beautiful. খ͍͞ͷඒ͍͠ •
Make each program do one thing well. 1ͭͷϓϩάϥϜʹ1ͭͷ͜ͱΛ͏·͘ ΒͤΔ • Choose portability over efficiency. ޮΑΓҠ২ੑΛ༏ઌ͢Δ • Use software leverage to your advantage. ιϑτΣΞΛᑏࢠ(ͯ͜)ͱͯ͠͏ • Avoid captive user interfaces. աͷରతΠϯλʔϑΣʔεΛආ͚Δ
Server Tools ͱ Client Tool • Ұͭͷ͜ͱΛ্ख͘Δπʔϧ࡞Γɺᑏࢠͱͯ͠ར༻ => ͍ͭͰަՄೳ •
ΫϥΠΞϯτଆ׳Ε͠Μͩύοέʔδཧ => homebrew Ͱ৴ • ΫϥΠΞϯτଆπʔϧΛݺͼग़ͨ͢Ίͷϥούʔʹઐ೦ • ΫϥΠΞϯτଆʹग़དྷΔݶΓϩδοΫΛೖΕͳ͍ => ॲཧ server ্ • ωοτஅΛఆ͢Δ => server ͱ container ҡ࣋
homebrew Golang πʔϧΛ homebrew Ͱ binary Λ৴ ΫϩείϯύΠϥͰOSґଘ͕ۃྗͳ͘πʔϧ͕͑ΔΑ͏ʹ
kube ࣾπʔϧ
खಈσϓϩΠ ϚχϑΣετϑΝΠϧʹ annotations Λه͢Δ͜ͱͰखಈσϓϩΠ͕Ͱ͖Δ k8ship deploy [BRANCH|COMMIT_SHA1] 1 apiVersion: extensions/v1beta1
2 kind: Deployment 3 metadata: 4 name: rails 5 namespace: rails 6 labels: 7 name: rails 8 role: web 9 annotations: 10 wantedly.com/deploy-target: “true" # ه 11 wantedly.com/deploy-target-container: rails # ه 12 wantedly.com/github: rails=wantedly/rails # ه https://github.com/dtan4/k8ship
kube deploy ίϚϯυ -dry-run ΦϓγϣϯͰ diff Λग़͢ merge ޙҙͷλΠϛϯάͰ deploy
͍ͨ͠߹ʹར༻ $ kube qa deploy master --dry-run [dry-run] deploy to (deployment: "rails", container: "rails") [dry-run] before: wantedly/rails:d1cb608ee61cb18f8c397c2e27576573879b2fcf [dry-run] after: wantedly/rails:cb19269526f946222110fbdfb3d107c4b5a18fe7
ϚϧνϚελʔ
ϚϧνϚελʔ •Kubernetes ͷ version up Ͳ͏͢Δͷʁ •Ϛελʔ͕ࢭ·ΔͱͲ͏ͳΔͷʁ •Ϋϥελͷఀࢭͳ͘ kops Λͬͯ
Rolling upgrade Ͱ͖ΔΑ͏ʹมߋ •ϚελʔΛࡾ༻ҙ͠ɺϚϧνϚελʔߏʹ͔͑ͯ SPOF վળ
ࣗಈϞχλϦϯά
ࣗಈϞχλϦϯά • 1ͭ1ͭઃఆ͢ΔͷͰͳ͘ɺ൚༻తʹద༻͞ΕΔΑ͏ʹ͢Δ • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}}
• request ͱ status code औಘ • μογϡϘʔυʹө • Deployment Ͱఆٛͨ͠
ࣗಈͰಉ࣭͡ͷϞχλϦϯάΛࢦ͢ https://www.datadoghq.com/blog/monitoring-101-collecting-data/ http://qiita.com/koudaiii/items/bc89368e1279649f2498
Dashboard
ϞχλϦϯάͰ༻͍ͯ͠Δͷ • kelseyhightower/konfd => PostgresqlͳͲͷใΛϦϙδτϦʹؚΊͳ͍࣌ʹར༻ • kubernetes/kube-state-metrics => deployment ͷ
available Λऔಘ • datadog/docker-dd-agent => docker ༻ɺ web ༻ɺ db ༻ͱׂͯ͠ผͰඋ • koudaiii/sltd => Service ͷ metadata Λݩʹ ELB ͷ tag ʹՃ • logentries => ΫϥελશͯͷϩάΛૹΔ
ਪҠ
αʔϏεਪҠ (2017/04Ἤ) Service 9 => 60 Container 700+ => 1,500
- 2,500+
·ͱΊ • kubernetes ɺΠϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬʹͳͬͨ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞
• ՝͔ΒऔΓΜͩ͜ͱ • खಈͰ͖ͬͯͨ͜ͱͷݶք => τΠϧ໓ӡಈ • Ϛελʔ͕ SPOF => ϚϧνϚελʔԽ • ϞχλϦϯάͷෳࡶ͞ => ࣗಈϞχλϦϯά
Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
IUUQTXXXXBOUFEMZDPNQSPKFDUT