Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kubernetes で解決したこと新たに出てきた課題 /#jtf2017
Search
Kodai Sakabe
August 27, 2017
Technology
7
7.4k
Kubernetes で解決したこと新たに出てきた課題 /#jtf2017
1年を通して Kubernetes で解決したこと新たに出てきた課題をどう立ち向かったを発表しました
at July Tech Festa 2017
Kodai Sakabe
August 27, 2017
Tweet
Share
More Decks by Kodai Sakabe
See All by Kodai Sakabe
Well-Architected Framework オペレーションエクセレンス
koudaiii
0
57
AIアプリを支えるチームの最適な開発環境を整備する / / Optimizing AI Development Environment with Platform/DevOps Engineers
koudaiii
0
150
第4回:Boundaryを使用して動的に作成されたAzureリソースにアクセスする / Using Boundary to access dynamically created azure
koudaiii
0
89
[INF-4] 改めて学ぶ Immutable Infrastructure Golden Image編
koudaiii
0
37
Microsoft Entra External ID / FUJIYAMA Tech Showcase #fujishow
koudaiii
1
180
第3回PackerでAzureにおけるイメージコンプライアンスを実現する / HCP Packer + GitHub Actions on Azure
koudaiii
1
150
Azure OpenAI Service リファレンスアーキテクチャからみる本番システムレベルの LLM アプリに必要な検討項目の解説 / From Azure OpenAI Reference Architecture to Production-Ready LLM Apps #serverlessdays #serverlesstokyo
koudaiii
11
3.7k
about Azure SREインフラ分科会 / Azure SRE and Infrastructure SIG
koudaiii
0
96
その Kubernetes は 効率的にマシンを活用できているか #CNDT2020_RJKT #CNDT2020_RJKT_B / CNDT2020 Rejekts
koudaiii
5
1.2k
Other Decks in Technology
See All in Technology
小さな勉強会の始め方、広げ方、あるいは友達の作り方 / How to Start, Grow, and Build Connections with Small Study Groups
ar_tama
6
3k
トークナイザー入門
payanotty
2
1k
Amplify Gen 2ではじめる 生成AIアプリ開発入門
tsukuboshi
0
270
SageMaker学習のツボ / The Key Points of Learning SageMaker
cmhiranofumio
0
210
TypeScript x Raycast x AIで変える開発者体験
nagauta
1
280
入門 バックアップ
ryuichi1208
18
7.1k
これはPerl? それともRuby? クイズ〜〜〜〜〜!!!- Perl or Ruby Quiz
moznion
2
1.9k
テストを楽に書きたい
tomorrowkey
1
210
スタサプ ForSCHOOLアプリのシンプルな設計
recruitengineers
PRO
3
700
Product Utilization of Large Language Models Starting Today
ymatsuwitter
3
1.5k
AWSの初級者向けAI・ML資格『AWS Certified AI Practitioner』の傾向と対策/So You Want To Pass AWS Certified AI Practitioner
quiver
0
560
Grafana エコシステムの活用事例 on ABEMA
tetsuya28
4
430
Featured
See All Featured
Unsuck your backbone
ammeep
668
57k
Web development in the modern age
philhawksworth
205
10k
Being A Developer After 40
akosma
84
590k
Into the Great Unknown - MozCon
thekraken
31
1.4k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
25
1.7k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
How To Stay Up To Date on Web Technology
chriscoyier
787
250k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
26
4.1k
Learning to Love Humans: Emotional Interface Design
aarron
272
40k
Happy Clients
brianwarren
97
6.7k
Bash Introduction
62gerente
608
210k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.5k
Transcript
,PEBJ4BLBCF!LPVEBJJJ8BOUFEMZ *OD Kubernetes Ͱղܾͨ͜͠ͱ ৽ͨʹग़͖ͯͨ՝ July Tech Festa 2017
ABOUT ME — Software Engineer — 2010/4- TIS ࡔ෦ େ
(KODAI SAKABE) 2015/8- Wantedly @koudaiii Site: https://koudaiii.com
None
Kubernetes ෳͷϗετؒͰίϯςφԽ͞ΕͨΞϓϦέʔγϣϯΛཧ͢ΔͨΊͷOSS ΞϓϦέʔγϣϯͷdeployment, maintenance, scalingͷͨΊͷجຊతͳϝΧχζϜΛఏڙ https://github.com/kubernetes/kubernetes
Kubernetes Λར༻ͨ݁͠Ռ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞ • ՝ʹͳͬͨ͜ͱ
• खಈͰ͖ͬͯͨ͜ͱͷݶք • Kubernetes ͷ Master ͕ SPOF • ϞχλϦϯάͷෳࡶ͞
Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
ΞδΣϯμ ڈ·ͰͷৼΓฦΓ ࣮ફ Kubernetes ղܾͨ͜͠ͱ / ՝ͷऔΓΈ τΠϧ໓ӡಈ The UNIX
Philosophy ϚϧνϚελʔ ࣗಈϞχλϦϯά
ڈ·ͰͷৼΓฦΓ
͜Ε·ͰͷऔΓΈ • Docker / Chef / Packer • Blue-Green Deployment
• Terraform • CoreOS • cell
Docker / Chef / Packer • Docker image ͷதʹ Chef
ΛͬͯΠϯϑϥΛߏங • Packer Ͱ AWS ͷ instance ͷ AMI Λ࡞Δ • Base ͱͳΔ template Λ༻ҙͯ͠ɺ৽͍͠αʔϏε͕ग़͖ͯͯྲྀ༻Ͱ ͖ΔΑ͏ʹͨ͠ • ΞϓϦέʔγϣϯͷߏங෦Λ Code Խ
Blue-Green Deployment • /healthcheck ͰϨεϙϯεΛฦͨ͠ΒΓସ͑Δ UBHT OBNFBQQ BQQSBJMT DPMPSHSFFO UBHT
OBNFBQQ BQQSBJMT DPMPSCMVF DBQEFQMPZTXJUDI@DPMPSCMVF QPSU QPSU
Terraform • AWS(S3/RDS/ELB etc) / dnsimple Λૢ࡞ IUUQTTQFBLFSEFDLDPNEUBOUFSSBGPSNBUXBOUFEMZUFDIDJSDMFOVNCFS
TAG ͱ AutoScaleGroup(ASG) • instance ͷׂΛɺ EC2 ͷ TAG Λͬͯཧ
• Կಈ͔͍͔ͨ͠ʁ => ASGͰઃఆ • ΠϯϑϥπʔϧΛ LaunchConfig ʹఆٛ • instance ্ཱ͕͕ͪΔͱ࣌ʹࣗʹ͚ΒΕͨ TAG Λ؍ͯɺඞཁͳ service Λ systemd ʹө͠ɺαʔϏεΛىಈ(cell)
cell • AWS TAG Λݩʹ systemd ʹαʔϏεΛొ • αʔϏεͷ status
͕ ok ʹͳͬͨΒ DNS ࣗΛՃ
ٕज़ελοΫ • ৽نαʔϏεΛ্ཱͪ͛Δࡍ͜ͷٕज़ ελοΫΛશ෦༻ҙ͢Δ • Πϯϑϥͷ҆ఆԽͷͨΊͷ༷ʑπʔϧͱ ΈΛ༻ҙ • αʔόʔΛߏங͢Δͱ͜Ζ͔ΒσϓϩΠɺ cell
·ͰΛίʔυԽ Web Application Monitoring Logging Auto Scale Load Balancer Internal DNS
ղͳखॱ
࣮ફ Kubernetes
Kubernetes ಋೖ αʔϏε͕Ͱ͖ΔʹҰݸͣͭαʔόʔΛ༻ҙͯ͠ߏங͢Δํ๏ ෳͷαʔόʔΛҰͭͷେ͖ͳαʔόʔʹݟཱͯͯར༻͢Δ ಉ͡Α͏ͳΠϯϑϥ࡞ۀΛݮΒ͠ɺϦιʔεΛޮతʹར༻͢Δ ࢀߟ: CodeZineʮKubernetesΛͬͨมԽʹڧ͍Πϯϑϥ――WantedlyͷΠϯϑϥνʔϜ͕େʹ͍ͯ͠Δ͜ ͱʯhttps://codezine.jp/article/detail/10357
The Twelve Factor App •12 ݸͷΞϓϦέʔγϣϯͷํ๏ https://12factor.net/ja/ •Herokuࣄଶίϯςφٕज़ɻDocker Λӡ༻͍ͯ͘͠࠷దͳํ๏ •σʔλετΞΛίϯςφʹؚΊͳ͍
•։ൃڥͱຊ൪ڥͷ ࠩҟΛڥมͰઃఆ •ܧଓతσϓϩΠ
ͦͷଞՃϧʔϧ Docker ͷ image ܰ͘ CI Λ௨ͯ͠ɺimage ࡞ɺdeploy Λߦͳ͏ ϦϙδτϦͱ
namespace ͕ 1ର1 ʹͳΔΑ͏ʹ
docker image ܰ͘ https://github.com/koudaiii/sltd/blob/master/Dockerfile Golang Ͱ͋ΕόΠφϦΛΆΜى͖ͩͱখ͍͞ 1 FROM alpine:3.6 2
3 RUN apk add --no-cache --update ca-certificates 4 5 COPY bin/sltd /sltd 6 7 ENTRYPOINT ["/sltd"] 8 CMD ["help"]
CI/CD .travis.yml ΑΓൈਮ 58 script: 59 - ./script/ci-test # Test
60 - ./script/ci-build # docker build 61 - ./script/ci-push # docker push 62 ɾɾɾɾɾɾɾɾɾɾɾ 69 deploy: 70 skip_cleanup: true 71 provider: script 72 script: ./script/ci-deploy # Release 73 on: 74 all_branches: true
None
ϦϙδτϦ = namespace ֤ϦϙδτϦʹ /kubernetes Λஔ $ tree kubernetes/ kubernetes/
ᵓᴷᴷ jobs ᴹ ᵓᴷᴷ create-blog-job.yaml ᴹ ᵓᴷᴷ destroy-blog-job.yaml ᴹ ᵋᴷᴷ update-monthly-ranking-job.yaml ᵓᴷᴷ namespace.yaml ᵓᴷᴷ rails-docker-sample-run.yaml ᵓᴷᴷ rails-docker-sample-hpa.yaml ᵋᴷᴷ rails-docker-sample-svc.yaml
ϚχϑΣετϑΝΠϧ Namespace ઐ༻ͷ໊લۭؒ Deployment ΞϓϦέʔγϣϯͷ࣮ߦ HorizontalPodAutoscaler cpu/memory ͔Β pod Φʔτεέʔϧ
Pod one-off ίϯςφ༻(rails c Λߦͳ͏༻) Service SSL͖ELB Job oneshot Ͱ࣮ߦ༻(rake db:migrate ) CronJob Cron ܗࣜͰλεΫΛ࣮ߦ(rake task )
Namespace ઐ༻ͷ໊લۭؒΛ࡞ 1 apiVersion: v1 2 kind: Namespace 3 metadata:
4 name: hoge
Deployment 1 apiVersion: extensions/v1beta1 2 kind: Deployment ɾɾɾɾɾɾɾɾɾɾɾ 9 spec:
11 strategy: 12 type: RollingUpdate 13 rollingUpdate: 14 maxSurge: 50% # ҰೖΕସ͑ 15 maxUnavailable: 0 # replicas Λҡ࣋ ɾɾɾɾɾɾɾɾɾɾɾ 23 spec: 24 containers: 25 - image: wantedly/python:latest 26 name: python 27 ports: 28 - containerPort: 8000 29 readinessProbe: # ready ͷఆٛ 30 httpGet: 31 path: /ping 32 port: 8000 33 initialDelaySeconds: 10 34 timeoutSeconds: 1 35 envFrom: 36 - secretRef: 37 name: dotenv 38 command: ["script/server"]
HorizontalPodAutoscaler Pod ͷ autoscale 1 apiVersion: autoscaling/v1 2 kind: HorizontalPodAutoscaler
3 metadata: 4 name: hoge 5 namespace: hoge 6 spec: 7 maxReplicas: 30 8 minReplicas: 1 9 scaleTargetRef: 10 apiVersion: extensions/v1beta1 11 kind: Deployment 12 name: python 13 targetCPUUtilizationPercentage: 50
Pod one-off Container(ྫ: rails c) 1 apiVersion: v1 2 kind:
Pod 3 metadata: 4 name: {USER} 5 namespace: hoge 6 labels: 7 role: console 8 spec: ɾɾɾɾɾɾɾɾɾɾɾ 11 containers: 12 - image: wantedly/bash:latest 13 imagePullPolicy: Always 14 name: {USER} 15 command: 16 - bash 17 stdin: true 18 stdinOnce: true 19 terminationMessagePath: /dev/termination-log 20 tty: true 21 envFrom: 22 - secretRef: 23 name: dotenv
Service SSLূ໌ॻ͖ ELB 1 apiVersion: v1 2 kind: Service 3
metadata: 4 name: hoge 6 annotations: 7 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:reagion:x:certificate/x 8 service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http 9 labels: 10 name: hoge 11 role: web 12 spec: 13 ports: 14 - port: 443 15 protocol: TCP 16 targetPort: 8000 17 selector: 18 name: hoge 19 role: web 20 type: LoadBalancer
Job (ྫ rake db:migrate) 1 apiVersion: batch/v1 2 kind: Job
ɾɾɾɾɾɾɾɾɾɾɾ 16 spec: 17 restartPolicy: Never 18 containers: 19 - name: db-migrate-[REPLACE_WITH_DATETIME] 20 image: wantedly/rails:[REPLACE_WITH_TAG] 21 command: ["bundle", "exec", "rake", "db:migrate"] 22 envFrom: 23 - secretRef: 24 name: dotenv
CronJob (ྫ rake task) concurrencyPolicy Ͱલ࣮ߦ͕ͬͨ߹ͷϋϯυϦϯά͕ग़དྷΔ 1 apiVersion: batch/v2alpha1 2
kind: CronJob 3 metadata: 4 namespace: hoge 5 name: create-post 6 labels: 7 job: create-post 9 spec: 10 schedule: "*/30 * * * *" 11 concurrencyPolicy: “Replace" # “Allow” or “Forbid” 12 successfulJobsHistoryLimit: 5 13 failedJobsHistoryLimit: 5 ɾɾɾɾɾɾɾɾɾɾɾ
ղܾͨ͜͠ͱ / ՝ͷऔΓΈ
ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞
৽نαʔϏεͷग़͢͠͞ ͜Ε·Ͱ ݱࡏ
εέʔϧͷ͢͠͞ •͜Ε·Ͱ instance ϨϕϧͰɺεέʔϧΛߦͳ͏ΈΛ࡞Δඞཁ͕͋ͬͨ •ϔϧενΣοΫ •αʔϏεσΟεΧόϦ •DNSొ / Load Balancer
•ࣗݾम෮ •Kubernetes ͔ΒɺԼهͷ͔Β༰қʹͳͬͨ •αʔϏεσΟεΧόϦͱDNSɺLoad Balancer ྑ͠ͳʹ͞Ε͍ͯΔ •ίϯςφϨϕϧͷεέʔϧग़དྷΔ $ kubectl autoscale deployment foo --min=2 —-max=10 —cpu-percent=50 # શମͷCPUUtilization 50% ᮢͰ૿ݮ $ kubectl scale --replicas=3 deployment foo # ͦͷͰ3ͭʹ͢Δ
՝ʹͳͬͨ͜ͱ 1. खಈͰ͖ͬͯͨ͜ͱͷݶք 2. Kubernetes ͷ Master ͕ SPOF 3.
ϞχλϦϯάͷෳࡶ͞
՝ͷऔΓΈ 1. τΠϧ໓ӡಈ 2. ϚϧνϚελʔ 3. ࣗಈϞχλϦϯά
“τΠϧͱɺϓϩμΫγϣϯαʔϏεΛಈ࡞ͤ͞Δ͜ͱʹ ؔ͢Δ࡞ۀͰɺख࡞ۀͰ܁Γฦ͠ߦΘΕɺࣗಈԽ͢Δ͜ͱ ͕ՄೳͰ͋Γɺઓज़తͰظతͳՁΛ࣋ͨͣɺ࡞ۀྔ͕αʔ Ϗεͷʹൺྫ͢Δͱ͍ͬͨΛ࣋ͭͷͰ͢ɻ” ൈਮɿ: Betsy Beyer “SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά”
ҰͭҎ্ͯ·ΔͱτΠϧͷՄೳੑ༗ • ख࡞ۀͰ͋Δ͜ͱ • ܁Γฦ͞ΕΔ͜ͱ • ࣗಈԽग़དྷΔ͜ͱ • ઓज़తͰ͋Δ͜ͱ(ઓུతͰ͋ͬͨΓ༧ଌʹجͮ͘ͷͰͳ͘ɺׂΓࠐΈͰ࢝·ΓɺͳͲ ͕ੜͨ͜͡ͱͷରԠ)
• ظతͳՁΛ࣋ͨͳ͍͜ͱ • αʔϏεͷʹରͯ͠O(n)Ͱ͋Δ͜ͱ
τΠϧ໓ӡಈ •test-build-push-release => CI ͷ template Խ •autoscale => HorizontalPodAutoscaler
ϚχϑΣετΛॻ͘ •migrate => Job ϚχϑΣετΛॻ͘ •one-off => ઐ༻ͷ Pod ϚχϑΣετΛॻ͘ •SSL ͷखܰ͞ => Service ϚχϑΣετʹ annotation Λه͢Δͱઃఆ͞ΕΔ •deploy & rollback => The UNIX Philosophy ଇͬͨπʔϧ࡞ ࣮΄΅,VCFSOFUFTͰ࣮ݱ
The UNIX Philosophy
The UNIX Philosophy ൈਮ • Small is beautiful. খ͍͞ͷඒ͍͠ •
Make each program do one thing well. 1ͭͷϓϩάϥϜʹ1ͭͷ͜ͱΛ͏·͘ ΒͤΔ • Choose portability over efficiency. ޮΑΓҠ২ੑΛ༏ઌ͢Δ • Use software leverage to your advantage. ιϑτΣΞΛᑏࢠ(ͯ͜)ͱͯ͠͏ • Avoid captive user interfaces. աͷରతΠϯλʔϑΣʔεΛආ͚Δ
Server Tools ͱ Client Tool • Ұͭͷ͜ͱΛ্ख͘Δπʔϧ࡞Γɺᑏࢠͱͯ͠ར༻ => ͍ͭͰަՄೳ •
ΫϥΠΞϯτଆ׳Ε͠Μͩύοέʔδཧ => homebrew Ͱ৴ • ΫϥΠΞϯτଆπʔϧΛݺͼग़ͨ͢Ίͷϥούʔʹઐ೦ • ΫϥΠΞϯτଆʹग़དྷΔݶΓϩδοΫΛೖΕͳ͍ => ॲཧ server ্ • ωοτஅΛఆ͢Δ => server ͱ container ҡ࣋
homebrew Golang πʔϧΛ homebrew Ͱ binary Λ৴ ΫϩείϯύΠϥͰOSґଘ͕ۃྗͳ͘πʔϧ͕͑ΔΑ͏ʹ
kube ࣾπʔϧ
खಈσϓϩΠ ϚχϑΣετϑΝΠϧʹ annotations Λه͢Δ͜ͱͰखಈσϓϩΠ͕Ͱ͖Δ k8ship deploy [BRANCH|COMMIT_SHA1] 1 apiVersion: extensions/v1beta1
2 kind: Deployment 3 metadata: 4 name: rails 5 namespace: rails 6 labels: 7 name: rails 8 role: web 9 annotations: 10 wantedly.com/deploy-target: “true" # ه 11 wantedly.com/deploy-target-container: rails # ه 12 wantedly.com/github: rails=wantedly/rails # ه https://github.com/dtan4/k8ship
kube deploy ίϚϯυ -dry-run ΦϓγϣϯͰ diff Λग़͢ merge ޙҙͷλΠϛϯάͰ deploy
͍ͨ͠߹ʹར༻ $ kube qa deploy master --dry-run [dry-run] deploy to (deployment: "rails", container: "rails") [dry-run] before: wantedly/rails:d1cb608ee61cb18f8c397c2e27576573879b2fcf [dry-run] after: wantedly/rails:cb19269526f946222110fbdfb3d107c4b5a18fe7
ϚϧνϚελʔ
ϚϧνϚελʔ •Kubernetes ͷ version up Ͳ͏͢Δͷʁ •Ϛελʔ͕ࢭ·ΔͱͲ͏ͳΔͷʁ •Ϋϥελͷఀࢭͳ͘ kops Λͬͯ
Rolling upgrade Ͱ͖ΔΑ͏ʹมߋ •ϚελʔΛࡾ༻ҙ͠ɺϚϧνϚελʔߏʹ͔͑ͯ SPOF վળ
ࣗಈϞχλϦϯά
ࣗಈϞχλϦϯά • 1ͭ1ͭઃఆ͢ΔͷͰͳ͘ɺ൚༻తʹద༻͞ΕΔΑ͏ʹ͢Δ • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}}
• request ͱ status code औಘ • μογϡϘʔυʹө • Deployment Ͱఆٛͨ͠
ࣗಈͰಉ࣭͡ͷϞχλϦϯάΛࢦ͢ https://www.datadoghq.com/blog/monitoring-101-collecting-data/ http://qiita.com/koudaiii/items/bc89368e1279649f2498
Dashboard
ϞχλϦϯάͰ༻͍ͯ͠Δͷ • kelseyhightower/konfd => PostgresqlͳͲͷใΛϦϙδτϦʹؚΊͳ͍࣌ʹར༻ • kubernetes/kube-state-metrics => deployment ͷ
available Λऔಘ • datadog/docker-dd-agent => docker ༻ɺ web ༻ɺ db ༻ͱׂͯ͠ผͰඋ • koudaiii/sltd => Service ͷ metadata Λݩʹ ELB ͷ tag ʹՃ • logentries => ΫϥελશͯͷϩάΛૹΔ
ਪҠ
αʔϏεਪҠ (2017/04Ἤ) Service 9 => 60 Container 700+ => 1,500
- 2,500+
·ͱΊ • kubernetes ɺΠϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬʹͳͬͨ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞
• ՝͔ΒऔΓΜͩ͜ͱ • खಈͰ͖ͬͯͨ͜ͱͷݶք => τΠϧ໓ӡಈ • Ϛελʔ͕ SPOF => ϚϧνϚελʔԽ • ϞχλϦϯάͷෳࡶ͞ => ࣗಈϞχλϦϯά
Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
IUUQTXXXXBOUFEMZDPNQSPKFDUT