Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kubernetes で解決したこと新たに出てきた課題 /#jtf2017
Search
Kodai Sakabe
August 27, 2017
Technology
7
7.5k
Kubernetes で解決したこと新たに出てきた課題 /#jtf2017
1年を通して Kubernetes で解決したこと新たに出てきた課題をどう立ち向かったを発表しました
at July Tech Festa 2017
Kodai Sakabe
August 27, 2017
Tweet
Share
More Decks by Kodai Sakabe
See All by Kodai Sakabe
Terraform のための AVM を使った IaC のモジュール戦略 / Azure Verified Modules for Terraform
koudaiii
0
83
Well-Architected Framework オペレーションエクセレンス
koudaiii
0
68
AIアプリを支えるチームの最適な開発環境を整備する / / Optimizing AI Development Environment with Platform/DevOps Engineers
koudaiii
0
160
第4回:Boundaryを使用して動的に作成されたAzureリソースにアクセスする / Using Boundary to access dynamically created azure
koudaiii
0
110
[INF-4] 改めて学ぶ Immutable Infrastructure Golden Image編
koudaiii
0
47
Microsoft Entra External ID / FUJIYAMA Tech Showcase #fujishow
koudaiii
1
230
第3回PackerでAzureにおけるイメージコンプライアンスを実現する / HCP Packer + GitHub Actions on Azure
koudaiii
1
180
Azure OpenAI Service リファレンスアーキテクチャからみる本番システムレベルの LLM アプリに必要な検討項目の解説 / From Azure OpenAI Reference Architecture to Production-Ready LLM Apps #serverlessdays #serverlesstokyo
koudaiii
11
3.9k
about Azure SREインフラ分科会 / Azure SRE and Infrastructure SIG
koudaiii
0
100
Other Decks in Technology
See All in Technology
Wvlet: A New Flow-Style Query Language For Functional Data Modeling and Interactive Data Analysis - Trino Summit 2024
xerial
1
110
AWS re:Invent 2024 ふりかえり
kongmingstrap
0
130
Amazon Kendra GenAI Index 登場でどう変わる? 評価から学ぶ最適なRAG構成
naoki_0531
0
100
KnowledgeBaseDocuments APIでベクトルインデックス管理を自動化する
iidaxs
1
250
podman_update_2024-12
orimanabu
1
260
OpenAIの蒸留機能(Model Distillation)を使用して運用中のLLMのコストを削減する取り組み
pharma_x_tech
4
540
Fanstaの1年を大解剖! 一人SREはどこまでできるのか!?
syossan27
2
160
日本版とグローバル版のモバイルアプリ統合の開発の裏側と今後の展望
miichan
1
120
[Ruby] Develop a Morse Code Learning Gem & Beep from Strings
oguressive
1
150
どちらを使う?GitHub or Azure DevOps Ver. 24H2
kkamegawa
0
630
LINE Developersプロダクト(LIFF/LINE Login)におけるフロントエンド開発
lycorptech_jp
PRO
0
120
組織に自動テストを書く文化を根付かせる戦略(2024冬版) / Building Automated Test Culture 2024 Winter Edition
twada
PRO
12
3.4k
Featured
See All Featured
No one is an island. Learnings from fostering a developers community.
thoeni
19
3k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
Speed Design
sergeychernyshev
25
670
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
169
50k
Adopting Sorbet at Scale
ufuk
73
9.1k
The Power of CSS Pseudo Elements
geoffreycrofte
73
5.4k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
28
8.3k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
32
2.7k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
1.2k
Designing on Purpose - Digital PM Summit 2013
jponch
116
7k
Product Roadmaps are Hard
iamctodd
PRO
49
11k
Transcript
,PEBJ4BLBCF!LPVEBJJJ8BOUFEMZ *OD Kubernetes Ͱղܾͨ͜͠ͱ ৽ͨʹग़͖ͯͨ՝ July Tech Festa 2017
ABOUT ME — Software Engineer — 2010/4- TIS ࡔ෦ େ
(KODAI SAKABE) 2015/8- Wantedly @koudaiii Site: https://koudaiii.com
None
Kubernetes ෳͷϗετؒͰίϯςφԽ͞ΕͨΞϓϦέʔγϣϯΛཧ͢ΔͨΊͷOSS ΞϓϦέʔγϣϯͷdeployment, maintenance, scalingͷͨΊͷجຊతͳϝΧχζϜΛఏڙ https://github.com/kubernetes/kubernetes
Kubernetes Λར༻ͨ݁͠Ռ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞ • ՝ʹͳͬͨ͜ͱ
• खಈͰ͖ͬͯͨ͜ͱͷݶք • Kubernetes ͷ Master ͕ SPOF • ϞχλϦϯάͷෳࡶ͞
Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
ΞδΣϯμ ڈ·ͰͷৼΓฦΓ ࣮ફ Kubernetes ղܾͨ͜͠ͱ / ՝ͷऔΓΈ τΠϧ໓ӡಈ The UNIX
Philosophy ϚϧνϚελʔ ࣗಈϞχλϦϯά
ڈ·ͰͷৼΓฦΓ
͜Ε·ͰͷऔΓΈ • Docker / Chef / Packer • Blue-Green Deployment
• Terraform • CoreOS • cell
Docker / Chef / Packer • Docker image ͷதʹ Chef
ΛͬͯΠϯϑϥΛߏங • Packer Ͱ AWS ͷ instance ͷ AMI Λ࡞Δ • Base ͱͳΔ template Λ༻ҙͯ͠ɺ৽͍͠αʔϏε͕ग़͖ͯͯྲྀ༻Ͱ ͖ΔΑ͏ʹͨ͠ • ΞϓϦέʔγϣϯͷߏங෦Λ Code Խ
Blue-Green Deployment • /healthcheck ͰϨεϙϯεΛฦͨ͠ΒΓସ͑Δ UBHT OBNFBQQ BQQSBJMT DPMPSHSFFO UBHT
OBNFBQQ BQQSBJMT DPMPSCMVF DBQEFQMPZTXJUDI@DPMPSCMVF QPSU QPSU
Terraform • AWS(S3/RDS/ELB etc) / dnsimple Λૢ࡞ IUUQTTQFBLFSEFDLDPNEUBOUFSSBGPSNBUXBOUFEMZUFDIDJSDMFOVNCFS
TAG ͱ AutoScaleGroup(ASG) • instance ͷׂΛɺ EC2 ͷ TAG Λͬͯཧ
• Կಈ͔͍͔ͨ͠ʁ => ASGͰઃఆ • ΠϯϑϥπʔϧΛ LaunchConfig ʹఆٛ • instance ্ཱ͕͕ͪΔͱ࣌ʹࣗʹ͚ΒΕͨ TAG Λ؍ͯɺඞཁͳ service Λ systemd ʹө͠ɺαʔϏεΛىಈ(cell)
cell • AWS TAG Λݩʹ systemd ʹαʔϏεΛొ • αʔϏεͷ status
͕ ok ʹͳͬͨΒ DNS ࣗΛՃ
ٕज़ελοΫ • ৽نαʔϏεΛ্ཱͪ͛Δࡍ͜ͷٕज़ ελοΫΛશ෦༻ҙ͢Δ • Πϯϑϥͷ҆ఆԽͷͨΊͷ༷ʑπʔϧͱ ΈΛ༻ҙ • αʔόʔΛߏங͢Δͱ͜Ζ͔ΒσϓϩΠɺ cell
·ͰΛίʔυԽ Web Application Monitoring Logging Auto Scale Load Balancer Internal DNS
ղͳखॱ
࣮ફ Kubernetes
Kubernetes ಋೖ αʔϏε͕Ͱ͖ΔʹҰݸͣͭαʔόʔΛ༻ҙͯ͠ߏங͢Δํ๏ ෳͷαʔόʔΛҰͭͷେ͖ͳαʔόʔʹݟཱͯͯར༻͢Δ ಉ͡Α͏ͳΠϯϑϥ࡞ۀΛݮΒ͠ɺϦιʔεΛޮతʹར༻͢Δ ࢀߟ: CodeZineʮKubernetesΛͬͨมԽʹڧ͍Πϯϑϥ――WantedlyͷΠϯϑϥνʔϜ͕େʹ͍ͯ͠Δ͜ ͱʯhttps://codezine.jp/article/detail/10357
The Twelve Factor App •12 ݸͷΞϓϦέʔγϣϯͷํ๏ https://12factor.net/ja/ •Herokuࣄଶίϯςφٕज़ɻDocker Λӡ༻͍ͯ͘͠࠷దͳํ๏ •σʔλετΞΛίϯςφʹؚΊͳ͍
•։ൃڥͱຊ൪ڥͷ ࠩҟΛڥมͰઃఆ •ܧଓతσϓϩΠ
ͦͷଞՃϧʔϧ Docker ͷ image ܰ͘ CI Λ௨ͯ͠ɺimage ࡞ɺdeploy Λߦͳ͏ ϦϙδτϦͱ
namespace ͕ 1ର1 ʹͳΔΑ͏ʹ
docker image ܰ͘ https://github.com/koudaiii/sltd/blob/master/Dockerfile Golang Ͱ͋ΕόΠφϦΛΆΜى͖ͩͱখ͍͞ 1 FROM alpine:3.6 2
3 RUN apk add --no-cache --update ca-certificates 4 5 COPY bin/sltd /sltd 6 7 ENTRYPOINT ["/sltd"] 8 CMD ["help"]
CI/CD .travis.yml ΑΓൈਮ 58 script: 59 - ./script/ci-test # Test
60 - ./script/ci-build # docker build 61 - ./script/ci-push # docker push 62 ɾɾɾɾɾɾɾɾɾɾɾ 69 deploy: 70 skip_cleanup: true 71 provider: script 72 script: ./script/ci-deploy # Release 73 on: 74 all_branches: true
None
ϦϙδτϦ = namespace ֤ϦϙδτϦʹ /kubernetes Λஔ $ tree kubernetes/ kubernetes/
ᵓᴷᴷ jobs ᴹ ᵓᴷᴷ create-blog-job.yaml ᴹ ᵓᴷᴷ destroy-blog-job.yaml ᴹ ᵋᴷᴷ update-monthly-ranking-job.yaml ᵓᴷᴷ namespace.yaml ᵓᴷᴷ rails-docker-sample-run.yaml ᵓᴷᴷ rails-docker-sample-hpa.yaml ᵋᴷᴷ rails-docker-sample-svc.yaml
ϚχϑΣετϑΝΠϧ Namespace ઐ༻ͷ໊લۭؒ Deployment ΞϓϦέʔγϣϯͷ࣮ߦ HorizontalPodAutoscaler cpu/memory ͔Β pod Φʔτεέʔϧ
Pod one-off ίϯςφ༻(rails c Λߦͳ͏༻) Service SSL͖ELB Job oneshot Ͱ࣮ߦ༻(rake db:migrate ) CronJob Cron ܗࣜͰλεΫΛ࣮ߦ(rake task )
Namespace ઐ༻ͷ໊લۭؒΛ࡞ 1 apiVersion: v1 2 kind: Namespace 3 metadata:
4 name: hoge
Deployment 1 apiVersion: extensions/v1beta1 2 kind: Deployment ɾɾɾɾɾɾɾɾɾɾɾ 9 spec:
11 strategy: 12 type: RollingUpdate 13 rollingUpdate: 14 maxSurge: 50% # ҰೖΕସ͑ 15 maxUnavailable: 0 # replicas Λҡ࣋ ɾɾɾɾɾɾɾɾɾɾɾ 23 spec: 24 containers: 25 - image: wantedly/python:latest 26 name: python 27 ports: 28 - containerPort: 8000 29 readinessProbe: # ready ͷఆٛ 30 httpGet: 31 path: /ping 32 port: 8000 33 initialDelaySeconds: 10 34 timeoutSeconds: 1 35 envFrom: 36 - secretRef: 37 name: dotenv 38 command: ["script/server"]
HorizontalPodAutoscaler Pod ͷ autoscale 1 apiVersion: autoscaling/v1 2 kind: HorizontalPodAutoscaler
3 metadata: 4 name: hoge 5 namespace: hoge 6 spec: 7 maxReplicas: 30 8 minReplicas: 1 9 scaleTargetRef: 10 apiVersion: extensions/v1beta1 11 kind: Deployment 12 name: python 13 targetCPUUtilizationPercentage: 50
Pod one-off Container(ྫ: rails c) 1 apiVersion: v1 2 kind:
Pod 3 metadata: 4 name: {USER} 5 namespace: hoge 6 labels: 7 role: console 8 spec: ɾɾɾɾɾɾɾɾɾɾɾ 11 containers: 12 - image: wantedly/bash:latest 13 imagePullPolicy: Always 14 name: {USER} 15 command: 16 - bash 17 stdin: true 18 stdinOnce: true 19 terminationMessagePath: /dev/termination-log 20 tty: true 21 envFrom: 22 - secretRef: 23 name: dotenv
Service SSLূ໌ॻ͖ ELB 1 apiVersion: v1 2 kind: Service 3
metadata: 4 name: hoge 6 annotations: 7 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:reagion:x:certificate/x 8 service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http 9 labels: 10 name: hoge 11 role: web 12 spec: 13 ports: 14 - port: 443 15 protocol: TCP 16 targetPort: 8000 17 selector: 18 name: hoge 19 role: web 20 type: LoadBalancer
Job (ྫ rake db:migrate) 1 apiVersion: batch/v1 2 kind: Job
ɾɾɾɾɾɾɾɾɾɾɾ 16 spec: 17 restartPolicy: Never 18 containers: 19 - name: db-migrate-[REPLACE_WITH_DATETIME] 20 image: wantedly/rails:[REPLACE_WITH_TAG] 21 command: ["bundle", "exec", "rake", "db:migrate"] 22 envFrom: 23 - secretRef: 24 name: dotenv
CronJob (ྫ rake task) concurrencyPolicy Ͱલ࣮ߦ͕ͬͨ߹ͷϋϯυϦϯά͕ग़དྷΔ 1 apiVersion: batch/v2alpha1 2
kind: CronJob 3 metadata: 4 namespace: hoge 5 name: create-post 6 labels: 7 job: create-post 9 spec: 10 schedule: "*/30 * * * *" 11 concurrencyPolicy: “Replace" # “Allow” or “Forbid” 12 successfulJobsHistoryLimit: 5 13 failedJobsHistoryLimit: 5 ɾɾɾɾɾɾɾɾɾɾɾ
ղܾͨ͜͠ͱ / ՝ͷऔΓΈ
ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞
৽نαʔϏεͷग़͢͠͞ ͜Ε·Ͱ ݱࡏ
εέʔϧͷ͢͠͞ •͜Ε·Ͱ instance ϨϕϧͰɺεέʔϧΛߦͳ͏ΈΛ࡞Δඞཁ͕͋ͬͨ •ϔϧενΣοΫ •αʔϏεσΟεΧόϦ •DNSొ / Load Balancer
•ࣗݾम෮ •Kubernetes ͔ΒɺԼهͷ͔Β༰қʹͳͬͨ •αʔϏεσΟεΧόϦͱDNSɺLoad Balancer ྑ͠ͳʹ͞Ε͍ͯΔ •ίϯςφϨϕϧͷεέʔϧग़དྷΔ $ kubectl autoscale deployment foo --min=2 —-max=10 —cpu-percent=50 # શମͷCPUUtilization 50% ᮢͰ૿ݮ $ kubectl scale --replicas=3 deployment foo # ͦͷͰ3ͭʹ͢Δ
՝ʹͳͬͨ͜ͱ 1. खಈͰ͖ͬͯͨ͜ͱͷݶք 2. Kubernetes ͷ Master ͕ SPOF 3.
ϞχλϦϯάͷෳࡶ͞
՝ͷऔΓΈ 1. τΠϧ໓ӡಈ 2. ϚϧνϚελʔ 3. ࣗಈϞχλϦϯά
“τΠϧͱɺϓϩμΫγϣϯαʔϏεΛಈ࡞ͤ͞Δ͜ͱʹ ؔ͢Δ࡞ۀͰɺख࡞ۀͰ܁Γฦ͠ߦΘΕɺࣗಈԽ͢Δ͜ͱ ͕ՄೳͰ͋Γɺઓज़తͰظతͳՁΛ࣋ͨͣɺ࡞ۀྔ͕αʔ Ϗεͷʹൺྫ͢Δͱ͍ͬͨΛ࣋ͭͷͰ͢ɻ” ൈਮɿ: Betsy Beyer “SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά”
ҰͭҎ্ͯ·ΔͱτΠϧͷՄೳੑ༗ • ख࡞ۀͰ͋Δ͜ͱ • ܁Γฦ͞ΕΔ͜ͱ • ࣗಈԽग़དྷΔ͜ͱ • ઓज़తͰ͋Δ͜ͱ(ઓུతͰ͋ͬͨΓ༧ଌʹجͮ͘ͷͰͳ͘ɺׂΓࠐΈͰ࢝·ΓɺͳͲ ͕ੜͨ͜͡ͱͷରԠ)
• ظతͳՁΛ࣋ͨͳ͍͜ͱ • αʔϏεͷʹରͯ͠O(n)Ͱ͋Δ͜ͱ
τΠϧ໓ӡಈ •test-build-push-release => CI ͷ template Խ •autoscale => HorizontalPodAutoscaler
ϚχϑΣετΛॻ͘ •migrate => Job ϚχϑΣετΛॻ͘ •one-off => ઐ༻ͷ Pod ϚχϑΣετΛॻ͘ •SSL ͷखܰ͞ => Service ϚχϑΣετʹ annotation Λه͢Δͱઃఆ͞ΕΔ •deploy & rollback => The UNIX Philosophy ଇͬͨπʔϧ࡞ ࣮΄΅,VCFSOFUFTͰ࣮ݱ
The UNIX Philosophy
The UNIX Philosophy ൈਮ • Small is beautiful. খ͍͞ͷඒ͍͠ •
Make each program do one thing well. 1ͭͷϓϩάϥϜʹ1ͭͷ͜ͱΛ͏·͘ ΒͤΔ • Choose portability over efficiency. ޮΑΓҠ২ੑΛ༏ઌ͢Δ • Use software leverage to your advantage. ιϑτΣΞΛᑏࢠ(ͯ͜)ͱͯ͠͏ • Avoid captive user interfaces. աͷରతΠϯλʔϑΣʔεΛආ͚Δ
Server Tools ͱ Client Tool • Ұͭͷ͜ͱΛ্ख͘Δπʔϧ࡞Γɺᑏࢠͱͯ͠ར༻ => ͍ͭͰަՄೳ •
ΫϥΠΞϯτଆ׳Ε͠Μͩύοέʔδཧ => homebrew Ͱ৴ • ΫϥΠΞϯτଆπʔϧΛݺͼग़ͨ͢Ίͷϥούʔʹઐ೦ • ΫϥΠΞϯτଆʹग़དྷΔݶΓϩδοΫΛೖΕͳ͍ => ॲཧ server ্ • ωοτஅΛఆ͢Δ => server ͱ container ҡ࣋
homebrew Golang πʔϧΛ homebrew Ͱ binary Λ৴ ΫϩείϯύΠϥͰOSґଘ͕ۃྗͳ͘πʔϧ͕͑ΔΑ͏ʹ
kube ࣾπʔϧ
खಈσϓϩΠ ϚχϑΣετϑΝΠϧʹ annotations Λه͢Δ͜ͱͰखಈσϓϩΠ͕Ͱ͖Δ k8ship deploy [BRANCH|COMMIT_SHA1] 1 apiVersion: extensions/v1beta1
2 kind: Deployment 3 metadata: 4 name: rails 5 namespace: rails 6 labels: 7 name: rails 8 role: web 9 annotations: 10 wantedly.com/deploy-target: “true" # ه 11 wantedly.com/deploy-target-container: rails # ه 12 wantedly.com/github: rails=wantedly/rails # ه https://github.com/dtan4/k8ship
kube deploy ίϚϯυ -dry-run ΦϓγϣϯͰ diff Λग़͢ merge ޙҙͷλΠϛϯάͰ deploy
͍ͨ͠߹ʹར༻ $ kube qa deploy master --dry-run [dry-run] deploy to (deployment: "rails", container: "rails") [dry-run] before: wantedly/rails:d1cb608ee61cb18f8c397c2e27576573879b2fcf [dry-run] after: wantedly/rails:cb19269526f946222110fbdfb3d107c4b5a18fe7
ϚϧνϚελʔ
ϚϧνϚελʔ •Kubernetes ͷ version up Ͳ͏͢Δͷʁ •Ϛελʔ͕ࢭ·ΔͱͲ͏ͳΔͷʁ •Ϋϥελͷఀࢭͳ͘ kops Λͬͯ
Rolling upgrade Ͱ͖ΔΑ͏ʹมߋ •ϚελʔΛࡾ༻ҙ͠ɺϚϧνϚελʔߏʹ͔͑ͯ SPOF վળ
ࣗಈϞχλϦϯά
ࣗಈϞχλϦϯά • 1ͭ1ͭઃఆ͢ΔͷͰͳ͘ɺ൚༻తʹద༻͞ΕΔΑ͏ʹ͢Δ • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}}
• request ͱ status code औಘ • μογϡϘʔυʹө • Deployment Ͱఆٛͨ͠
ࣗಈͰಉ࣭͡ͷϞχλϦϯάΛࢦ͢ https://www.datadoghq.com/blog/monitoring-101-collecting-data/ http://qiita.com/koudaiii/items/bc89368e1279649f2498
Dashboard
ϞχλϦϯάͰ༻͍ͯ͠Δͷ • kelseyhightower/konfd => PostgresqlͳͲͷใΛϦϙδτϦʹؚΊͳ͍࣌ʹར༻ • kubernetes/kube-state-metrics => deployment ͷ
available Λऔಘ • datadog/docker-dd-agent => docker ༻ɺ web ༻ɺ db ༻ͱׂͯ͠ผͰඋ • koudaiii/sltd => Service ͷ metadata Λݩʹ ELB ͷ tag ʹՃ • logentries => ΫϥελશͯͷϩάΛૹΔ
ਪҠ
αʔϏεਪҠ (2017/04Ἤ) Service 9 => 60 Container 700+ => 1,500
- 2,500+
·ͱΊ • kubernetes ɺΠϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬʹͳͬͨ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞
• ՝͔ΒऔΓΜͩ͜ͱ • खಈͰ͖ͬͯͨ͜ͱͷݶք => τΠϧ໓ӡಈ • Ϛελʔ͕ SPOF => ϚϧνϚελʔԽ • ϞχλϦϯάͷෳࡶ͞ => ࣗಈϞχλϦϯά
Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
IUUQTXXXXBOUFEMZDPNQSPKFDUT