Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kubernetes で解決したこと新たに出てきた課題 /#jtf2017
Search
Kodai Sakabe
August 27, 2017
Technology
7
7.6k
Kubernetes で解決したこと新たに出てきた課題 /#jtf2017
1年を通して Kubernetes で解決したこと新たに出てきた課題をどう立ち向かったを発表しました
at July Tech Festa 2017
Kodai Sakabe
August 27, 2017
Tweet
Share
More Decks by Kodai Sakabe
See All by Kodai Sakabe
Terraform のための AVM を使った IaC のモジュール戦略 / Azure Verified Modules for Terraform
koudaiii
0
120
Well-Architected Framework オペレーションエクセレンス
koudaiii
0
71
AIアプリを支えるチームの最適な開発環境を整備する / / Optimizing AI Development Environment with Platform/DevOps Engineers
koudaiii
0
180
第4回:Boundaryを使用して動的に作成されたAzureリソースにアクセスする / Using Boundary to access dynamically created azure
koudaiii
0
120
[INF-4] 改めて学ぶ Immutable Infrastructure Golden Image編
koudaiii
0
56
Microsoft Entra External ID / FUJIYAMA Tech Showcase #fujishow
koudaiii
1
270
第3回PackerでAzureにおけるイメージコンプライアンスを実現する / HCP Packer + GitHub Actions on Azure
koudaiii
1
210
Azure OpenAI Service リファレンスアーキテクチャからみる本番システムレベルの LLM アプリに必要な検討項目の解説 / From Azure OpenAI Reference Architecture to Production-Ready LLM Apps #serverlessdays #serverlesstokyo
koudaiii
11
4.1k
about Azure SREインフラ分科会 / Azure SRE and Infrastructure SIG
koudaiii
0
110
Other Decks in Technology
See All in Technology
Pwned Labsのすゝめ
ken5scal
2
440
Change Managerを活用して本番環境へのセキュアなGUIアクセスを統制する / Control Secure GUI Access to the Production Environment with Change Manager
yuj1osm
0
100
データエンジニアリング領域におけるDuckDBのユースケース
chanyou0311
9
2.2k
Apache Iceberg Case Study in LY Corporation
lycorptech_jp
PRO
0
330
NFV基盤のOpenStack更新 ~9世代バージョンアップへの挑戦~
vtj
0
360
日経のデータベース事業とElasticsearch
hinatades
PRO
0
240
AIエージェント元年@日本生成AIユーザ会
shukob
1
210
【5分でわかる】セーフィー エンジニア向け会社紹介
safie_recruit
0
19k
エンジニアリング価値を黒字化する バリューベース戦略を用いた 技術戦略策定の道のり
kzkmaeda
6
2.7k
Autonomous Database Serverless 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
17
45k
Fraxinus00tw assembly manual
fukumay
0
110
Snowflakeの開発・運用コストをApache Icebergで効率化しよう!~機能と活用例のご紹介~
sagara
1
460
Featured
See All Featured
How STYLIGHT went responsive
nonsquared
98
5.4k
[RailsConf 2023] Rails as a piece of cake
palkan
53
5.3k
A Tale of Four Properties
chriscoyier
158
23k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
10
1.3k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
129
19k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
59k
Designing Experiences People Love
moore
140
23k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
4
430
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.7k
Typedesign – Prime Four
hannesfritz
40
2.5k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
133
33k
Reflections from 52 weeks, 52 projects
jeffersonlam
348
20k
Transcript
,PEBJ4BLBCF!LPVEBJJJ8BOUFEMZ *OD Kubernetes Ͱղܾͨ͜͠ͱ ৽ͨʹग़͖ͯͨ՝ July Tech Festa 2017
ABOUT ME — Software Engineer — 2010/4- TIS ࡔ෦ େ
(KODAI SAKABE) 2015/8- Wantedly @koudaiii Site: https://koudaiii.com
None
Kubernetes ෳͷϗετؒͰίϯςφԽ͞ΕͨΞϓϦέʔγϣϯΛཧ͢ΔͨΊͷOSS ΞϓϦέʔγϣϯͷdeployment, maintenance, scalingͷͨΊͷجຊతͳϝΧχζϜΛఏڙ https://github.com/kubernetes/kubernetes
Kubernetes Λར༻ͨ݁͠Ռ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞ • ՝ʹͳͬͨ͜ͱ
• खಈͰ͖ͬͯͨ͜ͱͷݶք • Kubernetes ͷ Master ͕ SPOF • ϞχλϦϯάͷෳࡶ͞
Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
ΞδΣϯμ ڈ·ͰͷৼΓฦΓ ࣮ફ Kubernetes ղܾͨ͜͠ͱ / ՝ͷऔΓΈ τΠϧ໓ӡಈ The UNIX
Philosophy ϚϧνϚελʔ ࣗಈϞχλϦϯά
ڈ·ͰͷৼΓฦΓ
͜Ε·ͰͷऔΓΈ • Docker / Chef / Packer • Blue-Green Deployment
• Terraform • CoreOS • cell
Docker / Chef / Packer • Docker image ͷதʹ Chef
ΛͬͯΠϯϑϥΛߏங • Packer Ͱ AWS ͷ instance ͷ AMI Λ࡞Δ • Base ͱͳΔ template Λ༻ҙͯ͠ɺ৽͍͠αʔϏε͕ग़͖ͯͯྲྀ༻Ͱ ͖ΔΑ͏ʹͨ͠ • ΞϓϦέʔγϣϯͷߏங෦Λ Code Խ
Blue-Green Deployment • /healthcheck ͰϨεϙϯεΛฦͨ͠ΒΓସ͑Δ UBHT OBNFBQQ BQQSBJMT DPMPSHSFFO UBHT
OBNFBQQ BQQSBJMT DPMPSCMVF DBQEFQMPZTXJUDI@DPMPSCMVF QPSU QPSU
Terraform • AWS(S3/RDS/ELB etc) / dnsimple Λૢ࡞ IUUQTTQFBLFSEFDLDPNEUBOUFSSBGPSNBUXBOUFEMZUFDIDJSDMFOVNCFS
TAG ͱ AutoScaleGroup(ASG) • instance ͷׂΛɺ EC2 ͷ TAG Λͬͯཧ
• Կಈ͔͍͔ͨ͠ʁ => ASGͰઃఆ • ΠϯϑϥπʔϧΛ LaunchConfig ʹఆٛ • instance ্ཱ͕͕ͪΔͱ࣌ʹࣗʹ͚ΒΕͨ TAG Λ؍ͯɺඞཁͳ service Λ systemd ʹө͠ɺαʔϏεΛىಈ(cell)
cell • AWS TAG Λݩʹ systemd ʹαʔϏεΛొ • αʔϏεͷ status
͕ ok ʹͳͬͨΒ DNS ࣗΛՃ
ٕज़ελοΫ • ৽نαʔϏεΛ্ཱͪ͛Δࡍ͜ͷٕज़ ελοΫΛશ෦༻ҙ͢Δ • Πϯϑϥͷ҆ఆԽͷͨΊͷ༷ʑπʔϧͱ ΈΛ༻ҙ • αʔόʔΛߏங͢Δͱ͜Ζ͔ΒσϓϩΠɺ cell
·ͰΛίʔυԽ Web Application Monitoring Logging Auto Scale Load Balancer Internal DNS
ղͳखॱ
࣮ફ Kubernetes
Kubernetes ಋೖ αʔϏε͕Ͱ͖ΔʹҰݸͣͭαʔόʔΛ༻ҙͯ͠ߏங͢Δํ๏ ෳͷαʔόʔΛҰͭͷେ͖ͳαʔόʔʹݟཱͯͯར༻͢Δ ಉ͡Α͏ͳΠϯϑϥ࡞ۀΛݮΒ͠ɺϦιʔεΛޮతʹར༻͢Δ ࢀߟ: CodeZineʮKubernetesΛͬͨมԽʹڧ͍Πϯϑϥ――WantedlyͷΠϯϑϥνʔϜ͕େʹ͍ͯ͠Δ͜ ͱʯhttps://codezine.jp/article/detail/10357
The Twelve Factor App •12 ݸͷΞϓϦέʔγϣϯͷํ๏ https://12factor.net/ja/ •Herokuࣄଶίϯςφٕज़ɻDocker Λӡ༻͍ͯ͘͠࠷దͳํ๏ •σʔλετΞΛίϯςφʹؚΊͳ͍
•։ൃڥͱຊ൪ڥͷ ࠩҟΛڥมͰઃఆ •ܧଓతσϓϩΠ
ͦͷଞՃϧʔϧ Docker ͷ image ܰ͘ CI Λ௨ͯ͠ɺimage ࡞ɺdeploy Λߦͳ͏ ϦϙδτϦͱ
namespace ͕ 1ର1 ʹͳΔΑ͏ʹ
docker image ܰ͘ https://github.com/koudaiii/sltd/blob/master/Dockerfile Golang Ͱ͋ΕόΠφϦΛΆΜى͖ͩͱখ͍͞ 1 FROM alpine:3.6 2
3 RUN apk add --no-cache --update ca-certificates 4 5 COPY bin/sltd /sltd 6 7 ENTRYPOINT ["/sltd"] 8 CMD ["help"]
CI/CD .travis.yml ΑΓൈਮ 58 script: 59 - ./script/ci-test # Test
60 - ./script/ci-build # docker build 61 - ./script/ci-push # docker push 62 ɾɾɾɾɾɾɾɾɾɾɾ 69 deploy: 70 skip_cleanup: true 71 provider: script 72 script: ./script/ci-deploy # Release 73 on: 74 all_branches: true
None
ϦϙδτϦ = namespace ֤ϦϙδτϦʹ /kubernetes Λஔ $ tree kubernetes/ kubernetes/
ᵓᴷᴷ jobs ᴹ ᵓᴷᴷ create-blog-job.yaml ᴹ ᵓᴷᴷ destroy-blog-job.yaml ᴹ ᵋᴷᴷ update-monthly-ranking-job.yaml ᵓᴷᴷ namespace.yaml ᵓᴷᴷ rails-docker-sample-run.yaml ᵓᴷᴷ rails-docker-sample-hpa.yaml ᵋᴷᴷ rails-docker-sample-svc.yaml
ϚχϑΣετϑΝΠϧ Namespace ઐ༻ͷ໊લۭؒ Deployment ΞϓϦέʔγϣϯͷ࣮ߦ HorizontalPodAutoscaler cpu/memory ͔Β pod Φʔτεέʔϧ
Pod one-off ίϯςφ༻(rails c Λߦͳ͏༻) Service SSL͖ELB Job oneshot Ͱ࣮ߦ༻(rake db:migrate ) CronJob Cron ܗࣜͰλεΫΛ࣮ߦ(rake task )
Namespace ઐ༻ͷ໊લۭؒΛ࡞ 1 apiVersion: v1 2 kind: Namespace 3 metadata:
4 name: hoge
Deployment 1 apiVersion: extensions/v1beta1 2 kind: Deployment ɾɾɾɾɾɾɾɾɾɾɾ 9 spec:
11 strategy: 12 type: RollingUpdate 13 rollingUpdate: 14 maxSurge: 50% # ҰೖΕସ͑ 15 maxUnavailable: 0 # replicas Λҡ࣋ ɾɾɾɾɾɾɾɾɾɾɾ 23 spec: 24 containers: 25 - image: wantedly/python:latest 26 name: python 27 ports: 28 - containerPort: 8000 29 readinessProbe: # ready ͷఆٛ 30 httpGet: 31 path: /ping 32 port: 8000 33 initialDelaySeconds: 10 34 timeoutSeconds: 1 35 envFrom: 36 - secretRef: 37 name: dotenv 38 command: ["script/server"]
HorizontalPodAutoscaler Pod ͷ autoscale 1 apiVersion: autoscaling/v1 2 kind: HorizontalPodAutoscaler
3 metadata: 4 name: hoge 5 namespace: hoge 6 spec: 7 maxReplicas: 30 8 minReplicas: 1 9 scaleTargetRef: 10 apiVersion: extensions/v1beta1 11 kind: Deployment 12 name: python 13 targetCPUUtilizationPercentage: 50
Pod one-off Container(ྫ: rails c) 1 apiVersion: v1 2 kind:
Pod 3 metadata: 4 name: {USER} 5 namespace: hoge 6 labels: 7 role: console 8 spec: ɾɾɾɾɾɾɾɾɾɾɾ 11 containers: 12 - image: wantedly/bash:latest 13 imagePullPolicy: Always 14 name: {USER} 15 command: 16 - bash 17 stdin: true 18 stdinOnce: true 19 terminationMessagePath: /dev/termination-log 20 tty: true 21 envFrom: 22 - secretRef: 23 name: dotenv
Service SSLূ໌ॻ͖ ELB 1 apiVersion: v1 2 kind: Service 3
metadata: 4 name: hoge 6 annotations: 7 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:reagion:x:certificate/x 8 service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http 9 labels: 10 name: hoge 11 role: web 12 spec: 13 ports: 14 - port: 443 15 protocol: TCP 16 targetPort: 8000 17 selector: 18 name: hoge 19 role: web 20 type: LoadBalancer
Job (ྫ rake db:migrate) 1 apiVersion: batch/v1 2 kind: Job
ɾɾɾɾɾɾɾɾɾɾɾ 16 spec: 17 restartPolicy: Never 18 containers: 19 - name: db-migrate-[REPLACE_WITH_DATETIME] 20 image: wantedly/rails:[REPLACE_WITH_TAG] 21 command: ["bundle", "exec", "rake", "db:migrate"] 22 envFrom: 23 - secretRef: 24 name: dotenv
CronJob (ྫ rake task) concurrencyPolicy Ͱલ࣮ߦ͕ͬͨ߹ͷϋϯυϦϯά͕ग़དྷΔ 1 apiVersion: batch/v2alpha1 2
kind: CronJob 3 metadata: 4 namespace: hoge 5 name: create-post 6 labels: 7 job: create-post 9 spec: 10 schedule: "*/30 * * * *" 11 concurrencyPolicy: “Replace" # “Allow” or “Forbid” 12 successfulJobsHistoryLimit: 5 13 failedJobsHistoryLimit: 5 ɾɾɾɾɾɾɾɾɾɾɾ
ղܾͨ͜͠ͱ / ՝ͷऔΓΈ
ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞
৽نαʔϏεͷग़͢͠͞ ͜Ε·Ͱ ݱࡏ
εέʔϧͷ͢͠͞ •͜Ε·Ͱ instance ϨϕϧͰɺεέʔϧΛߦͳ͏ΈΛ࡞Δඞཁ͕͋ͬͨ •ϔϧενΣοΫ •αʔϏεσΟεΧόϦ •DNSొ / Load Balancer
•ࣗݾम෮ •Kubernetes ͔ΒɺԼهͷ͔Β༰қʹͳͬͨ •αʔϏεσΟεΧόϦͱDNSɺLoad Balancer ྑ͠ͳʹ͞Ε͍ͯΔ •ίϯςφϨϕϧͷεέʔϧग़དྷΔ $ kubectl autoscale deployment foo --min=2 —-max=10 —cpu-percent=50 # શମͷCPUUtilization 50% ᮢͰ૿ݮ $ kubectl scale --replicas=3 deployment foo # ͦͷͰ3ͭʹ͢Δ
՝ʹͳͬͨ͜ͱ 1. खಈͰ͖ͬͯͨ͜ͱͷݶք 2. Kubernetes ͷ Master ͕ SPOF 3.
ϞχλϦϯάͷෳࡶ͞
՝ͷऔΓΈ 1. τΠϧ໓ӡಈ 2. ϚϧνϚελʔ 3. ࣗಈϞχλϦϯά
“τΠϧͱɺϓϩμΫγϣϯαʔϏεΛಈ࡞ͤ͞Δ͜ͱʹ ؔ͢Δ࡞ۀͰɺख࡞ۀͰ܁Γฦ͠ߦΘΕɺࣗಈԽ͢Δ͜ͱ ͕ՄೳͰ͋Γɺઓज़తͰظతͳՁΛ࣋ͨͣɺ࡞ۀྔ͕αʔ Ϗεͷʹൺྫ͢Δͱ͍ͬͨΛ࣋ͭͷͰ͢ɻ” ൈਮɿ: Betsy Beyer “SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά”
ҰͭҎ্ͯ·ΔͱτΠϧͷՄೳੑ༗ • ख࡞ۀͰ͋Δ͜ͱ • ܁Γฦ͞ΕΔ͜ͱ • ࣗಈԽग़དྷΔ͜ͱ • ઓज़తͰ͋Δ͜ͱ(ઓུతͰ͋ͬͨΓ༧ଌʹجͮ͘ͷͰͳ͘ɺׂΓࠐΈͰ࢝·ΓɺͳͲ ͕ੜͨ͜͡ͱͷରԠ)
• ظతͳՁΛ࣋ͨͳ͍͜ͱ • αʔϏεͷʹରͯ͠O(n)Ͱ͋Δ͜ͱ
τΠϧ໓ӡಈ •test-build-push-release => CI ͷ template Խ •autoscale => HorizontalPodAutoscaler
ϚχϑΣετΛॻ͘ •migrate => Job ϚχϑΣετΛॻ͘ •one-off => ઐ༻ͷ Pod ϚχϑΣετΛॻ͘ •SSL ͷखܰ͞ => Service ϚχϑΣετʹ annotation Λه͢Δͱઃఆ͞ΕΔ •deploy & rollback => The UNIX Philosophy ଇͬͨπʔϧ࡞ ࣮΄΅,VCFSOFUFTͰ࣮ݱ
The UNIX Philosophy
The UNIX Philosophy ൈਮ • Small is beautiful. খ͍͞ͷඒ͍͠ •
Make each program do one thing well. 1ͭͷϓϩάϥϜʹ1ͭͷ͜ͱΛ͏·͘ ΒͤΔ • Choose portability over efficiency. ޮΑΓҠ২ੑΛ༏ઌ͢Δ • Use software leverage to your advantage. ιϑτΣΞΛᑏࢠ(ͯ͜)ͱͯ͠͏ • Avoid captive user interfaces. աͷରతΠϯλʔϑΣʔεΛආ͚Δ
Server Tools ͱ Client Tool • Ұͭͷ͜ͱΛ্ख͘Δπʔϧ࡞Γɺᑏࢠͱͯ͠ར༻ => ͍ͭͰަՄೳ •
ΫϥΠΞϯτଆ׳Ε͠Μͩύοέʔδཧ => homebrew Ͱ৴ • ΫϥΠΞϯτଆπʔϧΛݺͼग़ͨ͢Ίͷϥούʔʹઐ೦ • ΫϥΠΞϯτଆʹग़དྷΔݶΓϩδοΫΛೖΕͳ͍ => ॲཧ server ্ • ωοτஅΛఆ͢Δ => server ͱ container ҡ࣋
homebrew Golang πʔϧΛ homebrew Ͱ binary Λ৴ ΫϩείϯύΠϥͰOSґଘ͕ۃྗͳ͘πʔϧ͕͑ΔΑ͏ʹ
kube ࣾπʔϧ
खಈσϓϩΠ ϚχϑΣετϑΝΠϧʹ annotations Λه͢Δ͜ͱͰखಈσϓϩΠ͕Ͱ͖Δ k8ship deploy [BRANCH|COMMIT_SHA1] 1 apiVersion: extensions/v1beta1
2 kind: Deployment 3 metadata: 4 name: rails 5 namespace: rails 6 labels: 7 name: rails 8 role: web 9 annotations: 10 wantedly.com/deploy-target: “true" # ه 11 wantedly.com/deploy-target-container: rails # ه 12 wantedly.com/github: rails=wantedly/rails # ه https://github.com/dtan4/k8ship
kube deploy ίϚϯυ -dry-run ΦϓγϣϯͰ diff Λग़͢ merge ޙҙͷλΠϛϯάͰ deploy
͍ͨ͠߹ʹར༻ $ kube qa deploy master --dry-run [dry-run] deploy to (deployment: "rails", container: "rails") [dry-run] before: wantedly/rails:d1cb608ee61cb18f8c397c2e27576573879b2fcf [dry-run] after: wantedly/rails:cb19269526f946222110fbdfb3d107c4b5a18fe7
ϚϧνϚελʔ
ϚϧνϚελʔ •Kubernetes ͷ version up Ͳ͏͢Δͷʁ •Ϛελʔ͕ࢭ·ΔͱͲ͏ͳΔͷʁ •Ϋϥελͷఀࢭͳ͘ kops Λͬͯ
Rolling upgrade Ͱ͖ΔΑ͏ʹมߋ •ϚελʔΛࡾ༻ҙ͠ɺϚϧνϚελʔߏʹ͔͑ͯ SPOF վળ
ࣗಈϞχλϦϯά
ࣗಈϞχλϦϯά • 1ͭ1ͭઃఆ͢ΔͷͰͳ͘ɺ൚༻తʹద༻͞ΕΔΑ͏ʹ͢Δ • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}}
• request ͱ status code औಘ • μογϡϘʔυʹө • Deployment Ͱఆٛͨ͠
ࣗಈͰಉ࣭͡ͷϞχλϦϯάΛࢦ͢ https://www.datadoghq.com/blog/monitoring-101-collecting-data/ http://qiita.com/koudaiii/items/bc89368e1279649f2498
Dashboard
ϞχλϦϯάͰ༻͍ͯ͠Δͷ • kelseyhightower/konfd => PostgresqlͳͲͷใΛϦϙδτϦʹؚΊͳ͍࣌ʹར༻ • kubernetes/kube-state-metrics => deployment ͷ
available Λऔಘ • datadog/docker-dd-agent => docker ༻ɺ web ༻ɺ db ༻ͱׂͯ͠ผͰඋ • koudaiii/sltd => Service ͷ metadata Λݩʹ ELB ͷ tag ʹՃ • logentries => ΫϥελશͯͷϩάΛૹΔ
ਪҠ
αʔϏεਪҠ (2017/04Ἤ) Service 9 => 60 Container 700+ => 1,500
- 2,500+
·ͱΊ • kubernetes ɺΠϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬʹͳͬͨ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͢͠͞ • εέʔϧͷ͢͠͞
• ՝͔ΒऔΓΜͩ͜ͱ • खಈͰ͖ͬͯͨ͜ͱͷݶք => τΠϧ໓ӡಈ • Ϛελʔ͕ SPOF => ϚϧνϚελʔԽ • ϞχλϦϯάͷෳࡶ͞ => ࣗಈϞχλϦϯά
Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
IUUQTXXXXBOUFEMZDPNQSPKFDUT