agones deep dive

devfest 2023 songdo agones deep dive 발표 슬라이드입니다. 화질이 좀 깨지는데 다운로드 받으시면 잘 보여요!

Transcript

1. Agones Deep Dive Incheon/Songdo
3. Agones Deep Dive - game server on kubernetes - game server network - agones controller 어떤 기술로 Agones가 동작하는가?
4. game server on kubernetes Incheon/Songdo
6. game server on kubernetes 게임 서버 관점에서는? 비동기형 세션형 유지형 룸이 생기고 룸에 접속하여 진행하는 게임 많은 유저가 하나의 서버에 붙어 진행하는 게임 유저 별 비동기적으로 서비스하는 게임
7. game server on kubernetes 게임 서버 관점에서는? 비동기형 세션형 유지형
8. game server on kubernetes 게임 서버 관점에서는? 유저 별 비동기적으로 서비스하는 게임 룸이 생기고 룸에 접속하여 진행하는 게임 많은 유저가 하나의 서버에 붙어 진행하는 게임 다룰 주제! 비동기형 세션형 유지형
9. game server on kubernetes 세션형 서버를 구현하기 위해 필요한 요소 - 참가 대기 서버 - 매치메이킹 서버 - 게임 서버
10. game server on kubernetes 로비서버 LB resource 매치매이킹서버 LB resource 게임서버 resource 서버 할당 controller
11. kubernetes에서 게임?
12. 쿠버네티스는 컨테이너화된 워크로드와 서비스를 관리하기 위한 이식성이 있고, 확장가능한 오픈소스 플랫폼이다. kubernetes란 무엇인가? https://kubernetes.io/ko/docs/concepts/overview/
13. game server on kubernetes 왜 컨테이너 기반 게임 서버, 그리고 kubernetes를 써야 하는가? 1. 컨테이너 레벨에서 더 작은 단위로 컴퓨팅 리소스를 제한할 수 있다. 2. 방을 컨테이너로 독립시켜 다른 서버의 영향을 줄일 수 있다. 3. 이미지라는 규격화된 단위로 게임 서버를 배포할 수 있다. 4. 클라우드 종속성을 줄일 수 있다.
14. game server on kubernetes 세션형 서버를 구현하기 위해 필요한 요소 - 참가 대기 서버 - 매치메이킹 서버 - 게임 서버
15. game server on kubernetes 로비서버 LB resource 매치매이킹서버 LB resource 게임서버 resource 서버 할당 controller
16. outgame matchmaking ingame game server controller allocation
17. outgame matchmaking agones controller allocation agones fleet1 fleet2
18. game server on kubernetes Game server 게임서버를 구현한 컨테이너가 포함된 pod Fleet Game server 묶음을 정의한 설정 Fleet Auto Scaler Fleet의 크기를 확장할 수 있는 설정
19. game server on kubernetes Game server 게임서버를 구현한 컨테이너가 포함된 pod Fleet Game server 묶음을 정의한 설정 Fleet Auto Scaler Fleet의 크기를 확장할 수 있는 설정
20. fleet1 & fleet auto scaler minReplicas: 9 maxReplicas: 30 BufferSize: 9 ready: 9 바둑 서버라면?
21. fleet1 & fleet auto scaler ready: 9 유저 18명 minReplicas: 9 maxReplicas: 30 BufferSize: 9
22. fleet1 & fleet auto scaler 유저 18명 allocated: 9 minReplicas: 9 maxReplicas: 30 BufferSize: 9
23. fleet1 & fleet auto scaler allocated: 9 ready: 9 allocated: 9 minReplicas: 9 maxReplicas: 30 BufferSize: 9
24. https://youtu.be/95gQapJ7EEI
25. Game Server Network Incheon/Songdo
26. ? 그래서 이게 왜 접속돼죠?
27. Game Server Network 게임서버가 직접 호스팅된 node에서 netstat으로 조회하면? 아무것도 안나오는데요?
28. hostport 그럼 통신하는 방법은?
29. 반드시 필요한 것이 아니라면 파드에 hostPort 를 명시하지 않는다. 같은 이유로, hostNetwork를 ... 피한다. 구성 모범 사례 https://kubernetes.io/ko/docs/concepts/configuration/overview/
30. Game Server Network pod 명세를 자세히 보면? Port와 Host Port 확인 가능!
31. kubeproxy 그럼 hostport는 어떻게 동작하는가?(kind, 1.27, ipvs)
32. 7581 172.18.0.2 7654 10.244.2.9
33. Game Server Network 기본적으로 netfilter라는 linux kernel 패킷 필터링 기술을 사용한다. PREROUTING POSTROUTING FORWARD INPUT OUTPUT APPLICATION
34. Game Server Network 기본적으로 netfilter라는 linux kernel 패킷 필터링 기술을 사용한다. PREROUTING POSTROUTING FORWARD INPUT OUTPUT APPLICATION
35. 7581 172.18.0.2 7654 10.244.2.9 PREROUTING
36. Chain의 적용 규칙은 위에서 아래
37. 1 2 4 3 1번이 출발지이고, 2번이 목적지인 패킷이 3번 조건을 만족하면 4의 규칙을 적용
38. 외부에서 node IP로 접근한 경우, node port를 재외하면 해당되지 않음 X X X X
39. 7581 172.18.0.2 7654 10.244.2.9 PREROUTING chain CNI-HOSTPORT-DNAT
40. 0.0.0.0/0에서 0.0.0.0/0으로 가는 7581, 7630 port에 대하여 CNI-DN-6f07~~ 규칙이 적용된다.
41. 0.0.0.0/0에서 0.0.0.0/0으로 가는 7581 port에 대하여 10.244.2.9:7654로 DNAT시킨다. 즉, 도착지 주소를 변경시킨다.
42. 7581 172.18.0.2 7654 10.244.2.9 chain CNI-HOSTPORT-DNAT chain CNI-DN-6f0701~~ DNAT 10.244.2.9:7654 PREROUTING
43. 위의 10.244.2.9 IP주소는 vethf323f867 인터페이스로 라우팅
44. 7581 172.18.0.2 7654 10.244.2.9 chain CNI-HOSTPORT-DNAT chain CNI-DN-6f0701~~ DNAT 10.244.2.9:7654 route table 10.244.2.9 → vethf323 peer interface PREROUTING
45. 인터페이스 조회 시 해당 인터페이스가 속한 Network namespace를 확인할 수 있다!
46. Game Server Network 최종 결과: network namespace내부에서 처음에 정의한 port number를 확인할 수 있다!
47. 7581 172.18.0.2 7654 10.244.2.9 chain CNI-HOSTPORT-DNAT chain CNI-DN-6f0701~~ DNAT 10.244.2.9:7654 route table 10.244.2.9 → vethf323 network ns cni-ffee38eb-bf90-~~ peer interface PREROUTING
48. CRDs Incheon/Songdo
49. CRDs Gameserver, Fleet, FleetAutoScaler와 같은 개념은 원래 kubernetes에 없는 것! 어떻게 이를 사용할 수 있을까?
50. 커스텀리소스데피니션 API 리소스를 사용하면 커스텀 리소스를 정의할 수 있다. 커스텀 리소스 https://kubernetes.io/ko/docs/concepts/extend-kubernetes/api-extension/custom-resources/
51. CRDs agones에서 필요한 CRDs는 /install/helm/templates/crds에 존재 helm/install/agones ├── certs │ └── allocator │ └── client-ca ├── scripts └── templates ├── crds │ ├── _gameserverspecschema.yaml │ ├── _gameserverstatus.yaml │ ├── fleet.yaml │ ├── fleetautoscaler.yaml │ ├── gameserver.yaml │ ├── gameserverallocationpolicy.yaml │ ├── gameserverset.yaml │ └── k8s ├── hooks ├── service ├── serviceaccounts └── tests
52. 이제는 crds라는 특수 디렉토리가 있어서, CRD를 만들고자 할 때 차트 내에 생성할 수 있다. 리소스를 사용하기 전에 CRD 선언을 설치하기 https://helm.sh/ko/docs/chart_best_practices/custom_resource_definitions/
53. CRDs deploy other component CRDs CR deploy allocator controller ping
54. CRDs deploy other component CRDs CR deploy allocator controller ping
55. Controller Incheon/Songdo
56. CRDs는 혼자 동작하지 않는다. Controller
57. 로보틱스와 자동화에서 컨트롤루프 는 시스템 상태를 조절하는 종료되지 않는 루프이다. 컨트롤러 https://kubernetes.io/ko/docs/concepts/architecture/controller/
58. K8s API server custom Controller Controller Fleet GameServers ~~.yaml 1. 상태 변경 2. 변경 감지 3. 원하는 상태 적용
59. 구현을 살펴봅시다! (1.36) In Code
60. controller의 main 흐름
61. cmd ├── allocator │ ├── Dockerfile │ ├── main.go │ ├── main_test.go │ └── metrics.go ├── controller │ ├── Dockerfile │ ├── main.go │ ├── main_test.go │ └── pprof.go ├── extensions │ ├── Dockerfile │ ├── main.go │ └── pprof.go ├── ping │ ├── Dockerfile │ ├── main.go │ ├── udp.go │ └── udp_test.go ...
62. cmd ├── allocator │ ├── Dockerfile │ ├── main.go │ ├── main_test.go │ └── metrics.go ├── controller │ ├── Dockerfile │ ├── main.go │ ├── main_test.go │ └── pprof.go ├── extensions │ ├── Dockerfile │ ├── main.go │ └── pprof.go ├── ping │ ├── Dockerfile │ ├── main.go │ ├── udp.go │ └── udp_test.go ... Controller의 메인 진입점
63. // cmd/controller/main.go func main() { // ... agonesInformerFactory := externalversions.NewSharedInformerFactory( ... ) kubeInformerFactory := informers.NewSharedInformerFactory( ... ) var rs []runner // ... fasController := fleetautoscalers.NewController( ... ) rs = append( ... ) // ... runRunner := func(r runner) { if err := r.Run(ctx,ctlConf.NumWorkers); err != nil { ... } } // ... kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for _, r := range rs { go runRunner(r) } }
64. // cmd/controller/main.go func main() { // ... agonesInformerFactory := externalversions.NewSharedInformerFactory( ... ) kubeInformerFactory := informers.NewSharedInformerFactory( ... ) var rs []runner // ... fasController := fleetautoscalers.NewController( ... ) rs = append( ... ) // ... runRunner := func(r runner) { if err := r.Run(ctx,ctlConf.NumWorkers); err != nil { ... } } // ... kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for _, r := range rs { go runRunner(r) } } 1. Informer Factory 초기화
65. // cmd/controller/main.go func main() { // ... agonesInformerFactory := externalversions.NewSharedInformerFactory( ... ) kubeInformerFactory := informers.NewSharedInformerFactory( ... ) var rs []runner // ... fasController := fleetautoscalers.NewController( ... ) rs = append( ... ) // ... runRunner := func(r runner) { if err := r.Run(ctx,ctlConf.NumWorkers); err != nil { ... } } // ... kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for _, r := range rs { go runRunner(r) } } 2. Controller 생성
66. // cmd/controller/main.go func main() { // ... agonesInformerFactory := externalversions.NewSharedInformerFactory( ... ) kubeInformerFactory := informers.NewSharedInformerFactory( ... ) var rs []runner // ... fasController := fleetautoscalers.NewController( ... ) rs = append( ... ) // ... runRunner := func(r runner) { if err := r.Run(ctx,ctlConf.NumWorkers); err != nil { ... } } // ... kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for _, r := range rs { go runRunner(r) } } 3. Runner 추가 함수 runRunner 생성
67. // cmd/controller/main.go func main() { // ... agonesInformerFactory := externalversions.NewSharedInformerFactory( ... ) kubeInformerFactory := informers.NewSharedInformerFactory( ... ) var rs []runner // ... fasController := fleetautoscalers.NewController( ... ) rs = append( ... ) // ... runRunner := func(r runner) { if err := r.Run(ctx,ctlConf.NumWorkers); err != nil { ... } } // ... kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for _, r := range rs { go runRunner(r) } } 4. Controller 실행
68. // cmd/controller/main.go func main() { // ... agonesInformerFactory := externalversions.NewSharedInformerFactory( ... ) kubeInformerFactory := informers.NewSharedInformerFactory( ... ) var rs []runner // ... fasController := fleetautoscalers.NewController( ... ) rs = append( ... ) // ... runRunner := func(r runner) { if err := r.Run(ctx,ctlConf.NumWorkers); err != nil { ... } } // ... kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for _, r := range rs { go runRunner(r) } } 1. Informer Factory 초기화 2. Controller 생성 3. Runner 추가 함수 runRunner 생성 4. Controller 실행
69. main fasController informer workerqueue api server informerFactory 초기화 controller 생성 runner setting 실행
70. main fasController informer workerqueue api server informerFactory 초기화 controller 생성 runner setting 실행
71. code-generator 본격적으로 들어가기 전에
72. Controller kubernetes CRDs Custom Controller를 구현하기 위한 도구 · kubebuilder · kopf · code-generator
73. Controller kubernetes CRDs Custom Controller를 구현하기 위한 도구 · kubebuilder · kopf · code-generator https://github.com/kuber netes/code-generator
74. pkg/apis ├── agones │ ├── ... ├── allocation │ ├── ... ├── autoscaling │ ├── register.go │ └── v1 │ ├── doc.go │ ├── fleetautoscaler.go │ ├── fleetautoscaler_test.go │ ├── register.go │ └── zz_generated.deepcopy.go ├── doc.go ├── multicluster │ ├── ... └── scheduling.go pkg/apis code-generator에 대한 base code가 존재하는 곳! - agones - allocation - autoscaling ... 에 사용할 타입 선언
75. pkg/apis ├── agones │ ├── ... ├── allocation │ ├── ... ├── autoscaling │ ├── register.go │ └── v1 │ ├── doc.go │ ├── fleetautoscaler.go │ ├── fleetautoscaler_test.go │ ├── register.go │ └── zz_generated.deepcopy.go ├── doc.go ├── multicluster │ ├── ... └── scheduling.go pkg/apis code-generator에 대한 base code가 존재하는 곳! - agones - allocation - autoscaling ... 에 사용할 타입 선언
76. // pkg/apis/autoscaling/v1/fleetautoscale r.go // pkg/ // +genclient // +k8s:deepcopy-gen:interfaces=k8s.io/ap imachinery/pkg/runtime.Object // FleetAutoscaler is the data structure for a FleetAutoscaler resource type FleetAutoscaler struct{ metav1.TypeMeta `json: ... ` metav1.ObjectMeta `json: ... ` Spec FleetAutoscalerSpec ` ... ` Status FleetAutoscalerStatus ` ... ` } /install/helm/agones/templates/crds/fleet autoscaler.yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: ... spec: group: autoscaling.agones.dev names: ... scope: Namespaced versions: - name: v1 served: true storage: true schema: openAPIV3Schema: description: ... type: object properties: spec: ... status: ...
77. // pkg/apis/autoscaling/v1/fleetautoscale r.go // pkg/ // +genclient // +k8s:deepcopy-gen:interfaces=k8s.io/ap imachinery/pkg/runtime.Object // FleetAutoscaler is the data structure for a FleetAutoscaler resource type FleetAutoscaler struct{ metav1.TypeMeta `json: ... ` metav1.ObjectMeta `json: ... ` Spec FleetAutoscalerSpec ` ... ` Status FleetAutoscalerStatus ` ... ` } /install/helm/agones/templates/crds/fleet autoscaler.yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: ... spec: group: autoscaling.agones.dev names: ... scope: Namespaced versions: - name: v1 served: true storage: true schema: openAPIV3Schema: description: ... type: object properties: spec: ... status: ... struct ↔ yaml 대응성
78. // pkg/apis/autoscaling/v1/fleetautoscale r.go // pkg/ // +genclient // +k8s:deepcopy-gen:interfaces=k8s.io/ap imachinery/pkg/runtime.Object // FleetAutoscaler is the data structure for a FleetAutoscaler resource type FleetAutoscaler struct{ metav1.TypeMeta `json: ... ` metav1.ObjectMeta `json: ... ` Spec FleetAutoscalerSpec ` ... ` Status FleetAutoscalerStatus ` ... ` } /install/helm/agones/templates/crds/fleet autoscaler.yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: ... spec: group: autoscaling.agones.dev names: ... scope: Namespaced versions: - name: v1 served: true storage: true schema: openAPIV3Schema: description: ... type: object properties: spec: ... status: ... code generating용 주석 k8s object interface로 변환 타입과 k8s에 관한 메타데이터
79. // pkg/apis/autoscaling/v1/register.go // ... // Adds the list of known types to api.Scheme. func addKnownTypes(apiScheme *k8sruntime.Scheme) error { apiScheme.AddKnownTypes(SchemeGroupVersion, &GameServer{}, &GameServerList{}, &GameServerSet{}, &GameServerSetList{}, &Fleet{}, &FleetList{}, ) metav1.AddToGroupVersion(apiScheme,SchemeGroupVersion) return nil }
80. // pkg/apis/autoscaling/v1/register.go // ... // Adds the list of known types to api.Scheme. func addKnownTypes(apiScheme *k8sruntime.Scheme) error { apiScheme.AddKnownTypes(SchemeGroupVersion, &GameServer{}, &GameServerList{}, &GameServerSet{}, &GameServerSetList{}, &Fleet{}, &FleetList{}, ) metav1.AddToGroupVersion(apiScheme,SchemeGroupVersion) return nil } ~/register.go Add.KnownTypes(): 앞서 정의한 CRDs의 타입 등록
81. pkg/client ├── applyconfiguration │ ├── agones │ ├── autoscaling │ ├── internal │ └── multicluster ├── clientset │ └── versioned ├── informers │ └── externalversions └── listers ├── agones ├── autoscaling └── multicluster code-generator의 repository의 generate-groups.sh all ~ ~을 실행 code-generator의 결과!
82. pkg/client ├── applyconfiguration │ ├── agones │ ├── autoscaling │ ├── internal │ └── multicluster ├── clientset │ └── versioned ├── informers │ └── externalversions └── listers ├── agones ├── autoscaling └── multicluster kubernetes리소스 변화를 감시하는 역할
83. pkg/client ├── applyconfiguration │ ├── agones │ ├── autoscaling │ ├── internal │ └── multicluster ├── clientset │ └── versioned ├── informers │ └── externalversions └── listers ├── agones ├── autoscaling └── multicluster 가져온 리소스를 효율적으로 get 하는 역할
84. pkg/client ├── applyconfiguration │ ├── agones │ ├── autoscaling │ ├── internal │ └── multicluster ├── clientset │ └── versioned ├── informers │ └── externalversions └── listers ├── agones ├── autoscaling └── multicluster kubernetes config로 informer와 lister를 이용하는 역할
85. controller의 생성 다시 돌아와서...
86. // cmd/controller/main.go func main() { // ... agonesInformerFactory := externalversions.NewSharedInformerFactory( ... ) kubeInformerFactory := informers.NewSharedInformerFactory( ... ) var rs []runner // ... fasController := fleetautoscalers.NewController( ... ) rs = append( ... , fasController) // ... runRunner := func(r runner) { if err := r.Run(ctx,ctlConf.NumWorkers); err != nil { ... } } // ... kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for _, r := range rs { go runRunner(r) } } 2. Controller 생성 main에서...
87. // pkg/fleetautoscalers/controller.go // Controller is the FleetAutoscaler controller // //nolint:govet // ignore fieldalignment, singleton type Controller struct{ // ... fleetGetter typedagonesv1.FleetsGetter fleetLister listeragonesv1.FleetLister fleetSynced cache.InformerSynced fleetAutoscalerGetter typedautoscalingv1.FleetAutoscalersGetter fleetAutoscalerLister listerautoscalingv1.FleetAutoscalerLister fleetAutoscalerSynced cache.InformerSynced // ... }
88. // pkg/fleetautoscalers/controller.go // Controller is the FleetAutoscaler controller // //nolint:govet // ignore fieldalignment, singleton type Controller struct{ // ... fleetGetter typedagonesv1.FleetsGetter fleetLister listeragonesv1.FleetLister fleetSynced cache.InformerSynced fleetAutoscalerGetter typedautoscalingv1.FleetAutoscalersGetter fleetAutoscalerLister listerautoscalingv1.FleetAutoscalerLister fleetAutoscalerSynced cache.InformerSynced // ... } code-generator 결과를 사용하는 부분! fleet, fleetAutoscaler field
89. // pkg/fleetautoscalers/controller.go func NewController ( ) *Controller { autoscaler := agonesInformerFactory.Autoscaling().V1().FleetAutoscalers() fleetInformer := agonesInformerFactory.Agones().V1().Fleets() gameServers := agonesInformerFactory.Agones().V1().GameServers() c := &Controller{ fleetSynced: fleetInformer.Informer().HasSynced, fleetAutoscalerSynced: autoscaler.Informer().HasSynced, } c.workerqueue = workerqueue.NewWorkerQueueWithRateLimiter( c.syncFleetAutoscaler, ... ) ... _, _ = autoscaler.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { c.addFasThread( ... ) }, UpdateFunc: func(_,newObj interface{}) { c.updateFasThread( ... ) }, }) return c }
90. // pkg/fleetautoscalers/controller.go func NewController ( ) *Controller { autoscaler := agonesInformerFactory.Autoscaling().V1().FleetAutoscalers() fleetInformer := agonesInformerFactory.Agones().V1().Fleets() gameServers := agonesInformerFactory.Agones().V1().GameServers() c := &Controller{ fleetSynced: fleetInformer.Informer().HasSynced, fleetAutoscalerSynced: autoscaler.Informer().HasSynced, } c.workerqueue = workerqueue.NewWorkerQueueWithRateLimiter( c.syncFleetAutoscaler, ... ) ... _, _ = autoscaler.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { c.addFasThread( ... ) }, UpdateFunc: func(_,newObj interface{}) { c.updateFasThread( ... ) }, }) return c } 1. ~~Informer라는 instance생성
91. // pkg/fleetautoscalers/controller.go func NewController ( ) *Controller { autoscaler := agonesInformerFactory.Autoscaling().V1().FleetAutoscalers() fleetInformer := agonesInformerFactory.Agones().V1().Fleets() gameServers := agonesInformerFactory.Agones().V1().GameServers() c := &Controller{ fleetSynced: fleetInformer.Informer().HasSynced, fleetAutoscalerSynced: autoscaler.Informer().HasSynced, } c.workerqueue = workerqueue.NewWorkerQueueWithRateLimiter( c.syncFleetAutoscaler, ... ) ... _, _ = autoscaler.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { c.addFasThread( ... ) }, UpdateFunc: func(_,newObj interface{}) { c.updateFasThread( ... ) }, }) return c } 2. controller 생성
92. // pkg/fleetautoscalers/controller.go func NewController ( ) *Controller { autoscaler := agonesInformerFactory.Autoscaling().V1().FleetAutoscalers() fleetInformer := agonesInformerFactory.Agones().V1().Fleets() gameServers := agonesInformerFactory.Agones().V1().GameServers() c := &Controller{ fleetSynced: fleetInformer.Informer().HasSynced, fleetAutoscalerSynced: autoscaler.Informer().HasSynced, } c.workerqueue = workerqueue.NewWorkerQueueWithRateLimiter( c.syncFleetAutoscaler, ... ) ... _, _ = autoscaler.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { c.addFasThread( ... ) }, UpdateFunc: func(_,newObj interface{}) { c.updateFasThread( ... ) }, }) return c } 3. workerqueue생성 syncFleetAutoscaler() 등록
93. // pkg/fleetautoscalers/controller.go func NewController ( ) *Controller { autoscaler := agonesInformerFactory.Autoscaling().V1().FleetAutoscalers() fleetInformer := agonesInformerFactory.Agones().V1().Fleets() gameServers := agonesInformerFactory.Agones().V1().GameServers() c := &Controller{ fleetSynced: fleetInformer.Informer().HasSynced, fleetAutoscalerSynced: autoscaler.Informer().HasSynced, } c.workerqueue = workerqueue.NewWorkerQueueWithRateLimiter( c.syncFleetAutoscaler, ... ) ... _, _ = autoscaler.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { c.addFasThread( ... ) }, UpdateFunc: func(_,newObj interface{}) { c.updateFasThread( ... ) }, }) return c } 4. EventHandler(C/U/D 이벤트) 등록
94. // pkg/fleetautoscalers/controller.go func NewController ( ) *Controller { autoscaler := agonesInformerFactory.Autoscaling().V1().FleetAutoscalers() fleetInformer := agonesInformerFactory.Agones().V1().Fleets() gameServers := agonesInformerFactory.Agones().V1().GameServers() c := &Controller{ fleetSynced: fleetInformer.Informer().HasSynced, fleetAutoscalerSynced: autoscaler.Informer().HasSynced, } c.workerqueue = workerqueue.NewWorkerQueueWithRateLimiter( c.syncFleetAutoscaler, ... ) ... _, _ = autoscaler.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { c.addFasThread( ... ) }, UpdateFunc: func(_,newObj interface{}) { c.updateFasThread( ... ) }, }) return c } 1. ~~Informer라는 instance생성 2. controller 생성 3. workerqueue생성 syncFleetAutoscaler() 등록 4. EventHandler(C/U/D 이벤트) 등록
95. main fasController informer workerqueue api server ... controller 생성 타입에 대한 informer 생성
96. main fasController informer workerqueue api server ... controller 생성 타입에 대한 informer 생성 controller 인스턴스 생성
97. main fasController informer workerqueue api server ... controller 생성 타입에 대한 informer 생성 controller 인스턴스 생성 workerqueue 생성 & 핸들러 등록 informer event handler 등록 ...
98. Informer 추적 informer가 뭐하는건데?
99. // pkg/fleetautoscalers/controller.go func NewController ( ) *Controller { autoscaler := agonesInformerFactory.Autoscaling().V1().FleetAutoscalers() fleetInformer := agonesInformerFactory.Agones().V1().Fleets() gameServers := agonesInformerFactory.Agones().V1().GameServers() c := &Controller{ fleetSynced: fleetInformer.Informer().HasSynced, fleetAutoscalerSynced: autoscaler.Informer().HasSynced, } c.workerqueue = workerqueue.NewWorkerQueueWithRateLimiter( c.syncFleetAutoscaler, ... ) ... _, _ = autoscaler.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { c.addFasThread( ... ) }, UpdateFunc: func(_,newObj interface{}) { c.updateFasThread( ... ) }, }) return c } 2. controller 생성 controller에서...
100. // pkg/fleetautoscalers/controller.go func NewController ( ) *Controller { autoscaler := agonesInformerFactory.Autoscaling().V1().FleetAutoscalers() fleetInformer := agonesInformerFactory.Agones().V1().Fleets() gameServers := agonesInformerFactory.Agones().V1().GameServers() c := &Controller{ fleetSynced: fleetInformer.Informer().HasSynced, fleetAutoscalerSynced: autoscaler.Informer().HasSynced, } c.workerqueue = workerqueue.NewWorkerQueueWithRateLimiter( c.syncFleetAutoscaler, ... ) ... _, _ = autoscaler.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { c.addFasThread( ... ) }, UpdateFunc: func(_,newObj interface{}) { c.updateFasThread( ... ) }, }) return c } 내부 동작은?
101. // pkg/fleetautoscalers/controller.go func NewController ( ) *Controller { autoscaler := agonesInformerFactory.Autoscaling().V1().FleetAutoscalers() fleetInformer := agonesInformerFactory.Agones().V1().Fleets() gameServers := agonesInformerFactory.Agones().V1().GameServers() c := &Controller{ fleetSynced: fleetInformer.Informer().HasSynced, fleetAutoscalerSynced: autoscaler.Informer().HasSynced, } c.workerqueue = workerqueue.NewWorkerQueueWithRateLimiter( c.syncFleetAutoscaler, ... ) ... _, _ = autoscaler.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { c.addFasThread( ... ) }, UpdateFunc: func(_,newObj interface{}) { c.updateFasThread( ... ) }, }) return c } 내부 동작은? 1. informerFactory에 informer등록 2. 초기화, 감시함수 등록
102. func ( f * fleetAutoscalerInformer ) defaultInformer ( client versioned . Interface , resyncPeriod time.Duration)cache.SharedIndexInformer { returnNewFilteredFleetAutoscalerInformer(client,f.namespace,resyncPeriod,cache.Indexers{cache.NamespaceI ndex:cache.MetaNamespaceIndexFunc},f.tweakListOptions) } func(f *fleetAutoscalerInformer)Informer()cache.SharedIndexInformer { returnf.factory.InformerFor(&autoscalingv1.FleetAutoscaler{},f.defaultInformer) } func(f *fleetAutoscalerInformer)Lister()v1.FleetAutoscalerLister { returnv1.NewFleetAutoscalerLister(f.Informer().GetIndexer()) } // pkg/client/informers/externalversions/autoscaling/v1/fleetautoscaler.go ... func NewFilteredFleetAutoscalerInformer(...) cache.SharedIndexInformer { return cache.NewSharedIndexInformer( &cache.ListWatch{ ListFunc:func(options metav1.ListOptions)(runtime.Object, error){ ... }, WatchFunc:func(options metav1.ListOptions)(watch.Interface, error){ ... }, }, &autoscalingv1.FleetAutoscaler{}, resyncPeriod, indexers, ) }
103. func ( f * fleetAutoscalerInformer ) defaultInformer ( client versioned . Interface , resyncPeriod time.Duration)cache.SharedIndexInformer { returnNewFilteredFleetAutoscalerInformer(client,f.namespace,resyncPeriod,cache.Indexers{cache.NamespaceI ndex:cache.MetaNamespaceIndexFunc},f.tweakListOptions) } func(f *fleetAutoscalerInformer)Informer()cache.SharedIndexInformer { returnf.factory.InformerFor(&autoscalingv1.FleetAutoscaler{},f.defaultInformer) } func(f *fleetAutoscalerInformer)Lister()v1.FleetAutoscalerLister { returnv1.NewFleetAutoscalerLister(f.Informer().GetIndexer()) } // pkg/client/informers/externalversions/autoscaling/v1/fleetautoscaler.go ... func NewFilteredFleetAutoscalerInformer(...) cache.SharedIndexInformer { return cache.NewSharedIndexInformer( &cache.ListWatch{ ListFunc:func(options metav1.ListOptions)(runtime.Object, error){ ... }, WatchFunc:func(options metav1.ListOptions)(watch.Interface, error){ ... }, }, &autoscalingv1.FleetAutoscaler{}, resyncPeriod, indexers, ) } 초기화함수 ListFunc: 처음에 리소스 리스트를 가져옴 감시함수 WatchFunc: 리소스를 감시함
104. main fasController informer workerqueue api server informerFactory 초기화 ... ... controller 인스턴스 생성 ListFunc & WatchFunc 등록 ... ... informerFactory 초기화 Factory에 informer등록
105. 확인할 리소스 리스트를 가져오고 리소스 감시 informer가 뭐하는건데?
106. Informer생성과 실행을 관리하는 Factory 그럼 informerFactory는?
107. CR에 대한 Informer 생성 - FleetAutoscalerInformer - FleetInformer - GameServerInformer fleetautoscaler.NewController() Controller 생성 Factory에 Informer 등록 Informer생성 & 아래 함수 등록 - ListFunc - WatchFunc workqueue 생성 SyncHandler 함수 등록 - syncFleetAutoscaler() EventHandler 등록 CR의 CUD함수 등록
108. 생성 다음 부분! controller 실행
109. // cmd/controller/main.go func main() { // ... agonesInformerFactory := externalversions.NewSharedInformerFactory( ... ) kubeInformerFactory := informers.NewSharedInformerFactory( ... ) var rs []runner // ... fasController := fleetautoscalers.NewController( ... ) rs = append( ... ) // ... runRunner := func(r runner) { if err := r.Run(ctx,ctlConf.NumWorkers); err != nil { ... } } // ... kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for _, r := range rs { go runRunner(r) } } 4. Controller 실행 main에서...
110. // cmd/controller/main.go func main() { // ... agonesInformerFactory := externalversions.NewSharedInformerFactory( ... ) kubeInformerFactory := informers.NewSharedInformerFactory( ... ) var rs []runner // ... fasController := fleetautoscalers.NewController( ... ) rs = append( ... ) // ... runRunner := func(r runner) { if err := r.Run(ctx,ctlConf.NumWorkers); err != nil { ... } } // ... kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for _, r := range rs { go runRunner(r) } } Factory의 Start()의 역할?
111. // pkg/client/informers/externalversions/autoscaling/factory.go func(f *sharedInformerFactory) Start(stopCh <-chan struct{}){ f.lock.Lock() defer f.lock.Unlock() if f.shuttingDown { return } for informerType,informer := range f.informers { if !f.startedInformers[informerType] { f.wg.Add(1) // ... informer := informer go func(){ defer f.wg.Done() informer.Run(stopCh) }() f.startedInformers[informerType] = true } } }
112. // pkg/client/informers/externalversions/autoscaling/factory.go func(f *sharedInformerFactory) Start(stopCh <-chan struct{}){ f.lock.Lock() defer f.lock.Unlock() if f.shuttingDown { return } for informerType,informer := range f.informers { if !f.startedInformers[informerType] { f.wg.Add(1) // ... informer := informer go func(){ defer f.wg.Done() informer.Run(stopCh) }() f.startedInformers[informerType] = true } } } 해당 부분에서 기존 등록된 informer의 Run() 메서드를 호출
113. // pkg/client/informers/externalversions/autoscaling/factory.go func(f *sharedInformerFactory) Start(stopCh <-chan struct{}){ f.lock.Lock() defer f.lock.Unlock() if f.shuttingDown { return } for informerType,informer := range f.informers { if !f.startedInformers[informerType] { f.wg.Add(1) // ... informer := informer go func(){ defer f.wg.Done() informer.Run(stopCh) }() f.startedInformers[informerType] = true } } } 해당 부분에서 기존 등록된 informer의 Run() 메서드를 호출 - k8s.io shared_informer.go (*sharedIndexInformer)Run() - k8s.io controller.go wg.StartWithChannel(stopCh, r.Run) - k8s.io reflector.go r.ListAndWatch(stopChz) - k8s.io reflector.go err = r.list(stopCh) - k8s.io reflector.go r.watch(w, stopCh, resyncerrc)
114. // pkg/client/informers/externalversions/autoscaling/factory.go func(f *sharedInformerFactory) Start(stopCh <-chan struct{}){ f.lock.Lock() defer f.lock.Unlock() if f.shuttingDown { return } for informerType,informer := range f.informers { if !f.startedInformers[informerType] { f.wg.Add(1) // ... informer := informer go func(){ defer f.wg.Done() informer.Run(stopCh) }() f.startedInformers[informerType] = true } } } 해당 부분에서 기존 등록된 informer의 Run() 메서드를 호출 - k8s.io shared_informer.go (*sharedIndexInformer)Run() - k8s.io controller.go wg.StartWithChannel(stopCh, r.Run) - k8s.io reflector.go r.ListAndWatch(stopChz) - k8s.io reflector.go err = r.list(stopCh) - k8s.io reflector.go r.watch(w, stopCh, resyncerrc)
115. main fasController informer workerqueue api server 실행 informer의 List & WatchFunc 실행 kubernetes CR 감시 ... informerFactory.Start()
116. Controller 1. informer factory가 start하며 기존 등록한 informer를 초기화하고 감시를 시작
117. // cmd/controller/main.go func main() { // ... agonesInformerFactory := externalversions.NewSharedInformerFactory( ... ) kubeInformerFactory := informers.NewSharedInformerFactory( ... ) var rs []runner // ... fasController := fleetautoscalers.NewController( ... ) rs = append( ... ) // ... runRunner := func(r runner) { if err := r.Run(ctx,ctlConf.NumWorkers); err != nil { ... } } // ... kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for _, r := range rs { go runRunner(r) } } 4. Controller 실행
118. // cmd/controller/main.go func main() { // ... agonesInformerFactory := externalversions.NewSharedInformerFactory( ... ) kubeInformerFactory := informers.NewSharedInformerFactory( ... ) var rs []runner // ... fasController := fleetautoscalers.NewController( ... ) rs = append( ... ) // ... runRunner := func(r runner) { if err := r.Run(ctx,ctlConf.NumWorkers); err != nil { ... } } // ... kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for _, r := range rs { go runRunner(r) } } runRunner의 내부에서 일어나는 일은?
119. // cmd/controller/main.go func main() { // ... agonesInformerFactory := externalversions.NewSharedInformerFactory( ... ) kubeInformerFactory := informers.NewSharedInformerFactory( ... ) var rs []runner // ... fasController := fleetautoscalers.NewController( ... ) rs = append( ... , fasController) // ... runRunner := func(r runner) { if err := r.Run(ctx,ctlConf.NumWorkers); err != nil { ... } } // ... kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for _, r := range rs { go runRunner(r) } } runRunner의 내부에서 일어나는 일은? 위에 Controller마다 정의된 Run()메서드를 호출!
120. main fasController informer workerqueue api server informer의 List & WatchFunc 실행 informerFactory.Start() ... RunRunner() (*controller) Run() kubernetes CR 감시
121. 그럼 Run()안에서는? Controller.Run()
122. // pkg/fleetautoscalers/controller.go func(c *Controller) Run(ctx context.Context,workers int) error { err := crd.WaitForEstablishedCRD( ... ) // ... c.workerqueue.Run(ctx,workers) return nil }
123. // pkg/fleetautoscalers/controller.go func(c *Controller) Run(ctx context.Context,workers int) error { err := crd.WaitForEstablishedCRD( ... ) // ... c.workerqueue.Run(ctx,workers) return nil } 1.CRD 배포 대기 2. workerqueue 실행
124. main fasController informer workerqueue api server informer의 List & WatchFunc 실행 informerFactory.Start() ... RunRunner() (*controller) Run() kubernetes CR 감시 workerqueue.Run()
125. workerqueue.Run()
126. // pkg/utils/workerqueue/workerqueue.go func(wq *WorkerQueue) Run(ctx context.Context,workers int){ // ... for i := 0; i < workers; i++{ go wq.run(ctx) } // ... } func(wq *WorkerQueue) run(ctx context.Context){ // ... wait.Until(func() {wq.runWorker(ctx)}, workFx, ctx.Done()) }
127. // pkg/utils/workerqueue/workerqueue.go func(wq *WorkerQueue) Run(ctx context.Context,workers int){ // ... for i := 0; i < workers; i++{ go wq.run(ctx) } // ... } func(wq *WorkerQueue) run(ctx context.Context){ // ... wait.Until(func() {wq.runWorker(ctx)}, workFx, ctx.Done()) } 여러 goroutine이 run을 실행
128. // pkg/utils/workerqueue/workerqueue.go func(wq *WorkerQueue) Run(ctx context.Context,workers int){ // ... for i := 0; i < workers; i++{ go wq.run(ctx) } // ... } func(wq *WorkerQueue) run(ctx context.Context){ // ... wait.Until(func() {wq.runWorker(ctx)}, workFx, ctx.Done()) } 여러 goroutine이 run을 실행 반복적으로 wq.runWorker()를 종료될 때까지 호출
129. // pkg/utils/workerqueue/workerqueue.go func(wq *WorkerQueue) runWorker(ctx context.Context) { for wq.processNextWorkItem(ctx) {} } func(wq *WorkerQueue) processNextWorkItem(ctx context.Context) bool { obj, quit := wq.queue.Get() if quit { return false } // ... if key, ok = obj.(string); !ok { // ... return true } if err := wq.SyncHandler(ctx,key); err !=nil { // ... return true } wq.queue.Forget(obj) return true }
130. // pkg/utils/workerqueue/workerqueue.go func(wq *WorkerQueue) runWorker(ctx context.Context) { for wq.processNextWorkItem(ctx) {} } func(wq *WorkerQueue) processNextWorkItem(ctx context.Context) bool { obj, quit := wq.queue.Get() if quit { return false } // ... if key, ok = obj.(string); !ok { // ... return true } if err := wq.SyncHandler(ctx,key); err !=nil { // ... return true } wq.queue.Forget(obj) return true } 내부에서 wq.processNextWorkItem()이 false가 될 때까지 실행
131. // pkg/utils/workerqueue/workerqueue.go func(wq *WorkerQueue) runWorker(ctx context.Context) { for wq.processNextWorkItem(ctx) {} } func(wq *WorkerQueue) processNextWorkItem(ctx context.Context) bool { obj, quit := wq.queue.Get() if quit { return false } // ... if key, ok = obj.(string); !ok { // ... return true } if err := wq.SyncHandler(ctx,key); err !=nil { // ... return true } wq.queue.Forget(obj) return true } queue에서 원소를 blocking 방식으로 pop
132. // pkg/utils/workerqueue/workerqueue.go func(wq *WorkerQueue) runWorker(ctx context.Context) { for wq.processNextWorkItem(ctx) {} } func(wq *WorkerQueue) processNextWorkItem(ctx context.Context) bool { obj, quit := wq.queue.Get() if quit { return false } // ... if key, ok = obj.(string); !ok { // ... return true } if err := wq.SyncHandler(ctx,key); err !=nil { // ... return true } wq.queue.Forget(obj) return true } 1. obj를 key로 변경 2. 앞서 등록한 SyncHandler를 호출 3. 여기서는 SyncFleetAutoscaler()가 정상종료될 때까지 동작!
133. main fasController informer workerqueue api server informer의 List & WatchFunc 실행 (*controller) Run() kubernetes CR 감시 workerqueue.Run() queue blocking
134. Controller 1. informer factory가 start하며 기존 등록한 informer를 초기화하고 감시를 시작 2. worker queue가 비동기적으로 실행되면서 queue에 입력이 들어오면 SyncHandler를 호출
135. 그럼 queue에 무엇이 들어오는가?
136. // pkg/fleetautoscalers/controller.go func NewController ( ) *Controller { autoscaler := agonesInformerFactory.Autoscaling().V1().FleetAutoscalers() fleetInformer := agonesInformerFactory.Agones().V1().Fleets() gameServers := agonesInformerFactory.Agones().V1().GameServers() c := &Controller{ fleetSynced: fleetInformer.Informer().HasSynced, fleetAutoscalerSynced: autoscaler.Informer().HasSynced, } c.workerqueue = workerqueue.NewWorkerQueueWithRateLimiter( c.syncFleetAutoscaler, ... ) ... _, _ = autoscaler.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { c.addFasThread( ... ) }, UpdateFunc: func(_,newObj interface{}) { c.updateFasThread( ... ) }, }) return c } 4. EventHandler(C/U/D 이벤트) 등록
137. // pkg/fleetautoscalers/controller.go func NewController ( ) *Controller { autoscaler := agonesInformerFactory.Autoscaling().V1().FleetAutoscalers() fleetInformer := agonesInformerFactory.Agones().V1().Fleets() gameServers := agonesInformerFactory.Agones().V1().GameServers() c := &Controller{ fleetSynced: fleetInformer.Informer().HasSynced, fleetAutoscalerSynced: autoscaler.Informer().HasSynced, } c.workerqueue = workerqueue.NewWorkerQueueWithRateLimiter( c.syncFleetAutoscaler, ... ) ... _, _ = autoscaler.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { c.addFasThread( ... ) }, UpdateFunc: func(_,newObj interface{}) { c.updateFasThread( ... ) }, }) return c } queue에 무엇이 들어오는지 로직에서 결정! FasController는 FleetAutoScaler가 들어온다. Informer 내부에서 확인 가능
138. // pkg/fleetautoscalers/controller.go func(c *Controller) addFasThread(fas *autoscalingv1.FleetAutoscaler, lock bool){ // ... c.workerqueue.Enqueue(fas) go func() { wait.Until(func() { c.workerqueue.Enqueue(fas) }, duration,ctx.Done()) }() }
139. // pkg/fleetautoscalers/controller.go func(c *Controller) addFasThread(fas *autoscalingv1.FleetAutoscaler, lock bool){ // ... c.workerqueue.Enqueue(fas) go func() { wait.Until(func() { c.workerqueue.Enqueue(fas) }, duration,ctx.Done()) }() } queue에 fas(fleetAutoScaler)를 지속적으로 push ㅊ
140. main fasController informer workerqueue api server kubernetes CR 감시 workerqueue.Run() queue blocking fas Create 발생 Enqueue(fas) Enqueue(fas) Enqueue(fas) addFasThread()
141. Controller 1. informer factory가 start하며 기존 등록한 informer를 초기화하고 감시를 시작 2. worker queue가 비동기적으로 실행되면서 queue에 입력이 들어오면 SyncHandler를 호출 3. Create / Update가 호출될 때 queue에 지속적으로 입력하는 Thread 추가(사실 goroutine)
142. SyncHandler 동작
143. main fasController informer workerqueue api server Enqueue(fas) Enqueue(fas) Enqueue(fas) addFasThread() 큐에서 원소가 나오면?
144. // pkg/fleetautoscalers/controller.go func (c *Controller) syncFleetAutoscaler(ctx context.Context, key string) error { // ... fleet, err := c.fleetLister.Fleets(fas.Namespace).Get(fas.Spec.FleetName) // ... currentReplicas := fleet.Status.Replicas desiredReplicas, scalingLimited, err := computeDesiredFleetSize( ... ) // ... if err = c.scaleFleet(ctx,fas,fleet,desiredReplicas); err != nil { return errors.Wrapf( ... ) } return c.updateStatus( // ... ) }
145. // pkg/fleetautoscalers/controller.go func (c *Controller) syncFleetAutoscaler(ctx context.Context, key string) error { // ... fleet, err := c.fleetLister.Fleets(fas.Namespace).Get(fas.Spec.FleetName) // ... currentReplicas := fleet.Status.Replicas desiredReplicas, scalingLimited, err := computeDesiredFleetSize( ... ) // ... if err = c.scaleFleet(ctx,fas,fleet,desiredReplicas); err != nil { return errors.Wrapf( ... ) } return c.updateStatus( // ... ) } 1. Fleet의 상태를 가져와서
146. // pkg/fleetautoscalers/controller.go func (c *Controller) syncFleetAutoscaler(ctx context.Context, key string) error { // ... fleet, err := c.fleetLister.Fleets(fas.Namespace).Get(fas.Spec.FleetName) // ... currentReplicas := fleet.Status.Replicas desiredReplicas, scalingLimited, err := computeDesiredFleetSize( ... ) // ... if err = c.scaleFleet(ctx,fas,fleet,desiredReplicas); err != nil { return errors.Wrapf( ... ) } return c.updateStatus( // ... ) } 2. Fleet의 Size 계산
147. // pkg/fleetautoscalers/controller.go func (c *Controller) syncFleetAutoscaler(ctx context.Context, key string) error { // ... fleet, err := c.fleetLister.Fleets(fas.Namespace).Get(fas.Spec.FleetName) // ... currentReplicas := fleet.Status.Replicas desiredReplicas, scalingLimited, err := computeDesiredFleetSize( ... ) // ... if err = c.scaleFleet(ctx,fas,fleet,desiredReplicas); err != nil { return errors.Wrapf( ... ) } return c.updateStatus( // ... ) } 3. Fleet 스케일링
148. // pkg/fleetautoscalers/controller.go func (c *Controller) syncFleetAutoscaler(ctx context.Context, key string) error { // ... fleet, err := c.fleetLister.Fleets(fas.Namespace).Get(fas.Spec.FleetName) // ... currentReplicas := fleet.Status.Replicas desiredReplicas, scalingLimited, err := computeDesiredFleetSize( ... ) // ... if err = c.scaleFleet(ctx,fas,fleet,desiredReplicas); err != nil { return errors.Wrapf( ... ) } return c.updateStatus( // ... ) } fCopy.Spec.Replicas=replicas fCopy,err:=c.fleetGetter.Fleets( ... ).Update( ... ) Fleet의 replica 업데이트 3. Fleet 스케일링
149. // pkg/fleetautoscalers/controller.go func (c *Controller) syncFleetAutoscaler(ctx context.Context, key string) error { // ... fleet, err := c.fleetLister.Fleets(fas.Namespace).Get(fas.Spec.FleetName) // ... currentReplicas := fleet.Status.Replicas desiredReplicas, scalingLimited, err := computeDesiredFleetSize( ... ) // ... if err = c.scaleFleet(ctx,fas,fleet,desiredReplicas); err != nil { return errors.Wrapf( ... ) } return c.updateStatus( // ... ) } 4. 상태 업데이트 fas의 상태 갱신
150. // pkg/fleetautoscalers/controller.go func (c *Controller) syncFleetAutoscaler(ctx context.Context, key string) error { // ... fleet, err := c.fleetLister.Fleets(fas.Namespace).Get(fas.Spec.FleetName) // ... currentReplicas := fleet.Status.Replicas desiredReplicas, scalingLimited, err := computeDesiredFleetSize( ... ) // ... if err = c.scaleFleet(ctx,fas,fleet,desiredReplicas); err != nil { return errors.Wrapf( ... ) } return c.updateStatus( // ... ) } 1. Fleet의 상태를 가져와서 2. Fleet의 Size 계산 3. Fleet 스케일링 4. 상태 업데이트 fas의 상태 갱신
151. main fasController informer workerqueue api server Enqueue(fas) addFasThread() syncFleetAutoScaler() Fleet 상태 Get Fleet Size 계산 Fleet 스케일링 상태 업데이트
152. Controller 1. informer factory가 start하며 기존 등록한 informer를 초기화하고 감시를 시작 2. worker queue가 비동기적으로 실행되면서 queue에 입력이 들어오면 SyncHandler를 호출 3. Create / Update가 호출될 때 queue에 지속적으로 입력하는 Thread 추가(사실 goroutine) 4. 등록한 SyncHandler가 지속 호출되면서 syncFleetAutoScaler(~)실행 Fleet을 가져와서 → 사이즈 계산 → Scale → 상태 업데이트
153. InformerFactory.Start() kubernetes 리소스 초기화 & 감시 main()에서 controller 실행 감시 실행
154. workerQueue queue 실행 감시 실행 InformerFactory.Start() kubernetes 리소스 초기화 & 감시 main()에서 controller 실행 Controller.Run() workerqueue.Run()
155. workerQueue queue 실행 감시 실행 Create/Update/Delete Callback 호출 Add InformerFactory.Start() kubernetes 리소스 초기화 & 감시 main()에서 controller 실행 Controller.Run() workerqueue.Run()
156. InformerFactory.Start() kubernetes 리소스 초기화 & 감시 main()에서 controller 실행 Controller.Run() workerqueue.Run() workerQueue queue 실행 감시 실행 Create/Update/Delete Callback 호출 Add Get SyncHandler() syncFleetAutoScaler Fleets().Get() computeDesiredFleetSize()로 사이즈 계산 c.scaleFleet c.updateStatus
157. main fasController informer workerqueue api server informerFactory 초기화 controller 생성 타입에 대한 informer 생성 controller 인스턴스 생성 workerqueue 생성 & 핸들러 등록 informer event handler 등록 ListFunc & WatchFunc 등록
158. main fasController informer workerqueue api server informer의 List & WatchFunc 실행 informerFactory.Start() RunRunner() (*controller) Run() kubernetes CR 감시 workerqueue.Run() queue blocking fas Create 발생 runner setting 실행
159. main fasController informer workerqueue api server Enqueue(fas) addFasThread() syncFleetAutoScaler() ... Fleet 상태 Get Fleet Size 계산 Fleet 스케일링 상태 업데이트
160. Kubernetes Leader Election Incheon/Songdo
161. func ( f * fleetAutoscalerInformer ) defaultInformer ( client versioned . Interface , resyncPeriod time.Duration)cache.SharedIndexInformer { returnNewFilteredFleetAutoscalerInformer(client,f.namespace,resyncPeriod,cache.Indexers{cache.NamespaceI ndex:cache.MetaNamespaceIndexFunc},f.tweakListOptions) } func(f *fleetAutoscalerInformer)Informer()cache.SharedIndexInformer { returnf.factory.InformerFor(&autoscalingv1.FleetAutoscaler{},f.defaultInformer) } func(f *fleetAutoscalerInformer)Lister()v1.FleetAutoscalerLister { returnv1.NewFleetAutoscalerLister(f.Informer().GetIndexer()) } // cmd/controller/main.go // main starts the operator for the gameserver CRD func main() { // ... whenLeader( ... , func(ctx context.Context) { kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for_, r := range rs { go runRunner(r) } <-ctx.Done() logger.Info("Shut down agones controllers") }) }
162. func ( f * fleetAutoscalerInformer ) defaultInformer ( client versioned . Interface , resyncPeriod time.Duration)cache.SharedIndexInformer { returnNewFilteredFleetAutoscalerInformer(client,f.namespace,resyncPeriod,cache.Indexers{cache.NamespaceI ndex:cache.MetaNamespaceIndexFunc},f.tweakListOptions) } func(f *fleetAutoscalerInformer)Informer()cache.SharedIndexInformer { returnf.factory.InformerFor(&autoscalingv1.FleetAutoscaler{},f.defaultInformer) } func(f *fleetAutoscalerInformer)Lister()v1.FleetAutoscalerLister { returnv1.NewFleetAutoscalerLister(f.Informer().GetIndexer()) } // cmd/controller/main.go // main starts the operator for the gameserver CRD func main() { // ... whenLeader( ... , func(ctx context.Context) { kubeInformerFactory.Start(ctx.Done()) agonesInformerFactory.Start(ctx.Done()) for_, r := range rs { go runRunner(r) } <-ctx.Done() logger.Info("Shut down agones controllers") }) } whenLeader?
163. raft 분산 합의에 등장하는 Leader Election?
164. Kubernetes Leader Election raft 분산합의 알고리즘에서는 합의에 참여하는 Follower의 상태를 Leader라는 대상이 동기화해주기 위해 리더 선출 과정을 거친다. ex: etcd
165. Kubernetes Leader Election 그러나 Kubernetes의 client-go에서 사용하는 Leader election은 합의에 참여하는 대상의 상태 동기화의 관점이라기보다 여러 pod중 하나의 pod만 활성화하기 위해 사용한다. ex: 보고서를 하나의 pod만 자동으로 만든다거나 controller를 여러 대 관리하거나
166. func ( f * fleetAutoscalerInformer ) defaultInformer ( client versioned . Interface , resyncPeriod time.Duration)cache.SharedIndexInformer { returnNewFilteredFleetAutoscalerInformer(client,f.namespace,resyncPeriod,cache.Indexers{cache.NamespaceI ndex:cache.MetaNamespaceIndexFunc},f.tweakListOptions) } func(f *fleetAutoscalerInformer)Informer()cache.SharedIndexInformer { returnf.factory.InformerFor(&autoscalingv1.FleetAutoscaler{},f.defaultInformer) } func(f *fleetAutoscalerInformer)Lister()v1.FleetAutoscalerLister { returnv1.NewFleetAutoscalerLister(f.Informer().GetIndexer()) } // cmd/controller/main.go func whenLeader( ... ){ // ... id := uuid.New().String() lock := &resourcelock.LeaseLock{ LeaseMeta:metav1.ObjectMeta{ Name: "agones-controller-lock", Namespace: namespace, }, Client: kubeClient.CoordinationV1(), LockConfig: resourcelock.ResourceLockConfig{ Identity: id, }, }
167. func ( f * fleetAutoscalerInformer ) defaultInformer ( client versioned . Interface , resyncPeriod time.Duration)cache.SharedIndexInformer { returnNewFilteredFleetAutoscalerInformer(client,f.namespace,resyncPeriod,cache.Indexers{cache.NamespaceI ndex:cache.MetaNamespaceIndexFunc},f.tweakListOptions) } func(f *fleetAutoscalerInformer)Informer()cache.SharedIndexInformer { returnf.factory.InformerFor(&autoscalingv1.FleetAutoscaler{},f.defaultInformer) } func(f *fleetAutoscalerInformer)Lister()v1.FleetAutoscalerLister { returnv1.NewFleetAutoscalerLister(f.Informer().GetIndexer()) } // cmd/controller/main.go func whenLeader( ... ){ // ... id := uuid.New().String() lock := &resourcelock.LeaseLock{ LeaseMeta:metav1.ObjectMeta{ Name: "agones-controller-lock", Namespace: namespace, }, Client: kubeClient.CoordinationV1(), LockConfig: resourcelock.ResourceLockConfig{ Identity: id, }, } k8s.io/client-go의 leaseLock 사용해서 Lock instance 생성
168. 리스는 쿠버네티스에서도 특정 시간 동안 컴포넌트의 인스턴스 하나만 실행되도록 보장하는 데에도 사용된다. 리스(Lease) https://kubernetes.io/ko/docs/concepts/architecture/leases/
169. Lease agones-controller-lock Leader update time & resource version agones-controller
170. func ( f * fleetAutoscalerInformer ) defaultInformer ( client versioned . Interface , resyncPeriod time.Duration)cache.SharedIndexInformer { returnNewFilteredFleetAutoscalerInformer(client,f.namespace,resyncPeriod,cache.Indexers{cache.NamespaceI ndex:cache.MetaNamespaceIndexFunc},f.tweakListOptions) } func(f *fleetAutoscalerInformer)Informer()cache.SharedIndexInformer { returnf.factory.InformerFor(&autoscalingv1.FleetAutoscaler{},f.defaultInformer) } func(f *fleetAutoscalerInformer)Lister()v1.FleetAutoscalerLister { returnv1.NewFleetAutoscalerLister(f.Informer().GetIndexer()) } // cmd/controller/main.go leaderelection.RunOrDie(ctx,leaderelection.LeaderElectionConfig{ Lock: lock, Callbacks: leaderelection.LeaderCallbacks{ OnStartedLeading: start, OnStoppedLeading: func() { ... }, OnNewLeader: func(identity string) { ... }, }, }) }
171. func ( f * fleetAutoscalerInformer ) defaultInformer ( client versioned . Interface , resyncPeriod time.Duration)cache.SharedIndexInformer { returnNewFilteredFleetAutoscalerInformer(client,f.namespace,resyncPeriod,cache.Indexers{cache.NamespaceI ndex:cache.MetaNamespaceIndexFunc},f.tweakListOptions) } func(f *fleetAutoscalerInformer)Informer()cache.SharedIndexInformer { returnf.factory.InformerFor(&autoscalingv1.FleetAutoscaler{},f.defaultInformer) } func(f *fleetAutoscalerInformer)Lister()v1.FleetAutoscalerLister { returnv1.NewFleetAutoscalerLister(f.Informer().GetIndexer()) } // cmd/controller/main.go leaderelection.RunOrDie(ctx,leaderelection.LeaderElectionConfig{ Lock: lock, Callbacks: leaderelection.LeaderCallbacks{ OnStartedLeading: start, OnStoppedLeading: func() { ... }, OnNewLeader: func(identity string) { ... }, }, }) } OnStartedLeading: 리더로 선출됐을 때 OnStoppedLeading: 리더에서 내려왔을 때 OnNewLeader: 새로운 리더가 당선됐을 때 callback을 등록해서 시작
172. Kubernetes Leader Election Lease 오브젝트 확인해보기 Leader로 할당된 pod를 제거하면 Holder Identity가 변경되고 다른 pod가 Leader로 선출된다.
174. Agones Deep Dive Incheon/Songdo