Extensibility and Possibility of Kubernetes Custom Resource @ Open Source Summit Japan 2019 - Masaya Aoyama, CyberAgent, Inc / ossj2019-k8s

Extensibility and Possibility of Kubernetes Custom Resource @ Open Source Summit Japan 2019 - Masaya Aoyama, CyberAgent, Inc
------------------------------------------------------------------------------------------------------------------
Kubernetes is implemented some resources (eg. Deployment, Service and ConfigMap).
The basic resources is used for general purpose, but we need specific resources in each own domain.
For example, if developers build MySQL cluster with basic resources, startup scripts will be so complex. But the MySQL custom resource with simply manifest will make MySQL cluster as managed Database services.

Kubernetes is not only for container platform, but Kubernetes is also a framework for applications using kubebuilder, Operator SDK and so on.At the CyberAgent, we implement GPUaaS resources and some application cluster resources for sophisticated application management.

In this talk, attendees will learn about how to implement own custom resources and the controller, and I will talk about the possibility and extensibility of kubernetes.

------------------------------------------------------------------------------------------------------------------
Masaya Aoyama is a Infrastructure Engineer for CyberAgent, Inc.

And also, technical advisor of CREATIONLINE, inc., and visiting researcher of SAKURA Internet, Inc.

He is co-chair at Cloud Native Days Tokyo, and he published Kubernetes books.
And he also organizes Cloud Native Meetup Tokyo, Kubernetes Meetup Tokyo and Japanese meetup in KubeCon at the venue.
He passed Kubernetes certification #2 in the world.

Involved in Kubernetes for more than few years, he contributes for Kubernetes (reviewer).
He has been working on production private cloud, which is includes an OpenStack based Kubernetes as a Service Platform.

Transcript

1. Masaya Aoyama CyberAgent - adtech studio Extensibility and Possibility of Kubernetes Custom Resource @Open Source Summit Japan 2019 amsy810 @amsy810
2. Publicity （excerpt） 　 Books 『Kubernetes Perfect Guide』 『Docker/K8s for everyone』 　Keynotes 『Japan Container Days v18.04』 『Google Cloud K8s Day』　 　Invitations 『IPSJ Computer System Symposium』 　　　　　　　 『AWS Dev Day Tokyo』 『IBM Think Japan』 『JEITA Committee』 　 Sessions 『KubeCon + CloudNativeCon China 2019』 and so on 　Certifications 『CKAD #2』 『CKA #138』 Masaya Aoyama (@amsy810) Infrastructure Engineer Community 　 Co-chair 『Cloud Native Days Tokyo （旧 Japan Container Days）』 　Organizer 『Cloud Native Meetup Tokyo』 　 『Kubernetes Meetup Tokyo』 　 『KubeCon Japanese exchange meeting』 　 Contribute to OpenStack and Kubernetes Main job: * Implementing K8s as a Service * Architect for related K8s + CREATIONLINE - Technology Advisor + SAKURA Internet Research Center - Visiting Researcher
3. Agenda 1.  What is Kubernetes? 2.  Extends your own CustomResource with kubebuilder 3.  GPUaaS abstraction plan at CyberAgent
4. Do you know Kubernetes?
5. Do you use Kubernetes in production?
6. Do you like Kubernetes?
7. Palm-sized Personal Kubernetes Cluster Everyone has a personal Kubernetes cluster just like a smartphone, right?
8. Kubernetes is anywhere We can use phyisical Kubernetes cluster anywhere
9. Kubernetes is anywhere Be careful, this is not a bomb, this is Kubernetes.
10. What is Kubernetes?
11. What is As a “Container / Application Execution Platform “ As a “X as a Service Platform” As a “Framework for Distributed System”
12. As a “Container / Application Execution Platform” Sophisticated platform based on Google Borg •  Self-healing •  Integration with load balancer •  auto scaling •  data management •  adaption for several workload types
13. As a ”X as a Service Platform” Platform for Platform •  Database as a Service on Kubernetes •  Queue as a Service on Kubernetes •  Serverless as a Service on Kubernetes •  ML as a Service on Kubernetes oracle/mysql-operator Automate “cluster repair at failure” “operation such as backup” like a managed service
14. As a ”X as a Service Platform” Kubernetes like a “small public cloud” oracle/mysql-operator Relational DB Key Value Store Document DB Queue Developer Automate “cluster repair at failure” “operation such as backup” like a managed service
15. As a “Framework for distributed systems” “Decralative API” and “Framework for distributed system” ※ strictly, controller also create and delete pod/container via API reconcile() { … } Register (via API Request) Watch cluster state Create and delete pod Controller
16. Reconcile Loop converges to desired state  ＝ not only to launch, but also to keep the replicas Observe Diff Act Actual state (Cluster state) Desired state reconcile() { … } Controller As a “Framework for distributed systems” Observe actual state calculate Difference Action for filling difference
17. As a “Framework for distributed systems” e.g.) when only 2 pods are running at cluster Observe: desired pods=3, actual pods=2 Observe Diff Act reconcile() { … } Controller Actual state (Cluster state) Desired state Observe actual state calculate Difference Action for filling difference
18. As a “Framework for distributed systems” e.g.) when only 2 pods are running at cluster Diff: 1 pod is missing Observe Diff Act reconcile() { … } Controller Actual state (Cluster state) Desired state Observe actual state calculate Difference Action for filling difference
19. As a “Framework for distributed systems” e.g.) when only 2 pods are running at cluster Act: launch 1 pod includes a nginx:1.12 container Observe Diff Act reconcile() { … } Controller Actual state (Cluster state) Desired state Observe actual state calculate Difference Action for filling difference
20. As a “Framework for distributed systems” Reconcile Loop converges to desired state  ＝ not only launch, keep the replicas operation knowledge convert to program for automation This program is called as “Controller (Operator)” ReplicaSet Controller is working with actual replicas •  if missing, then launchs new pods •  if over, then stops some pods Observe Diff Act Observe actual state calculate Difference Action for filling difference
21. As a “Framework for distributed systems” A lot of controllers are running on Kubernetes •  ReplicaSet Controller •  Deployment Controller •  Endpoints Controller •  Cloud Controller •  etc. There are many controllers that work asynchronously and they make Kubernetes a distributed system. reconcile() { … } Controller reconcile() { … } Controller reconcile() { … } Controller reconcile() { … } Controller reconcile() { … } Controller Actual state Desired state watch
22. As a “Framework for distributed systems” “Decralative API” and “Framework for distributed system” ※ strictly, controller also create and delete pod/container via API reconcile() { … } Register (via API Request) Watch cluster state Controller Create and delete pod
23. Extending Kubernetes Resource using CustomResource CustomResource (extended resource) ※ strictly, controller also create and delete pod/container via API reconcile() { … } Register (via API Request) Watch Manage MySQL cluster Controller reconcile() { … } Controller What to do for Custom Resource Write a controller for make operation easy （Operation knowledges become a program）
24. Build your CustomResource and controller
25. How to create your Kubernetes-native app? we need to create CustomResource: 1.  Scheme (definition) 2.  Controller ※ strictly, controller also create and delete pod/container via API reconcile() { … } Register (via API Request) Watch Manage MySQL cluster Controller
26. How to create your Kubernetes-native app? ※ strictly, controller also create and delete pod/container via API reconcile() { … } Register (via API Request) Watch Manage MySQL cluster Controller reconcile() { … } Controller What to do for Custom Resource Write a controller for make operation easy （Operation knowledges become a program）
27. Example, WebServer Custom Resource we need to create CustomResource: 1.  Scheme (definition) 2.  Controller ※ strictly, controller also create and delete pod/container via API reconcile() { … } Register (via API Request) Watch Manage simple web server hosting WebServer Controller hello, ossummit! hello, ossummit! hello, ossummit!
28. kubebuilder (v1) # Initialize project $ kubebuilder init \ --domain example.com \ --license apache2 \ --owner "amsy810" # Create CustomResource scheme and controller skeleton $ kubebuilder create api \ --group servers \ --version v1beta1 \ --kind WebServer apiVersion: servers.example.com/v1beta1 kind: WebServer metadata: name: webserver-sample spec: content: "hello, ossummit!" replicas: 3 https://github.com/kubernetes-sigs/kubebuilder
29. Directory structure $ tree ./pkg/ pkg ├─ apis │ ├─ addtoscheme_servers_v1beta1.go │ ├─ apis.go │ └─ servers │ ├─ group.go │ └─ v1beta1 │ ├─ doc.go │ ├─ register.go │ ├─ v1beta1_suite_test.go │ ├─ webserver_types.go │ ├─ webserver_types_test.go │ └─ zz_generated.deepcopy.go │ ├─ controller │ ├─ add_webserver.go │ ├─ controller.go │ └─ webserver │ ├─ webserver_controller.go │ ├─ webserver_controller_suite_test.go │ └─ webserver_controller_test.go └─ webhook └─ webhook.go apiVersion: servers.example.com/v1beta1 kind: WebServer metadata: name: webserver-sample spec: content: "hello, ossummit!" replicas: 3 1.  API Scheme part 2.  Controller logic part
30. Default generated scheme code kind: WebServer metadata: name: webserver-sample webserver_types.go
31. Update CustomResource scheme kind: WebServer metadata: name: webserver-sample spec: content: "hello, ossummit!" replicas: 3 here webserver_types.go
32. Update CustomResource scheme Insert additional fileds kind: WebServer metadata: name: webserver-sample spec: content: "hello, ossummit!" replicas: 3 webserver_types.go
33. Update CustomResource scheme Of course, we can insert nested map struct kind: WebServer metadata: name: webserver-sample spec: content: "hello, ossummit!" replicas: 3 software: name: nginx version: 1.12 webserver_types.go
34. Update controller logic When CustomResource and related resources are changed, Reconcile() method is called webserver_controller.go
35. Update controller logic At the default, controller has small logic which is creating nginx Deployment webserver_controller.go
36. Update controller logic kind: WebServer metadata: name: webserver-sample spec: content: "hello, ossummit!" replicas: 3 At the default, controller has small logic which is creating nginx Deployment
37. Update controller logic “instance” variable is WebServer resoruce kind: WebServer metadata: name: webserver-sample spec: content: "hello, ossummit!" replicas: 3
38. Example, WebServer CRD we need to create CustomResource: 1.  Scheme 2.  Controller ※ strictly, controller also create and delete pod/container via API reconcile() { … } Register (via API Request) Watch Manage simple web server hosting WebServer Controller hello, ossummit! hello, ossummit! hello, ossummit!
39. Example, WebServer CRD reconcile() { … } Register (via API Request) Watch Manage Deployment WebServer Controller hello, ossummit! hello, ossummit! hello, ossummit!
40. MySQL operator reconcile() { … } Register (via API Request) Watch Manage StatefulSet MySQL Operator
41. At CyberAgent
42. GPU environment GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU with nvidia-docker Developer Developer Developer Developer Developer Developer Developer Developer GPU GPU GPU 払い出し
43. Complex YAML manifest GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU GPU with nvidia-docker Developer Developer Developer Developer Developer Developer Developer Developer GPU GPU GPU 払い出し
44. MLTask CustomResource for GPUaaS abstraction Abstract GPUaaS for ML engineers Difficult for most ML engineers to understand complex manifest Simple settings •  Which image to use •  Which training data to use •  Where to place the calculated models •  Where to place the temporary shared files •  How many parallels to run •  How many GPU resources to use
45. reconcile() { … } Register (via API Request) Watch Manage StatefulSet MLTask Controller GPU GPU GPU GPU GPU GPU MLTask CustomResource for GPUaaS abstraction
46. our company own Ingress controller L4 LB VM Network Pod Network VM . VM . Software LB cluster Ingress Controller Watch Manage
47. Conclusion
48. Conclusion As a “Container / Application Execution Platform “ As a “X as a Service Platform” As a “Framework for Distributed System” Observe Diff Act Controller (operation knowledge) CustomResource (scheme) Watch Processing
49. Thank you for your attention Let’s enjoy with Kubernetes follow me: @amsy810