Slide 1

Slide 1 text

Prototyping Kubernetes CRI(Container Runtime Interface) With Ruby and gRPC RubyConf India 2018 Kei Sawada(@remore)

Slide 2

Slide 2 text

How many of you guys Are using container for your work? Are using kubernetes for your work already? Have ever implemented your own Kubernetes CRI implementation?

Slide 3

Slide 3 text

Hi RubyConf India

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

Container Runtime Interface

Slide 6

Slide 6 text

? ? ?

Slide 7

Slide 7 text

Container Runtime == Docker ?

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

Not Exactly The Same

Slide 10

Slide 10 text

Docker is one of implementations of Container Runtime Container Runtime is something you use to manipulate container images There are many docker alternatives out there such as runc, rkt, and runv. What is important about them is, they are interchangeable since they all are compatible with OCI speci cation, which is a standard speci cation of container runtime and image. OCI stands for Open Container Initiative

Slide 11

Slide 11 text

Then What Is Container Runtime Interface?

Slide 12

Slide 12 text

In Kubernetes, it is de ned like this: CRI, Container Runtime Interface provides a set of APIs you can call from client application. The original image is taken from http://blog.kubernetes.io/2016/12/container-runtime-interface-cri-in- kubernetes.html

Slide 13

Slide 13 text

In Kubernetes, it is de ned like this: And actually, APIs are called over gRPC.

Slide 14

Slide 14 text

Now We Know Docker is a one of Container Runtime, but not "the only one" runtime Kubernetes is using gRPC to manage Container Runtime, and spec name for that is CRI

Slide 15

Slide 15 text

OK Now We Know CRI Works Over gRPC. But What Kind of APIs Are Available Actually?

Slide 16

Slide 16 text

Since we are Dev, we reading API Docs. Don't we? I do.

Slide 17

Slide 17 text

CRI Protocol Buffer's IDL CRI APIs consists of 2 services( RuntimeService and ImageService ). In the RuntimeService , it looks like we can Run/Stop/Remove PodSandbox... service RuntimeService { // Managing PodSandbox rpc RunPodSandbox(RunPodSandboxRequest) returns (RunPodSandb rpc StopPodSandbox(StopPodSandboxRequest) returns (StopPodSa rpc RemovePodSandbox(RemovePodSandboxRequest) returns (Remov rpc PodSandboxStatus(PodSandboxStatusRequest) returns (PodSa rpc ListPodSandbox(ListPodSandboxRequest) returns (ListPodSa https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/v1alpha1/runtime/api.prot

Slide 18

Slide 18 text

And we can Create/Start/Stop/Remove Container... // Managing Containers rpc CreateContainer(CreateContainerRequest) returns (CreateC rpc StartContainer(StartContainerRequest) returns (StartCont rpc StopContainer(StopContainerRequest) returns (StopContain rpc RemoveContainer(RemoveContainerRequest) returns (RemoveC rpc ListContainers(ListContainersRequest) returns (ListConta rpc ContainerStatus(ContainerStatusRequest) returns (Contain rpc UpdateContainerResources(UpdateContainerResourcesRequest rpc ReopenContainerLog(ReopenContainerLogRequest) returns rpc ContainerStats(ContainerStatsRequest) returns (Container rpc ListContainerStats(ListContainerStatsRequest) returns https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/v1alpha1/runtime/api.prot

Slide 19

Slide 19 text

And misc(etc etc) // Misc rpc Version(VersionRequest) returns (VersionResponse) {} rpc ExecSync(ExecSyncRequest) returns (ExecSyncResponse) {} rpc Exec(ExecRequest) returns (ExecResponse) {} rpc Attach(AttachRequest) returns (AttachResponse) {} rpc PortForward(PortForwardRequest) returns (PortForwardResp rpc UpdateRuntimeConfig(UpdateRuntimeConfigRequest) returns rpc Status(StatusRequest) returns (StatusResponse) {} }

Slide 20

Slide 20 text

Oh last but not least, we can also call List/Pull/Remove Image Operations: service ImageService { // Managing Image rpc ListImages(ListImagesRequest) returns (ListImagesRespons rpc ImageStatus(ImageStatusRequest) returns (ImageStatusResp rpc PullImage(PullImageRequest) returns (PullImageResponse) rpc RemoveImage(RemoveImageRequest) returns (RemoveImageResp rpc ImageFsInfo(ImageFsInfoRequest) returns (ImageFsInfoResp }

Slide 21

Slide 21 text

Now We Know Docker is a one of Container Runtime, but not "the only one" runtime Kubernetes is using gRPC to manage Container Runtime, and spec name for that is CRI CRI de nes 22 APIs for RuntimeService and 5 APIs for ImageService(as of 2018 Feb). Management of Container, Image, and PodSandbox are the main purpose of these APIs.

Slide 22

Slide 22 text

A Bit Closer Look at Speci cations CRI Speci cation de nes standard way to manage PodSandbox, Container, and Image including networking/metrics/logging requirements. OCI Runtime Speci cation de nes state, lifecycle, errors/warnings, operations such as create/start/kill/delete , and hooks. OCI Image Speci cation de nes image layout, manifest, index, lesystem layers, and image con guration etc.

Slide 23

Slide 23 text

Prototyping Kubernetes CRI(Container Runtime Interface) With Ruby and gRPC RubyConf India 2018 Kei Sawada(@remore)

Slide 24

Slide 24 text

Prototyping Kubernetes CRI

Slide 25

Slide 25 text

OK Now We Know CRI. BTW What Is It Like To Manage Servers Using

Slide 26

Slide 26 text

A Very Quick Introduction As you can see, Kubernetes will let you deploy your arbitrary container image on any cloud service without vendor lock-in # Configure kubectl(here we use gcp just as an example) $ gcloud config set project PROJECT_ID $ gcloud container clusters get-credentials PROJECT_ID \ --zone us-central1-b # kubectl is set to use gcp # Deploy $ kubectl run myapp \ --image=remore/simple-ruby-server \ # From DockerHub --port 8080 $ kubectl expose deployment myapp --type="LoadBalancer" $ curl 130.211.134.59:8080

Slide 27

Slide 27 text

How Kubernetes Works ` This Image is taken from github.com/kubernetes-incubator/rktlet

Slide 28

Slide 28 text

Now We Know Docker is a one of Container Runtime, but not "the only one" runtime Kubernetes is using gRPC through $kubectl command to manage Container Runtime, and spec name for that is CRI CRI de nes 22 APIs for RuntimeService and 5 APIs for ImageService(as of 2018 Feb). Management of Container, Image, and PodSandbox are the main purpose of these APIs.

Slide 29

Slide 29 text

Given These Understandings, Now Today's Objective Turns Out: What is it like to create our own gRPC server works as a CRI Shim with Ruby? “ “

Slide 30

Slide 30 text

SHIFU: "If you only do what you can do, you will never be more than you are now"

Slide 31

Slide 31 text

Today's Scenario Now is the time to get our hands dirty. Today we experimentally do: 1. Build Our Own Container Image 2. Build Our Own Container Runtime 3. Prototype Our Own Kubernetes CRI Implementation With Ruby and gRPC And let's wonder what Ruby can do to make this containerized world better.

Slide 32

Slide 32 text

1. Build Our Own Container Image

Slide 33

Slide 33 text

1. Build Our Own Container Image First of all, container images are just tarballs. $ docker pull busybox Using default tag: latest latest: Pulling from library/busybox 57310166fe88: Pull complete Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090a Status: Downloaded newer image for busybox:latest $

Slide 34

Slide 34 text

1. Build Our Own Container Image First of all, container images are just tarballs. $ docker pull busybox Using default tag: latest latest: Pulling from library/busybox 57310166fe88: Pull complete Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090a Status: Downloaded newer image for busybox:latest $ docker save busybox > busybox.tar $

Slide 35

Slide 35 text

1. Build Our Own Container Image First of all, container images are just tarballs. $ docker pull busybox Using default tag: latest latest: Pulling from library/busybox 57310166fe88: Pull complete Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090a Status: Downloaded newer image for busybox:latest $ docker save busybox > busybox.tar $ tar xzvf busybox.tar x 5b0d5902672....e7c25bd2c3.json x ee12c7d73ed....4704362eb2/ x ee12c7d73ed....4704362eb2/VERSION x ee12c7d73ed....4704362eb2/json x ee12c7d73ed....4704362eb2/layer.tar x manifest.json x repositories

Slide 36

Slide 36 text

1. Build Our Own Container Image FYI: If you are interested in the contents: $ cat manifest.json | jq [ { "Config": "5b0d5902672....e7c25bd2c3.json", "RepoTags": [ "busybox:latest"], "Layers": ["ee12c7d73ed....4704362eb2/layer.tar"] } ] $

Slide 37

Slide 37 text

1. Build Our Own Container Image FYI: If you are interested in the contents: $ cat manifest.json | jq [ { "Config": "5b0d5902672....e7c25bd2c3.json", "RepoTags": [ "busybox:latest"], "Layers": ["ee12c7d73ed....4704362eb2/layer.tar"] } ] $ cat repositories | jq { "busybox": { "latest": "ee12c7d73ed....4704362eb2" } }

Slide 38

Slide 38 text

1. Build Our Own Container Image Since container image is just tarballs, that means we can create one without any hassle: # In case of Debian $ mkdir /var/lib/myroot $ debootstrap wheezy \ /var/lib/myroot/ http://debianmirror.nkn.in/debian/ $ tar -c /var/lib/myroot | docker import - $ docker push Or of course, for example alternatively we can use Dockerfile or buildah to build OCI compliant images. e.g. Creating a OCI Container within Docker Container

Slide 39

Slide 39 text

2. Build Our Own Container Runtime

Slide 40

Slide 40 text

2. Build Our Own Container Runtime Although there are many RubyGems for virtualization and containerization, there is no popular OCI-compatible container runtime written in Ruby. Most popular container runtimes are written in Go(e.g. runc, rtk) Try to search "lxc" at rubygems.org for example

Slide 41

Slide 41 text

Introducing Haconiwa github.com/haconiwa/haconiwa Helper tools with DSL for your handmade linux containers, written in mruby My favorite toolkit to DIY our container work

Slide 42

Slide 42 text

A Demo Container with Haconiwa $ touch demo.haco $ cat > demo.haco < Namespace.unshare(Namespace::CLONE_NEWNS) > Namespace.unshare(Namespace::CLONE_NEWPID) > Mount.make_private "/" > Mount.bind_mount "/var/lib/myroot", "/var/lib/myroot/root" > Dir.chroot "/var/lib/myroot" > Dir.chdir "/" > c = Process.fork { > Mount.mount "proc", "/proc", :type => "proc" > Exec.exec "/bin/date" > } > pid, ret = Process.waitpid2 c > puts "Container exited with: #{ret.inspect}" > EOF

Slide 43

Slide 43 text

A Demo Container with Haconiwa It's not OCI speci cation compatible, but it works anyways: # `date` command is executed inside the container $ haconiwa start ./demo.haco Mon Feb 5 01:32:03 UTC 2018 Container exited with: #

Slide 44

Slide 44 text

3. Prototype Our Own Kubernetes CRI Implementation with Ruby and gRPC

Slide 45

Slide 45 text

gRPC in Ruby World Looks like grpc rubygem, which is the of cial package provided by The gRPC Authors backed by CNCF, is the only way to create reliable gRPC server

Slide 46

Slide 46 text

gRPC Server Implementation This example simply returns version number( grpc and runtime_services_pb rubygem is required) class RuntimeServiceServer < Runtime::RuntimeService::Service def version(request, _unused_call) Runtime::VersionResponse.new( version: "1.1", runtime_name:"foobar_runtime", runtime_version:"9.9.9", runtime_api_version:"9999" ) # Here we just use dummy values end end s = GRPC::RpcServer.new s.add_http2_port('unix:/var/run/k8s_cri_prototype.sock', :this_port_is_insecure) s.handle(RuntimeServiceServer) s.run_till_terminated

Slide 47

Slide 47 text

Give It A Try Only a few lines of bash command are needed to test if this gRPC server works: BTW I have pushed this sample source code at: github.com/remore/kubernetes_cri_prototype $ git clone \ https://github.com/remore/kubernetes_cri_prototype.git $ cd ./kubernetes_cri_prototype $ bundle install $ bundle exec ruby ./server.rb & [1] 10712 $ bundle exec ruby ./demo_client.rb "Reponse Message: #"

Slide 48

Slide 48 text

Mocking Up Real World Examples Now as the last example, we're going to a few more methods to RuntimeServiceServer class. This is a method corresponding to RunPodSandboxRequest : class RuntimeServiceServer def run_pod_sandbox(request, _unused_call) Runtime::RunPodSandboxResponse.new( pod_sandbox_id: "c605182c-ca3b-11e4-ad0b-525400c788eb" ) end end

Slide 49

Slide 49 text

Mocking Up Real World Examples And this is for CreateContainerRequest : class RuntimeServiceServer def create_container(request, _unused_call) pipe_cmd_in, pipe_cmd_out = IO.pipe @pid = Process.spawn( "haconiwa run #{File.dirname(__FILE__)}/container.haco" :out => pipe_cmd_out, :err => pipe_cmd_out ) Process.wait @pid pipe_cmd_out.close out = pipe_cmd_in.read Process.detach @pid Runtime::CreateContainerResponse.new( container_id: "container-#{out.match(/(\d+)/).to_s}" ) end end

Slide 50

Slide 50 text

Mocking Up Real World Examples With these preparation, we can respond to both RunPodSandboxRequest and CreateContainerRequest triggered by crictl, which is a CRI client simulator: # crictl -r # Inquire about version info $ crictl -r /var/run/k8s_cri_prototype.sock version Version: 1.1 RuntimeName: foobar_runtime RuntimeVersion: 2.2.3 RuntimeApiVersion: 9999 # Run a new pod sandbox("runp" stands for RUN Pod maybe) $ crictl -r /var/run/k8s_cri_prototype.sock runp \ test/testdata/sandbox_config.json # PodSandbox config c605182c-ca3b-11e4-ad0b-525400c788eb

Slide 51

Slide 51 text

# This json is downloaded from # https://github.com/kubernetes-incubator/cri-o $ cat test/testdata/sandbox_config.json { "metadata": { "name": "podsandbox1", "uid": "redhat-test-crio", "namespace": "redhat.test.crio", "attempt": 1 }, "hostname": "crictl_host", "log_directory": "", "dns_config": {"searches": ["8.8.8.8"]}, "port_mappings": [], "resources": { "cpu": { "limits": 3, "requests": 2 }, "memory": { "limits": 50000000,

Slide 52

Slide 52 text

Mocking Up Real World Examples And nally testing CreateContainerRequest : # Start to run a container $ crictl -r /var/run/k8s_cri_prototype.sock create \ "c605182c-ca3b-11e4-ad0b-525400c788eb" \#PodSandbox ID test/testdata/container_redis.json \#Container config test/testdata/sandbox_config.json #PodSandbox config container-3939 # Check if the container is running $ curl -s localhost:8000 ... Tada! Finally we have responded to several requests sent from an of cial CRI client.

Slide 53

Slide 53 text

Takeaway Docker is a one of Container Runtime, but not "the only one" runtime Kubernetes is using gRPC through $kubectl command to manage Container Runtime, and spec name for that is CRI CRI de nes 22 APIs for RuntimeService and 5 APIs for ImageService(as of 2018 Feb). Management of Container, Image, and PodSandbox are the main purpose of these APIs It looks like we can create our own CRI implementation, there are tons of todos though

Slide 54

Slide 54 text

[Discussion] Kubernetes ecosystem is evolving rapidly. Many tooling and concepts are still on the way to getting matured. Much more effort is needed to improve. Good thing is essential parts of container orchestration - Container Image, Container Runtime, and Interface to Container Runtime has been standardized by CRI and OCI speci cation. I have not mentioned so much in this session, but in the real world most of the tools for kubernetes seems written in Go but not Ruby

Slide 55

Slide 55 text

[Discussion] Potentially Ruby have something to do with this situation. What about the idea, for example: Write ruby script as a con guration le, just like Jsonnet , instead of writing redundant too many YAML con g les? Create your own FaaS(like AWS lambda) platform specially designed for Ruby? Write alternative to crictl sounds also interesting. It's always good to stick to a minimum set of tools until you really need it. (e.g. use grpc-gateway as an endpoint)

Slide 56

Slide 56 text

(Let Us) Have A Nice Hacking

Slide 57

Slide 57 text

Appendix: One Of The Best Free Resource To Learn Kubernetes https://kubernetes.io/docs/tutorials/kubernetes-basics/deploy-interactive/

Slide 58

Slide 58 text

Appendix: Background

Slide 59

Slide 59 text

irb(main):001:0> Kubernetes.is_incubated_by => :CNCF CNCF is founded Dec 2015 as a part of Linux Foundation CNCF stands for "Cloud Native Computing Foundation" Cloud Native Computing is a concept of open source software stack to be: - Containerized - Dynamically orchestrated - Microservices oriented

Slide 60

Slide 60 text

irb(main):002:0> Kubernetes.is_trending? => true The Image is taken from http://bit.ly/2E1jIYh

Slide 61

Slide 61 text

irb(main):003:0> Kubernetes.is_used_by => :most_of_cloud_vendors irb(main):004:0> Kubernetes.is_de_facto_in(2018) => true What happened last year was: CoreOS has moved from eet to Kubernetes Docker has started to support kubernetes Big cloud vendor such as AWS and Microsoft has started managed kubernetes service(such as AKS and EKS)