Prototyping Kubernetes CRI(Container Runtime Interface) With Ruby and gRPC RubyConf India 2018 Kei Sawada(@remore)

How many of you guys Are using container for your work? Are using kubernetes for your work already? Have ever implemented your own Kubernetes CRI implementation?

Hi RubyConf India

No content

Container Runtime Interface

? ? ?

Container Runtime == Docker ?

No content

Not Exactly The Same

Docker is one of implementations of Container Runtime Container Runtime is something you use to manipulate container images There are many docker alternatives out there such as runc, rkt, and runv. What is important about them is, they are interchangeable since they all are compatible with OCI speci cation, which is a standard speci cation of container runtime and image. OCI stands for Open Container Initiative

Then What Is Container Runtime Interface?

In Kubernetes, it is de ned like this: CRI, Container Runtime Interface provides a set of APIs you can call from client application. The original image is taken from http://blog.kubernetes.io/2016/12/container-runtime-interface-cri-in- kubernetes.html

In Kubernetes, it is de ned like this: And actually, APIs are called over gRPC.

Now We Know Docker is a one of Container Runtime, but not "the only one" runtime Kubernetes is using gRPC to manage Container Runtime, and spec name for that is CRI

OK Now We Know CRI Works Over gRPC. But What Kind of APIs Are Available Actually?

Since we are Dev, we reading API Docs. Don't we? I do.

CRI Protocol Buffer's IDL CRI APIs consists of 2 services( RuntimeService and ImageService ). In the RuntimeService , it looks like we can Run/Stop/Remove PodSandbox... service RuntimeService { // Managing PodSandbox rpc RunPodSandbox(RunPodSandboxRequest) returns (RunPodSandb rpc StopPodSandbox(StopPodSandboxRequest) returns (StopPodSa rpc RemovePodSandbox(RemovePodSandboxRequest) returns (Remov rpc PodSandboxStatus(PodSandboxStatusRequest) returns (PodSa rpc ListPodSandbox(ListPodSandboxRequest) returns (ListPodSa https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/v1alpha1/runtime/api.prot

And we can Create/Start/Stop/Remove Container... // Managing Containers rpc CreateContainer(CreateContainerRequest) returns (CreateC rpc StartContainer(StartContainerRequest) returns (StartCont rpc StopContainer(StopContainerRequest) returns (StopContain rpc RemoveContainer(RemoveContainerRequest) returns (RemoveC rpc ListContainers(ListContainersRequest) returns (ListConta rpc ContainerStatus(ContainerStatusRequest) returns (Contain rpc UpdateContainerResources(UpdateContainerResourcesRequest rpc ReopenContainerLog(ReopenContainerLogRequest) returns rpc ContainerStats(ContainerStatsRequest) returns (Container rpc ListContainerStats(ListContainerStatsRequest) returns https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/v1alpha1/runtime/api.prot

And misc(etc etc) // Misc rpc Version(VersionRequest) returns (VersionResponse) {} rpc ExecSync(ExecSyncRequest) returns (ExecSyncResponse) {} rpc Exec(ExecRequest) returns (ExecResponse) {} rpc Attach(AttachRequest) returns (AttachResponse) {} rpc PortForward(PortForwardRequest) returns (PortForwardResp rpc UpdateRuntimeConfig(UpdateRuntimeConfigRequest) returns rpc Status(StatusRequest) returns (StatusResponse) {} }

Oh last but not least, we can also call List/Pull/Remove Image Operations: service ImageService { // Managing Image rpc ListImages(ListImagesRequest) returns (ListImagesRespons rpc ImageStatus(ImageStatusRequest) returns (ImageStatusResp rpc PullImage(PullImageRequest) returns (PullImageResponse) rpc RemoveImage(RemoveImageRequest) returns (RemoveImageResp rpc ImageFsInfo(ImageFsInfoRequest) returns (ImageFsInfoResp }

Now We Know Docker is a one of Container Runtime, but not "the only one" runtime Kubernetes is using gRPC to manage Container Runtime, and spec name for that is CRI CRI de nes 22 APIs for RuntimeService and 5 APIs for ImageService(as of 2018 Feb). Management of Container, Image, and PodSandbox are the main purpose of these APIs.

A Bit Closer Look at Speci cations CRI Speci cation de nes standard way to manage PodSandbox, Container, and Image including networking/metrics/logging requirements. OCI Runtime Speci cation de nes state, lifecycle, errors/warnings, operations such as create/start/kill/delete , and hooks. OCI Image Speci cation de nes image layout, manifest, index, lesystem layers, and image con guration etc.

Prototyping Kubernetes CRI(Container Runtime Interface) With Ruby and gRPC RubyConf India 2018 Kei Sawada(@remore)

Prototyping Kubernetes CRI

OK Now We Know CRI. BTW What Is It Like To Manage Servers Using

A Very Quick Introduction As you can see, Kubernetes will let you deploy your arbitrary container image on any cloud service without vendor lock-in # Configure kubectl(here we use gcp just as an example) $ gcloud config set project PROJECT_ID $ gcloud container clusters get-credentials PROJECT_ID \ --zone us-central1-b # kubectl is set to use gcp # Deploy $ kubectl run myapp \ --image=remore/simple-ruby-server \ # From DockerHub --port 8080 $ kubectl expose deployment myapp --type="LoadBalancer" $ curl 130.211.134.59:8080

How Kubernetes Works ` This Image is taken from github.com/kubernetes-incubator/rktlet

Now We Know Docker is a one of Container Runtime, but not "the only one" runtime Kubernetes is using gRPC through $kubectl command to manage Container Runtime, and spec name for that is CRI CRI de nes 22 APIs for RuntimeService and 5 APIs for ImageService(as of 2018 Feb). Management of Container, Image, and PodSandbox are the main purpose of these APIs.

Given These Understandings, Now Today's Objective Turns Out: What is it like to create our own gRPC server works as a CRI Shim with Ruby? “ “

SHIFU: "If you only do what you can do, you will never be more than you are now"

Today's Scenario Now is the time to get our hands dirty. Today we experimentally do: 1. Build Our Own Container Image 2. Build Our Own Container Runtime 3. Prototype Our Own Kubernetes CRI Implementation With Ruby and gRPC And let's wonder what Ruby can do to make this containerized world better.

1. Build Our Own Container Image

1. Build Our Own Container Image First of all, container images are just tarballs. $ docker pull busybox Using default tag: latest latest: Pulling from library/busybox 57310166fe88: Pull complete Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090a Status: Downloaded newer image for busybox:latest $

1. Build Our Own Container Image First of all, container images are just tarballs. $ docker pull busybox Using default tag: latest latest: Pulling from library/busybox 57310166fe88: Pull complete Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090a Status: Downloaded newer image for busybox:latest $ docker save busybox > busybox.tar $

1. Build Our Own Container Image First of all, container images are just tarballs. $ docker pull busybox Using default tag: latest latest: Pulling from library/busybox 57310166fe88: Pull complete Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090a Status: Downloaded newer image for busybox:latest $ docker save busybox > busybox.tar $ tar xzvf busybox.tar x 5b0d5902672....e7c25bd2c3.json x ee12c7d73ed....4704362eb2/ x ee12c7d73ed....4704362eb2/VERSION x ee12c7d73ed....4704362eb2/json x ee12c7d73ed....4704362eb2/layer.tar x manifest.json x repositories

1. Build Our Own Container Image FYI: If you are interested in the contents: $ cat manifest.json | jq [ { "Config": "5b0d5902672....e7c25bd2c3.json", "RepoTags": [ "busybox:latest"], "Layers": ["ee12c7d73ed....4704362eb2/layer.tar"] } ] $

1. Build Our Own Container Image FYI: If you are interested in the contents: $ cat manifest.json | jq [ { "Config": "5b0d5902672....e7c25bd2c3.json", "RepoTags": [ "busybox:latest"], "Layers": ["ee12c7d73ed....4704362eb2/layer.tar"] } ] $ cat repositories | jq { "busybox": { "latest": "ee12c7d73ed....4704362eb2" } }

1. Build Our Own Container Image Since container image is just tarballs, that means we can create one without any hassle: # In case of Debian $ mkdir /var/lib/myroot $ debootstrap wheezy \ /var/lib/myroot/ http://debianmirror.nkn.in/debian/ $ tar -c /var/lib/myroot | docker import - $ docker push Or of course, for example alternatively we can use Dockerfile or buildah to build OCI compliant images. e.g. Creating a OCI Container within Docker Container

2. Build Our Own Container Runtime

2. Build Our Own Container Runtime Although there are many RubyGems for virtualization and containerization, there is no popular OCI-compatible container runtime written in Ruby. Most popular container runtimes are written in Go(e.g. runc, rtk) Try to search "lxc" at rubygems.org for example

Introducing Haconiwa github.com/haconiwa/haconiwa Helper tools with DSL for your handmade linux containers, written in mruby My favorite toolkit to DIY our container work

A Demo Container with Haconiwa $ touch demo.haco $ cat > demo.haco < Namespace.unshare(Namespace::CLONE_NEWNS) > Namespace.unshare(Namespace::CLONE_NEWPID) > Mount.make_private "/" > Mount.bind_mount "/var/lib/myroot", "/var/lib/myroot/root" > Dir.chroot "/var/lib/myroot" > Dir.chdir "/" > c = Process.fork { > Mount.mount "proc", "/proc", :type => "proc" > Exec.exec "/bin/date" > } > pid, ret = Process.waitpid2 c > puts "Container exited with: #{ret.inspect}" > EOF

A Demo Container with Haconiwa It's not OCI speci cation compatible, but it works anyways: # `date` command is executed inside the container $ haconiwa start ./demo.haco Mon Feb 5 01:32:03 UTC 2018 Container exited with: #

3. Prototype Our Own Kubernetes CRI Implementation with Ruby and gRPC

gRPC in Ruby World Looks like grpc rubygem, which is the of cial package provided by The gRPC Authors backed by CNCF, is the only way to create reliable gRPC server

gRPC Server Implementation This example simply returns version number( grpc and runtime_services_pb rubygem is required) class RuntimeServiceServer < Runtime::RuntimeService::Service def version(request, _unused_call) Runtime::VersionResponse.new( version: "1.1", runtime_name:"foobar_runtime", runtime_version:"9.9.9", runtime_api_version:"9999" ) # Here we just use dummy values end end s = GRPC::RpcServer.new s.add_http2_port('unix:/var/run/k8s_cri_prototype.sock', :this_port_is_insecure) s.handle(RuntimeServiceServer) s.run_till_terminated

Give It A Try Only a few lines of bash command are needed to test if this gRPC server works: BTW I have pushed this sample source code at: github.com/remore/kubernetes_cri_prototype $ git clone \ https://github.com/remore/kubernetes_cri_prototype.git $ cd ./kubernetes_cri_prototype $ bundle install $ bundle exec ruby ./server.rb & [1] 10712 $ bundle exec ruby ./demo_client.rb "Reponse Message: #"

Mocking Up Real World Examples Now as the last example, we're going to a few more methods to RuntimeServiceServer class. This is a method corresponding to RunPodSandboxRequest : class RuntimeServiceServer def run_pod_sandbox(request, _unused_call) Runtime::RunPodSandboxResponse.new( pod_sandbox_id: "c605182c-ca3b-11e4-ad0b-525400c788eb" ) end end

Mocking Up Real World Examples And this is for CreateContainerRequest : class RuntimeServiceServer def create_container(request, _unused_call) pipe_cmd_in, pipe_cmd_out = IO.pipe @pid = Process.spawn( "haconiwa run #{File.dirname(__FILE__)}/container.haco" :out => pipe_cmd_out, :err => pipe_cmd_out ) Process.wait @pid pipe_cmd_out.close out = pipe_cmd_in.read Process.detach @pid Runtime::CreateContainerResponse.new( container_id: "container-#{out.match(/(\d+)/).to_s}" ) end end

Mocking Up Real World Examples With these preparation, we can respond to both RunPodSandboxRequest and CreateContainerRequest triggered by crictl, which is a CRI client simulator: # crictl -r # Inquire about version info $ crictl -r /var/run/k8s_cri_prototype.sock version Version: 1.1 RuntimeName: foobar_runtime RuntimeVersion: 2.2.3 RuntimeApiVersion: 9999 # Run a new pod sandbox("runp" stands for RUN Pod maybe) $ crictl -r /var/run/k8s_cri_prototype.sock runp \ test/testdata/sandbox_config.json # PodSandbox config c605182c-ca3b-11e4-ad0b-525400c788eb

# This json is downloaded from # https://github.com/kubernetes-incubator/cri-o $ cat test/testdata/sandbox_config.json { "metadata": { "name": "podsandbox1", "uid": "redhat-test-crio", "namespace": "redhat.test.crio", "attempt": 1 }, "hostname": "crictl_host", "log_directory": "", "dns_config": {"searches": ["8.8.8.8"]}, "port_mappings": [], "resources": { "cpu": { "limits": 3, "requests": 2 }, "memory": { "limits": 50000000,

Mocking Up Real World Examples And nally testing CreateContainerRequest : # Start to run a container $ crictl -r /var/run/k8s_cri_prototype.sock create \ "c605182c-ca3b-11e4-ad0b-525400c788eb" \#PodSandbox ID test/testdata/container_redis.json \#Container config test/testdata/sandbox_config.json #PodSandbox config container-3939 # Check if the container is running $ curl -s localhost:8000 ... Tada! Finally we have responded to several requests sent from an of cial CRI client.

Takeaway Docker is a one of Container Runtime, but not "the only one" runtime Kubernetes is using gRPC through $kubectl command to manage Container Runtime, and spec name for that is CRI CRI de nes 22 APIs for RuntimeService and 5 APIs for ImageService(as of 2018 Feb). Management of Container, Image, and PodSandbox are the main purpose of these APIs It looks like we can create our own CRI implementation, there are tons of todos though

[Discussion] Kubernetes ecosystem is evolving rapidly. Many tooling and concepts are still on the way to getting matured. Much more effort is needed to improve. Good thing is essential parts of container orchestration - Container Image, Container Runtime, and Interface to Container Runtime has been standardized by CRI and OCI speci cation. I have not mentioned so much in this session, but in the real world most of the tools for kubernetes seems written in Go but not Ruby

[Discussion] Potentially Ruby have something to do with this situation. What about the idea, for example: Write ruby script as a con guration le, just like Jsonnet , instead of writing redundant too many YAML con g les? Create your own FaaS(like AWS lambda) platform specially designed for Ruby? Write alternative to crictl sounds also interesting. It's always good to stick to a minimum set of tools until you really need it. (e.g. use grpc-gateway as an endpoint)

(Let Us) Have A Nice Hacking

Appendix: One Of The Best Free Resource To Learn Kubernetes https://kubernetes.io/docs/tutorials/kubernetes-basics/deploy-interactive/

Appendix: Background

irb(main):001:0> Kubernetes.is_incubated_by => :CNCF CNCF is founded Dec 2015 as a part of Linux Foundation CNCF stands for "Cloud Native Computing Foundation" Cloud Native Computing is a concept of open source software stack to be: - Containerized - Dynamically orchestrated - Microservices oriented

irb(main):002:0> Kubernetes.is_trending? => true The Image is taken from http://bit.ly/2E1jIYh

irb(main):003:0> Kubernetes.is_used_by => :most_of_cloud_vendors irb(main):004:0> Kubernetes.is_de_facto_in(2018) => true What happened last year was: CoreOS has moved from eet to Kubernetes Docker has started to support kubernetes Big cloud vendor such as AWS and Microsoft has started managed kubernetes service(such as AKS and EKS)