$30 off During Our Annual Pro Sale. View Details »

Prototyping Kubernetes CRI With Ruby and gRPC

remore
February 10, 2018

Prototyping Kubernetes CRI With Ruby and gRPC

Slides for my talk at RubyConf India 2018

remore

February 10, 2018
Tweet

More Decks by remore

Other Decks in Technology

Transcript

  1. Prototyping Kubernetes
    CRI(Container Runtime Interface)
    With Ruby and gRPC
    RubyConf India 2018
    Kei Sawada(@remore)

    View Slide

  2. How many of you guys
    Are using container for your work?
    Are using kubernetes for your work already?
    Have ever implemented your own Kubernetes CRI
    implementation?

    View Slide

  3. Hi RubyConf India

    View Slide

  4. View Slide

  5. Container Runtime Interface

    View Slide

  6. ? ? ?

    View Slide

  7. Container Runtime == Docker ?

    View Slide

  8. View Slide

  9. Not Exactly The Same

    View Slide

  10. Docker is one of implementations
    of Container Runtime
    Container Runtime is something you use to
    manipulate container images
    There are many docker alternatives out there such
    as runc, rkt, and runv. What is important about
    them is, they are interchangeable since they all
    are compatible with OCI speci cation, which is a
    standard speci cation of container runtime and
    image.
    OCI stands for Open Container Initiative

    View Slide

  11. Then What Is
    Container Runtime Interface?

    View Slide

  12. In Kubernetes, it is de ned like
    this:
    CRI, Container Runtime Interface provides a set of
    APIs you can call from client application.
    The original image is taken from http://blog.kubernetes.io/2016/12/container-runtime-interface-cri-in-
    kubernetes.html

    View Slide

  13. In Kubernetes, it is de ned like
    this:
    And actually, APIs are called over gRPC.

    View Slide

  14. Now We Know
    Docker is a one of Container Runtime, but not "the
    only one" runtime
    Kubernetes is using gRPC to manage Container
    Runtime, and spec name for that is CRI

    View Slide

  15. OK Now We Know
    CRI Works Over gRPC.
    But What Kind of APIs
    Are Available Actually?

    View Slide

  16. Since we are Dev,
    we reading API Docs.
    Don't we? I do.

    View Slide

  17. CRI Protocol Buffer's IDL
    CRI APIs consists of 2 services( RuntimeService and
    ImageService ).
    In the RuntimeService , it looks like we can
    Run/Stop/Remove PodSandbox...
    service RuntimeService {
    // Managing PodSandbox
    rpc RunPodSandbox(RunPodSandboxRequest) returns (RunPodSandb
    rpc StopPodSandbox(StopPodSandboxRequest) returns (StopPodSa
    rpc RemovePodSandbox(RemovePodSandboxRequest) returns (Remov
    rpc PodSandboxStatus(PodSandboxStatusRequest) returns (PodSa
    rpc ListPodSandbox(ListPodSandboxRequest) returns (ListPodSa
    https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/v1alpha1/runtime/api.prot

    View Slide

  18. And we can Create/Start/Stop/Remove Container...
    // Managing Containers
    rpc CreateContainer(CreateContainerRequest) returns (CreateC
    rpc StartContainer(StartContainerRequest) returns (StartCont
    rpc StopContainer(StopContainerRequest) returns (StopContain
    rpc RemoveContainer(RemoveContainerRequest) returns (RemoveC
    rpc ListContainers(ListContainersRequest) returns (ListConta
    rpc ContainerStatus(ContainerStatusRequest) returns (Contain
    rpc UpdateContainerResources(UpdateContainerResourcesRequest
    rpc ReopenContainerLog(ReopenContainerLogRequest) returns
    rpc ContainerStats(ContainerStatsRequest) returns (Container
    rpc ListContainerStats(ListContainerStatsRequest) returns
    https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/v1alpha1/runtime/api.prot

    View Slide

  19. And misc(etc etc)
    // Misc
    rpc Version(VersionRequest) returns (VersionResponse) {}
    rpc ExecSync(ExecSyncRequest) returns (ExecSyncResponse) {}
    rpc Exec(ExecRequest) returns (ExecResponse) {}
    rpc Attach(AttachRequest) returns (AttachResponse) {}
    rpc PortForward(PortForwardRequest) returns (PortForwardResp
    rpc UpdateRuntimeConfig(UpdateRuntimeConfigRequest) returns
    rpc Status(StatusRequest) returns (StatusResponse) {}
    }

    View Slide

  20. Oh last but not least, we can also call
    List/Pull/Remove Image Operations:
    service ImageService {
    // Managing Image
    rpc ListImages(ListImagesRequest) returns (ListImagesRespons
    rpc ImageStatus(ImageStatusRequest) returns (ImageStatusResp
    rpc PullImage(PullImageRequest) returns (PullImageResponse)
    rpc RemoveImage(RemoveImageRequest) returns (RemoveImageResp
    rpc ImageFsInfo(ImageFsInfoRequest) returns (ImageFsInfoResp
    }

    View Slide

  21. Now We Know
    Docker is a one of Container Runtime, but not "the
    only one" runtime
    Kubernetes is using gRPC to manage Container
    Runtime, and spec name for that is CRI
    CRI de nes 22 APIs for RuntimeService and 5
    APIs for ImageService(as of 2018 Feb).
    Management of Container, Image, and
    PodSandbox are the main purpose of these APIs.

    View Slide

  22. A Bit Closer Look at Speci cations
    CRI Speci cation de nes standard way to manage
    PodSandbox, Container, and Image including
    networking/metrics/logging requirements.
    OCI Runtime Speci cation de nes state, lifecycle,
    errors/warnings, operations such as
    create/start/kill/delete , and hooks.
    OCI Image Speci cation de nes image layout,
    manifest, index, lesystem layers, and image
    con guration etc.

    View Slide

  23. Prototyping Kubernetes
    CRI(Container Runtime Interface)
    With Ruby and gRPC
    RubyConf India 2018
    Kei Sawada(@remore)

    View Slide

  24. Prototyping Kubernetes CRI

    View Slide

  25. OK Now We Know CRI. BTW
    What Is It Like To Manage Servers
    Using

    View Slide

  26. A Very Quick Introduction
    As you can see, Kubernetes will let you deploy your
    arbitrary container image on any cloud service
    without vendor lock-in
    # Configure kubectl(here we use gcp just as an example)
    $ gcloud config set project PROJECT_ID
    $ gcloud container clusters get-credentials PROJECT_ID \
    --zone us-central1-b # kubectl is set to use gcp
    # Deploy
    $ kubectl run myapp \
    --image=remore/simple-ruby-server \ # From DockerHub
    --port 8080
    $ kubectl expose deployment myapp --type="LoadBalancer"
    $ curl 130.211.134.59:8080

    View Slide

  27. How Kubernetes Works
    `
    This Image is taken from github.com/kubernetes-incubator/rktlet

    View Slide

  28. Now We Know
    Docker is a one of Container Runtime, but not "the
    only one" runtime
    Kubernetes is using gRPC through $kubectl
    command to manage Container Runtime, and spec
    name for that is CRI
    CRI de nes 22 APIs for RuntimeService and 5
    APIs for ImageService(as of 2018 Feb).
    Management of Container, Image, and
    PodSandbox are the main purpose of these APIs.

    View Slide

  29. Given These Understandings, Now
    Today's Objective Turns Out:
    What is it like to create our own gRPC server
    works as a CRI Shim with Ruby?


    View Slide

  30. SHIFU:
    "If you only do what you can
    do, you will never be more
    than you are now"

    View Slide

  31. Today's Scenario
    Now is the time to get our hands dirty. Today we
    experimentally do:
    1. Build Our Own Container Image
    2. Build Our Own Container Runtime
    3. Prototype Our Own Kubernetes CRI
    Implementation With Ruby and gRPC
    And let's wonder what Ruby can do to make
    this containerized world better.

    View Slide

  32. 1. Build Our Own
    Container Image

    View Slide

  33. 1. Build Our Own Container Image
    First of all, container images are just tarballs.
    $ docker pull busybox
    Using default tag: latest
    latest: Pulling from library/busybox
    57310166fe88: Pull complete
    Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090a
    Status: Downloaded newer image for busybox:latest
    $

    View Slide

  34. 1. Build Our Own Container Image
    First of all, container images are just tarballs.
    $ docker pull busybox
    Using default tag: latest
    latest: Pulling from library/busybox
    57310166fe88: Pull complete
    Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090a
    Status: Downloaded newer image for busybox:latest
    $ docker save busybox > busybox.tar
    $

    View Slide

  35. 1. Build Our Own Container Image
    First of all, container images are just tarballs.
    $ docker pull busybox
    Using default tag: latest
    latest: Pulling from library/busybox
    57310166fe88: Pull complete
    Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090a
    Status: Downloaded newer image for busybox:latest
    $ docker save busybox > busybox.tar
    $ tar xzvf busybox.tar
    x 5b0d5902672....e7c25bd2c3.json
    x ee12c7d73ed....4704362eb2/
    x ee12c7d73ed....4704362eb2/VERSION
    x ee12c7d73ed....4704362eb2/json
    x ee12c7d73ed....4704362eb2/layer.tar
    x manifest.json
    x repositories

    View Slide

  36. 1. Build Our Own Container Image
    FYI: If you are interested in the contents:
    $ cat manifest.json | jq
    [
    {
    "Config": "5b0d5902672....e7c25bd2c3.json",
    "RepoTags": [ "busybox:latest"],
    "Layers": ["ee12c7d73ed....4704362eb2/layer.tar"]
    }
    ]
    $

    View Slide

  37. 1. Build Our Own Container Image
    FYI: If you are interested in the contents:
    $ cat manifest.json | jq
    [
    {
    "Config": "5b0d5902672....e7c25bd2c3.json",
    "RepoTags": [ "busybox:latest"],
    "Layers": ["ee12c7d73ed....4704362eb2/layer.tar"]
    }
    ]
    $ cat repositories | jq
    {
    "busybox": {
    "latest": "ee12c7d73ed....4704362eb2"
    }
    }

    View Slide

  38. 1. Build Our Own Container Image
    Since container image is just tarballs, that means we
    can create one without any hassle:
    # In case of Debian
    $ mkdir /var/lib/myroot
    $ debootstrap wheezy \
    /var/lib/myroot/ http://debianmirror.nkn.in/debian/
    $ tar -c /var/lib/myroot | docker import -
    $ docker push
    Or of course, for example alternatively we can use
    Dockerfile or buildah to build OCI compliant images.
    e.g. Creating a OCI Container within Docker
    Container

    View Slide

  39. 2. Build Our Own
    Container Runtime

    View Slide

  40. 2. Build Our Own Container
    Runtime
    Although there are many RubyGems for
    virtualization and containerization, there is no
    popular OCI-compatible container runtime
    written in Ruby. Most popular container runtimes
    are written in Go(e.g. runc, rtk)
    Try to search "lxc" at rubygems.org for example

    View Slide

  41. Introducing Haconiwa
    github.com/haconiwa/haconiwa
    Helper tools with DSL for your handmade linux
    containers, written in mruby
    My favorite toolkit to DIY our container work

    View Slide

  42. A Demo Container with Haconiwa
    $ touch demo.haco
    $ cat > demo.haco <> Namespace.unshare(Namespace::CLONE_NEWNS)
    > Namespace.unshare(Namespace::CLONE_NEWPID)
    > Mount.make_private "/"
    > Mount.bind_mount "/var/lib/myroot", "/var/lib/myroot/root"
    > Dir.chroot "/var/lib/myroot"
    > Dir.chdir "/"
    > c = Process.fork {
    > Mount.mount "proc", "/proc", :type => "proc"
    > Exec.exec "/bin/date"
    > }
    > pid, ret = Process.waitpid2 c
    > puts "Container exited with: #{ret.inspect}"
    > EOF

    View Slide

  43. A Demo Container with Haconiwa
    It's not OCI speci cation compatible, but it works
    anyways:
    # `date` command is executed inside the container
    $ haconiwa start ./demo.haco
    Mon Feb 5 01:32:03 UTC 2018
    Container exited with: #

    View Slide

  44. 3. Prototype Our Own
    Kubernetes CRI Implementation
    with Ruby and gRPC

    View Slide

  45. gRPC in Ruby World
    Looks like grpc rubygem, which is the of cial
    package provided by The gRPC Authors backed by
    CNCF, is the only way to create reliable gRPC
    server

    View Slide

  46. gRPC Server Implementation
    This example simply returns version number( grpc
    and runtime_services_pb rubygem is required)
    class RuntimeServiceServer < Runtime::RuntimeService::Service
    def version(request, _unused_call)
    Runtime::VersionResponse.new(
    version: "1.1", runtime_name:"foobar_runtime",
    runtime_version:"9.9.9", runtime_api_version:"9999"
    ) # Here we just use dummy values
    end
    end
    s = GRPC::RpcServer.new
    s.add_http2_port('unix:/var/run/k8s_cri_prototype.sock',
    :this_port_is_insecure)
    s.handle(RuntimeServiceServer)
    s.run_till_terminated

    View Slide

  47. Give It A Try
    Only a few lines of bash command are needed to test
    if this gRPC server works:
    BTW I have pushed this sample source code at:
    github.com/remore/kubernetes_cri_prototype
    $ git clone \
    https://github.com/remore/kubernetes_cri_prototype.git
    $ cd ./kubernetes_cri_prototype
    $ bundle install
    $ bundle exec ruby ./server.rb &
    [1] 10712
    $ bundle exec ruby ./demo_client.rb
    "Reponse Message: #"

    View Slide

  48. Mocking Up Real World Examples
    Now as the last example, we're going to a few more
    methods to RuntimeServiceServer class.
    This is a method corresponding to
    RunPodSandboxRequest :
    class RuntimeServiceServer
    def run_pod_sandbox(request, _unused_call)
    Runtime::RunPodSandboxResponse.new(
    pod_sandbox_id: "c605182c-ca3b-11e4-ad0b-525400c788eb"
    )
    end
    end

    View Slide

  49. Mocking Up Real World Examples
    And this is for CreateContainerRequest :
    class RuntimeServiceServer
    def create_container(request, _unused_call)
    pipe_cmd_in, pipe_cmd_out = IO.pipe
    @pid = Process.spawn(
    "haconiwa run #{File.dirname(__FILE__)}/container.haco"
    :out => pipe_cmd_out, :err => pipe_cmd_out
    )
    Process.wait @pid
    pipe_cmd_out.close
    out = pipe_cmd_in.read
    Process.detach @pid
    Runtime::CreateContainerResponse.new(
    container_id: "container-#{out.match(/(\d+)/).to_s}"
    )
    end
    end

    View Slide

  50. Mocking Up Real World Examples
    With these preparation, we can respond to both
    RunPodSandboxRequest and CreateContainerRequest
    triggered by crictl, which is a CRI client simulator:
    # crictl -r
    # Inquire about version info
    $ crictl -r /var/run/k8s_cri_prototype.sock version
    Version: 1.1
    RuntimeName: foobar_runtime
    RuntimeVersion: 2.2.3
    RuntimeApiVersion: 9999
    # Run a new pod sandbox("runp" stands for RUN Pod maybe)
    $ crictl -r /var/run/k8s_cri_prototype.sock runp \
    test/testdata/sandbox_config.json # PodSandbox config
    c605182c-ca3b-11e4-ad0b-525400c788eb

    View Slide

  51. # This json is downloaded from
    # https://github.com/kubernetes-incubator/cri-o
    $ cat test/testdata/sandbox_config.json
    {
    "metadata": {
    "name": "podsandbox1",
    "uid": "redhat-test-crio",
    "namespace": "redhat.test.crio",
    "attempt": 1
    },
    "hostname": "crictl_host",
    "log_directory": "",
    "dns_config": {"searches": ["8.8.8.8"]},
    "port_mappings": [],
    "resources": {
    "cpu": {
    "limits": 3,
    "requests": 2
    },
    "memory": {
    "limits": 50000000,

    View Slide

  52. Mocking Up Real World Examples
    And nally testing CreateContainerRequest :
    # Start to run a container
    $ crictl -r /var/run/k8s_cri_prototype.sock create \
    "c605182c-ca3b-11e4-ad0b-525400c788eb" \#PodSandbox ID
    test/testdata/container_redis.json \#Container config
    test/testdata/sandbox_config.json #PodSandbox config
    container-3939
    # Check if the container is running
    $ curl -s localhost:8000
    ...
    Tada! Finally we have responded to several
    requests sent from an of cial CRI client.

    View Slide

  53. Takeaway
    Docker is a one of Container Runtime, but not "the
    only one" runtime
    Kubernetes is using gRPC through $kubectl
    command to manage Container Runtime, and spec
    name for that is CRI
    CRI de nes 22 APIs for RuntimeService and 5
    APIs for ImageService(as of 2018 Feb).
    Management of Container, Image, and
    PodSandbox are the main purpose of these APIs
    It looks like we can create our own CRI
    implementation, there are tons of todos though

    View Slide

  54. [Discussion]
    Kubernetes ecosystem is evolving rapidly. Many
    tooling and concepts are still on the way to getting
    matured. Much more effort is needed to improve.
    Good thing is essential parts of container
    orchestration - Container Image, Container
    Runtime, and Interface to Container Runtime has
    been standardized by CRI and OCI speci cation.
    I have not mentioned so much in this session, but
    in the real world most of the tools for kubernetes
    seems written in Go but not Ruby

    View Slide

  55. [Discussion]
    Potentially Ruby have something to do with this
    situation. What about the idea, for example:
    Write ruby script as a con guration le, just like
    Jsonnet , instead of writing redundant too many
    YAML con g les?
    Create your own FaaS(like AWS lambda)
    platform specially designed for Ruby?
    Write alternative to crictl sounds also
    interesting. It's always good to stick to a
    minimum set of tools until you really need it.
    (e.g. use grpc-gateway as an endpoint)

    View Slide

  56. (Let Us) Have A Nice Hacking

    View Slide

  57. Appendix: One Of The Best Free
    Resource To Learn Kubernetes
    https://kubernetes.io/docs/tutorials/kubernetes-basics/deploy-interactive/

    View Slide

  58. Appendix: Background

    View Slide

  59. irb(main):001:0> Kubernetes.is_incubated_by
    => :CNCF
    CNCF is founded Dec 2015 as a part of Linux
    Foundation
    CNCF stands for "Cloud Native Computing
    Foundation"
    Cloud Native Computing is a concept of open
    source software stack to be:
    - Containerized
    - Dynamically orchestrated
    - Microservices oriented

    View Slide

  60. irb(main):002:0> Kubernetes.is_trending?
    => true
    The Image is taken from http://bit.ly/2E1jIYh

    View Slide

  61. irb(main):003:0> Kubernetes.is_used_by
    => :most_of_cloud_vendors
    irb(main):004:0> Kubernetes.is_de_facto_in(2018)
    => true
    What happened last year was:
    CoreOS has moved from eet to Kubernetes
    Docker has started to support kubernetes
    Big cloud vendor such as AWS and Microsoft has
    started managed kubernetes service(such as AKS
    and EKS)

    View Slide