Tweet
Tweet

Slide 1

Slide 1 text

Как написать операционную систему на Go Андрей Смирнов Implementing OS in Go (Andrey Smirnov)

Slide 2

Slide 2 text

Andrey Smirnov GitHub, Twitter: @smira Contributor to Talos project

Slide 3

Slide 3 text

Talos: OS for Kubernetes in Go

Slide 4

Slide 4 text

Why another OS for Kubernetes? Linux distribution Kubernetes

Slide 5

Slide 5 text

Why another OS for Kubernetes? Linux distribution Kubernetes We want this

Slide 6

Slide 6 text

Why another OS for Kubernetes? Linux distribution Kubernetes We want this We have to support this

Slide 7

Slide 7 text

Automation to build Linux distribution Kubernetes Ansible, Chef, VM Template...

Slide 8

Slide 8 text

Configuration drift Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes

Slide 9

Slide 9 text

Configuration drift Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes

Slide 10

Slide 10 text

Kubernetes in the cloud Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes

Slide 11

Slide 11 text

Kubernetes in the cloud Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes

Slide 12

Slide 12 text

Why another OS for Kubernetes? Package management issues, configuration drift across nodes Inconsistent Kubernetes experience across clouds and bare-metal Security/compliance issues due to extra bloat in the OS Requires automation tools to manage the cluster Hard to keep up with Kubernetes release schedule

Slide 13

Slide 13 text

What is Talos? Open-source modern OS designed for Kubernetes (only) Linux kernel + user-space in Go Talos Kubernetes Linux kernel User-space (Go)

Slide 14

Slide 14 text

Talos goals Minimal: no bloat Fast: fast to boot Hardened: security best practices applied by default Secure: PKI infrastructure for auth Immutable: read-only rootfs

Slide 15

Slide 15 text

API-driven Talos Talos Talos No shell access gRPC API

Slide 16

Slide 16 text

Bare metal Virtual machines Clouds: Amazon, Google, Azure Docker (testing) Talos platforms

Slide 17

Slide 17 text

Talos Architecture Linux kernel init machined services containerd ntpd networkd udevd osd trustd proxyd kubelet kubeadm etcd kube-apiserver customer workloads initramfs rootfs/ squashfs KSPP mounts userdata Talos K8s

Slide 18

Slide 18 text

Talos Architecture Linux kernel Talos K8s

Slide 19

Slide 19 text

Talos Architecture Linux kernel init initramfs Talos K8s

Slide 20

Slide 20 text

Talos Architecture Linux kernel init machined services initramfs rootfs/ squashfs KSPP mounts userdata Talos K8s

Slide 21

Slide 21 text

Talos Architecture Linux kernel init machined services containerd ntpd networkd udevd osd trustd proxyd initramfs rootfs/ squashfs KSPP mounts userdata Talos K8s

Slide 22

Slide 22 text

Talos Architecture Linux kernel init machined services containerd ntpd networkd udevd osd trustd proxyd kubelet kubeadm initramfs rootfs/ squashfs KSPP mounts userdata Talos K8s

Slide 23

Slide 23 text

Talos Architecture Linux kernel init machined services containerd ntpd networkd udevd osd trustd proxyd kubelet kubeadm etcd kube-apiserver customer workloads initramfs rootfs/ squashfs KSPP mounts userdata Talos K8s

Slide 24

Slide 24 text

What is common between these?

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

Why doing OS in Go is so cool? Easy to reason about the code

Slide 28

Slide 28 text

Why doing OS in Go is so cool? Easy to reason about the code Static linking

Slide 29

Slide 29 text

Why doing OS in Go is so cool? Easy to reason about the code Static linking Great concurrency primitives

Slide 30

Slide 30 text

Why doing OS in Go is so cool? Easy to reason about the code Static linking Great concurrency primitives Wrappers for low-level OS primitives (e.g. netlink)

Slide 31

Slide 31 text

Why doing OS in Go is so cool? Easy to reason about the code Static linking Great concurrency primitives Wrappers for low-level OS primitives (e.g. netlink) Easy integration with other services in Go

Slide 32

Slide 32 text

Concurrency

Slide 33

Slide 33 text

Concurrency in OS API servers Event handling Startup (optimizing boot time)

Slide 34

Slide 34 text

Service Startup containerd networkd kubeadm kubelet ntpd kubeadm-flags.env osd udevd proxyd ca.crt trustd

Slide 35

Slide 35 text

Service A way to run the service (Runner) Service state, health Service logs Service dependencies Service control (stop, start, restart) Service pre-run hooks (prepare)

Slide 36

Slide 36 text

Service Runner Process Container via containerd API (oci) Container via CRI Goroutine Automatic restarts

Slide 37

Slide 37 text

Runner API - v.0 type Runner interface { Run() error } Run() returns when runnable object terminates

Slide 38

Slide 38 text

Runner API - v.1 type Runner interface { Run(ctx context.Context) error } type Runner interface { Run() error Stop() } Run() returns when runnable object terminates

Slide 39

Slide 39 text

Runner API v.2 type EventSink func(event Event) type Runner interface { Run(eventSink EventSink) error Stop() } [Waiting]: Waiting for service "containerd" to be "up", service "kubeadm" to be "up", file "/var/lib/kubelet/kubeadm-flags.env" to exist (1m28s ago) [Waiting]: Waiting for service "kubeadm" to be "up", file "/var/lib/kubelet/kubeadm-flags.env" to exist (1m26s ago) [Waiting]: Waiting for file "/var/lib/kubelet/kubeadm-flags.env" to exist (52s ago) [Preparing]: Running pre state (36s ago) [Preparing]: Creating service runner (36s ago) [Running]: Started task kubelet (PID 551) for container kubelet (36s ago) [Running]: Health check failed: Get http://127.0.0.1:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused (34s ago) [Running]: Health check successful (29s ago)

Slide 40

Slide 40 text

Auto-restarting runner type Restarter struct { wrapped Runner } func (rr *Restarter) Run(eventSink EventSink) error { for { err := rr.wrapped.Run(eventSink) eventSink(Event(“restarting due to %v”, err)) time.Sleep(time.Second) } }

Slide 41

Slide 41 text

Auto-restarting runner type Restarter struct { wrapped Runner } func (rr *Restarter) Run(eventSink EventSink) error { for { err := rr.wrapped.Run(eventSink) eventSink(Event(“restarting due to %v”, err)) time.Sleep(time.Second) } }

Slide 42

Slide 42 text

Auto-restarting runner type Restarter struct { wrapped Runner } func (rr *Restarter) Run(eventSink EventSink) error { for { err := rr.wrapped.Run(eventSink) eventSink(Event(“restarting due to %v”, err)) time.Sleep(time.Second) } }

Slide 43

Slide 43 text

Auto-restarting runner type Restarter struct { wrapped Runner } func (rr *Restarter) Run(eventSink EventSink) error { for { err := rr.wrapped.Run(eventSink) eventSink(Event(“restarting due to %v”, err)) time.Sleep(time.Second) } }

Slide 44

Slide 44 text

Auto-restarting runner: with stop type Restarter struct { wrapped Runner stop chan struct{} } func (rr *Restarter) Stop() { close(rr.stop) }

Slide 45

Slide 45 text

Auto-restarting runner: with stop type Restarter struct { wrapped Runner stop chan struct{} } func (rr *Restarter) Stop() { close(rr.stop) }

Slide 46

Slide 46 text

Auto-restarting runner: with stop for { errCh := make(chan error) go func() { errCh <- rr.wrapped.Run(eventSink) } select { case <-rr.stop: rr.wrapped.Stop() return <-errCh case err := <- errCh: eventSink(Event(“restarting due to %v”, err)) } time.Sleep(time.Second) }

Slide 47

Slide 47 text

Auto-restarting runner: with stop for { errCh := make(chan error) go func() { errCh <- rr.wrapped.Run(eventSink) } select { case <-rr.stop: rr.wrapped.Stop() return <-errCh case err := <- errCh: eventSink(Event(“restarting due to %v”, err)) } time.Sleep(time.Second) }

Slide 48

Slide 48 text

Auto-restarting runner: with stop for { errCh := make(chan error) go func() { errCh <- rr.wrapped.Run(eventSink) } select { case <-rr.stop: rr.wrapped.Stop() return <-errCh case err := <- errCh: eventSink(Event(“restarting due to %v”, err)) } time.Sleep(time.Second) }

Slide 49

Slide 49 text

Auto-restarting runner: with stop for { errCh := make(chan error) go func() { errCh <- rr.wrapped.Run(eventSink) } select { case <-rr.stop: rr.wrapped.Stop() return <-errCh case err := <- errCh: eventSink(Event(“restarting due to %v”, err)) } time.Sleep(time.Second) }

Slide 50

Slide 50 text

Auto-restarting runner: with stop for { errCh := make(chan error) go func() { errCh <- rr.wrapped.Run(eventSink) } select { case <-rr.stop: rr.wrapped.Stop() return <-errCh case err := <- errCh: eventSink(Event(“restarting due to %v”, err)) } time.Sleep(time.Second) }

Slide 51

Slide 51 text

Auto-restarting runner: with stop for { errCh := make(chan error) go func() { errCh <- rr.wrapped.Run(eventSink) } select { case <-rr.stop: rr.wrapped.Stop() return <-errCh case err := <- errCh: eventSink(Event(“restarting due to %v”, err)) } time.Sleep(time.Second) }

Slide 52

Slide 52 text

Service type Service interface { Id() string Runner() Runner Dependencies() []string PreCondition() Condition }

Slide 53

Slide 53 text

Condition type Condition interface { fmt.Stringer Wait(ctx context.Context) error } Condition represents anything which can be waited for Conditions are composable [Waiting]: Waiting for service "containerd" to be "up", service "kubeadm" to be "up", file "/var/lib/kubelet/kubeadm-flags.env" to exist (1m28s ago) [Waiting]: Waiting for service "kubeadm" to be "up", file "/var/lib/kubelet/kubeadm-flags.env" to exist (1m26s ago) [Waiting]: Waiting for file "/var/lib/kubelet/kubeadm-flags.env" to exist (52s ago)

Slide 54

Slide 54 text

Example Conditions func FileExists(path string) Condition func ServiceRunning(id string) Condition

Slide 55

Slide 55 text

Condition composition func FileExists(path string) Condition func ServiceRunning(id string) Condition func WaitForAll(conditions ...Condition) Condition

Slide 56

Slide 56 text

Condition composition func FileExists(path string) Condition func ServiceRunning(id string) Condition func WaitForAll(conditions ...Condition) Condition condition := WaitForAll(ServiceRunning("trustd"), FileExists("/var/lib/kubelet/kubeadm-flags.env"))

Slide 57

Slide 57 text

Condition composition type waitAll struct{ conditions []Condition } func WaitForAll(conditions ...Condition) Condition { return &waitAll{conditions: conditions} }

Slide 58

Slide 58 text

Condition composition func (w *waitAll) Wait(ctx context.Context) error { errCh := make(chan error) for _, cond := range w.conditions { go func(cond Condition) { errCh <- cond.Wait(ctx) }(cond) } var multiErr multierror.Error for range w.conditions { multiErr := multierror.Append(multiErr, <-errCh) } return multiErr.ErrorOrNil() }

Slide 59

Slide 59 text

Service: getting things together for _, svc := range services { go func(svc Service) { cnd := WaitForAll(svc.PreCondition(), ServiceRunning(svc.Dependencies()...)) cnd.Wait(ctx) svc.Runner().Run(eventSink) }(svc) }

Slide 60

Slide 60 text

Service: getting things together for _, svc := range services { go func(svc Service) { cnd := WaitForAll(svc.PreCondition(), ServiceRunning(svc.Dependencies()...)) cnd.Wait(ctx) svc.Runner().Run(eventSink) }(svc) }

Slide 61

Slide 61 text

Service: getting things together for _, svc := range services { go func(svc Service) { cnd := WaitForAll(svc.PreCondition(), ServiceRunning(svc.Dependencies()...)) cnd.Wait(ctx) svc.Runner().Run(eventSink) }(svc) }

Slide 62

Slide 62 text

Service: getting things together for _, svc := range services { go func(svc Service) { cnd := WaitForAll(svc.PreCondition(), ServiceRunning(svc.Dependencies()...)) cnd.Wait(ctx) svc.Runner().Run(eventSink) }(svc) }

Slide 63

Slide 63 text

Service: getting things together for _, svc := range services { go func(svc Service) { cnd := WaitForAll(svc.PreCondition(), ServiceRunning(svc.Dependencies()...)) cnd.Wait(ctx) svc.Runner().Run(eventSink) }(svc) }

Slide 64

Slide 64 text

Building

Slide 65

Slide 65 text

Building OS in 3 minutes Parallel builds (stages, go parallelization over packages) Efficient caching (Go build cache, buildkitd caching) Go compiler is blazingly fast!

Slide 66

Slide 66 text

Build Graph toolchain kernel tools Go sources init container images initramfs rootfs OS

Slide 67

Slide 67 text

Build Graph toolchain kernel tools Go sources init container images initramfs rootfs OS

Slide 68

Slide 68 text

Building with buildkit tl;dr: better docker build Environment-agnostic reproducible builds Efficient caching Internal parallelization Shared cache

Slide 69

Slide 69 text

Slow Go modules download COPY ./ ./

Slide 70

Slide 70 text

Slow Go modules download COPY ./ ./ RUN go mod download

Slide 71

Slide 71 text

Slow Go modules download COPY ./ ./ RUN go mod download RUN go mod verify RUN go build ./...

Slide 72

Slide 72 text

Slow Go modules download COPY ./ ./ RUN go mod download RUN go mod verify RUN go build ./... SLOOOOW! Any source code change invalidates build cache

Slide 73

Slide 73 text

Caching Go modules download COPY ./go.mod ./ COPY ./go.sum ./ RUN go mod download RUN go mod verify

Slide 74

Slide 74 text

Caching Go modules download COPY ./go.mod ./ COPY ./go.sum ./ RUN go mod download RUN go mod verify COPY ./cmd ./cmd COPY ./pkg ./pkg RUN go build ./...

Slide 75

Slide 75 text

Caching Go modules download COPY ./go.mod ./ COPY ./go.sum ./ RUN go mod download RUN go mod verify COPY ./cmd ./cmd COPY ./pkg ./pkg RUN go build ./... cached until Go modules requirements change cache invalidates with source code changes

Slide 76

Slide 76 text

Caching Go modules go.{mod,sum} $ go mod download $ go mod verify ./src $ go build go.{mod,sum} $ go mod download $ go mod verify ./src $ go build

Slide 77

Slide 77 text

Mounting Go build cache Mounted volume doesn’t affect build cache and persists across builds # syntax = docker/dockerfile-upstream:1.1.2-experimental RUN --mount=type=cache,target=/root/.cache/go-build go build ./...

Slide 78

Slide 78 text

Caching tools COPY ./src . RUN go build ./... RUN curl -sfL .../golangci-lint.sh | bash -s -- -b /bin v1.16.0 RUN golangci-lint run

Slide 79

Slide 79 text

Caching tools FROM base AS tools RUN curl -sfL .../golangci-lint.sh | bash -s -- -b /bin v1.16.0

Slide 80

Slide 80 text

Caching tools FROM base AS tools RUN curl -sfL .../golangci-lint.sh | bash -s -- -b /bin v1.16.0 FROM tools AS src COPY ./src .

Slide 81

Slide 81 text

Caching tools FROM base AS tools RUN curl -sfL .../golangci-lint.sh | bash -s -- -b /bin v1.16.0 FROM tools AS src COPY ./src . FROM src AS build RUN go build ./... FROM src AS lint RUN golangci-lint run

Slide 82

Slide 82 text

Caching tools tools src run build run linter image golangci-lint

Slide 83

Slide 83 text

Testing

Slide 84

Slide 84 text

Testing OS ● unit-tests: ○ algorithms, concurrent code (race detector!), hard to hit conditions. ● integration: ○ boot Talos in local mode as set of docker containers. ○ integration test as part of the build. ● Cluster API: ○ boot inception cluster, create more clusters on different cloud providers via CAPI. ● conformance: ○ run conformance tests against cluster. ● upgrade tests: ○ upgrade existing Talos cluster. ● bare-metal tests: ○ hardware, networking, various configurations.

Slide 85

Slide 85 text

Unit-tests in buildkit ‘Unit-tests’ are not quite ‘unit’ Some tests require containerd Should be same containerd as OS: requires rootfs Run tests in buildkit! Russian doll: buildkit - runc - test - containerd - runc - container

Slide 86

Slide 86 text

Running steps in insecure mode # syntax = docker/dockerfile-upstream:1.1.2-experimental RUN --security=insecure go test ./... Equivalent to “docker run --privileged” Contributed back to buildkit Russian doll now possible!

Slide 87

Slide 87 text

Integration tests 1. Bring up Talos cluster (3 masters + 1 worker, HA control plane) as Docker containers 2. Wait for kubeadm to boot up Kubernetes cluster 3. Deploy Kubernetes addons (CNI, DNS, …) 4. Wait for all the nodes to report ‘Ready’ Overall time: ~5 mins

Slide 88

Slide 88 text

Summary Talos OS - currently in beta, certified K8s installer Go: concurrency, correctness, integration, build speed Fast: seconds to boot Minimal: OS image < 150 MB Immutable: read-only filesystem, easy upgrades Hardened & Secure: KSPP, no ssh, extra bloat, auth via certs (API)

Slide 89

Slide 89 text

Thanks! GitHub: https://github.com/talos-systems/talos Project: https://www.talos-systems.com Contributions are welcome! me: @smira Go gopher by Renee French, licensed under Creative Commons 3.0 Attributions license.