Implementing OS in Go

Implementing operating system in Go - what it means, toolchain, builds, fun facts, some developer notes.

1. Как написать операционную систему на Go Андрей Смирнов Implementing OS in Go (Andrey Smirnov)
2. Andrey Smirnov GitHub, Twitter: @smira Contributor to Talos project
3. Talos: OS for Kubernetes in Go
4. Why another OS for Kubernetes? Linux distribution Kubernetes
5. Why another OS for Kubernetes? Linux distribution Kubernetes We want this
6. Why another OS for Kubernetes? Linux distribution Kubernetes We want this We have to support this
7. Automation to build Linux distribution Kubernetes Ansible, Chef, VM Template...
8. Configuration drift Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes
9. Configuration drift Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes
10. Kubernetes in the cloud Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes
11. Kubernetes in the cloud Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes Linux distribution Kubernetes
12. Why another OS for Kubernetes? Package management issues, configuration drift across nodes Inconsistent Kubernetes experience across clouds and bare-metal Security/compliance issues due to extra bloat in the OS Requires automation tools to manage the cluster Hard to keep up with Kubernetes release schedule
13. What is Talos? Open-source modern OS designed for Kubernetes (only) Linux kernel + user-space in Go Talos Kubernetes Linux kernel User-space (Go)
14. Talos goals Minimal: no bloat Fast: fast to boot Hardened: security best practices applied by default Secure: PKI infrastructure for auth Immutable: read-only rootfs
15. API-driven Talos Talos Talos No shell access gRPC API
16. Bare metal Virtual machines Clouds: Amazon, Google, Azure Docker (testing) Talos platforms
17. Talos Architecture Linux kernel init machined services containerd ntpd networkd udevd osd trustd proxyd kubelet kubeadm etcd kube-apiserver customer workloads initramfs rootfs/ squashfs KSPP mounts userdata Talos K8s
18. Talos Architecture Linux kernel Talos K8s
19. Talos Architecture Linux kernel init initramfs Talos K8s
20. Talos Architecture Linux kernel init machined services initramfs rootfs/ squashfs KSPP mounts userdata Talos K8s
21. Talos Architecture Linux kernel init machined services containerd ntpd networkd udevd osd trustd proxyd initramfs rootfs/ squashfs KSPP mounts userdata Talos K8s
22. Talos Architecture Linux kernel init machined services containerd ntpd networkd udevd osd trustd proxyd kubelet kubeadm initramfs rootfs/ squashfs KSPP mounts userdata Talos K8s
23. Talos Architecture Linux kernel init machined services containerd ntpd networkd udevd osd trustd proxyd kubelet kubeadm etcd kube-apiserver customer workloads initramfs rootfs/ squashfs KSPP mounts userdata Talos K8s
24. What is common between these?
27. Why doing OS in Go is so cool? Easy to reason about the code
28. Why doing OS in Go is so cool? Easy to reason about the code Static linking
29. Why doing OS in Go is so cool? Easy to reason about the code Static linking Great concurrency primitives
30. Why doing OS in Go is so cool? Easy to reason about the code Static linking Great concurrency primitives Wrappers for low-level OS primitives (e.g. netlink)
31. Why doing OS in Go is so cool? Easy to reason about the code Static linking Great concurrency primitives Wrappers for low-level OS primitives (e.g. netlink) Easy integration with other services in Go
32. Concurrency
33. Concurrency in OS API servers Event handling Startup (optimizing boot time)
34. Service Startup containerd networkd kubeadm kubelet ntpd kubeadm-flags.env osd udevd proxyd ca.crt trustd
35. Service A way to run the service (Runner) Service state, health Service logs Service dependencies Service control (stop, start, restart) Service pre-run hooks (prepare)
36. Service Runner Process Container via containerd API (oci) Container via CRI Goroutine Automatic restarts
37. Runner API - v.0 type Runner interface { Run() error } Run() returns when runnable object terminates
38. Runner API - v.1 type Runner interface { Run(ctx context.Context) error } type Runner interface { Run() error Stop() } Run() returns when runnable object terminates
39. Runner API v.2 type EventSink func(event Event) type Runner interface { Run(eventSink EventSink) error Stop() } [Waiting]: Waiting for service "containerd" to be "up", service "kubeadm" to be "up", file "/var/lib/kubelet/kubeadm-flags.env" to exist (1m28s ago) [Waiting]: Waiting for service "kubeadm" to be "up", file "/var/lib/kubelet/kubeadm-flags.env" to exist (1m26s ago) [Waiting]: Waiting for file "/var/lib/kubelet/kubeadm-flags.env" to exist (52s ago) [Preparing]: Running pre state (36s ago) [Preparing]: Creating service runner (36s ago) [Running]: Started task kubelet (PID 551) for container kubelet (36s ago) [Running]: Health check failed: Get http://127.0.0.1:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused (34s ago) [Running]: Health check successful (29s ago)
40. Auto-restarting runner type Restarter struct { wrapped Runner } func (rr *Restarter) Run(eventSink EventSink) error { for { err := rr.wrapped.Run(eventSink) eventSink(Event(“restarting due to %v”, err)) time.Sleep(time.Second) } }
41. Auto-restarting runner type Restarter struct { wrapped Runner } func (rr *Restarter) Run(eventSink EventSink) error { for { err := rr.wrapped.Run(eventSink) eventSink(Event(“restarting due to %v”, err)) time.Sleep(time.Second) } }
42. Auto-restarting runner type Restarter struct { wrapped Runner } func (rr *Restarter) Run(eventSink EventSink) error { for { err := rr.wrapped.Run(eventSink) eventSink(Event(“restarting due to %v”, err)) time.Sleep(time.Second) } }
43. Auto-restarting runner type Restarter struct { wrapped Runner } func (rr *Restarter) Run(eventSink EventSink) error { for { err := rr.wrapped.Run(eventSink) eventSink(Event(“restarting due to %v”, err)) time.Sleep(time.Second) } }
44. Auto-restarting runner: with stop type Restarter struct { wrapped Runner stop chan struct{} } func (rr *Restarter) Stop() { close(rr.stop) }
45. Auto-restarting runner: with stop type Restarter struct { wrapped Runner stop chan struct{} } func (rr *Restarter) Stop() { close(rr.stop) }
46. Auto-restarting runner: with stop for { errCh := make(chan error) go func() { errCh <- rr.wrapped.Run(eventSink) } select { case <-rr.stop: rr.wrapped.Stop() return <-errCh case err := <- errCh: eventSink(Event(“restarting due to %v”, err)) } time.Sleep(time.Second) }
47. Auto-restarting runner: with stop for { errCh := make(chan error) go func() { errCh <- rr.wrapped.Run(eventSink) } select { case <-rr.stop: rr.wrapped.Stop() return <-errCh case err := <- errCh: eventSink(Event(“restarting due to %v”, err)) } time.Sleep(time.Second) }
48. Auto-restarting runner: with stop for { errCh := make(chan error) go func() { errCh <- rr.wrapped.Run(eventSink) } select { case <-rr.stop: rr.wrapped.Stop() return <-errCh case err := <- errCh: eventSink(Event(“restarting due to %v”, err)) } time.Sleep(time.Second) }
49. Auto-restarting runner: with stop for { errCh := make(chan error) go func() { errCh <- rr.wrapped.Run(eventSink) } select { case <-rr.stop: rr.wrapped.Stop() return <-errCh case err := <- errCh: eventSink(Event(“restarting due to %v”, err)) } time.Sleep(time.Second) }
50. Auto-restarting runner: with stop for { errCh := make(chan error) go func() { errCh <- rr.wrapped.Run(eventSink) } select { case <-rr.stop: rr.wrapped.Stop() return <-errCh case err := <- errCh: eventSink(Event(“restarting due to %v”, err)) } time.Sleep(time.Second) }
51. Auto-restarting runner: with stop for { errCh := make(chan error) go func() { errCh <- rr.wrapped.Run(eventSink) } select { case <-rr.stop: rr.wrapped.Stop() return <-errCh case err := <- errCh: eventSink(Event(“restarting due to %v”, err)) } time.Sleep(time.Second) }
52. Service type Service interface { Id() string Runner() Runner Dependencies() []string PreCondition() Condition }
53. Condition type Condition interface { fmt.Stringer Wait(ctx context.Context) error } Condition represents anything which can be waited for Conditions are composable [Waiting]: Waiting for service "containerd" to be "up", service "kubeadm" to be "up", file "/var/lib/kubelet/kubeadm-flags.env" to exist (1m28s ago) [Waiting]: Waiting for service "kubeadm" to be "up", file "/var/lib/kubelet/kubeadm-flags.env" to exist (1m26s ago) [Waiting]: Waiting for file "/var/lib/kubelet/kubeadm-flags.env" to exist (52s ago)
54. Example Conditions func FileExists(path string) Condition func ServiceRunning(id string) Condition
55. Condition composition func FileExists(path string) Condition func ServiceRunning(id string) Condition func WaitForAll(conditions ...Condition) Condition
56. Condition composition func FileExists(path string) Condition func ServiceRunning(id string) Condition func WaitForAll(conditions ...Condition) Condition condition := WaitForAll(ServiceRunning("trustd"), FileExists("/var/lib/kubelet/kubeadm-flags.env"))
57. Condition composition type waitAll struct{ conditions []Condition } func WaitForAll(conditions ...Condition) Condition { return &waitAll{conditions: conditions} }
58. Condition composition func (w *waitAll) Wait(ctx context.Context) error { errCh := make(chan error) for _, cond := range w.conditions { go func(cond Condition) { errCh <- cond.Wait(ctx) }(cond) } var multiErr multierror.Error for range w.conditions { multiErr := multierror.Append(multiErr, <-errCh) } return multiErr.ErrorOrNil() }
59. Service: getting things together for _, svc := range services { go func(svc Service) { cnd := WaitForAll(svc.PreCondition(), ServiceRunning(svc.Dependencies()...)) cnd.Wait(ctx) svc.Runner().Run(eventSink) }(svc) }
60. Service: getting things together for _, svc := range services { go func(svc Service) { cnd := WaitForAll(svc.PreCondition(), ServiceRunning(svc.Dependencies()...)) cnd.Wait(ctx) svc.Runner().Run(eventSink) }(svc) }
61. Service: getting things together for _, svc := range services { go func(svc Service) { cnd := WaitForAll(svc.PreCondition(), ServiceRunning(svc.Dependencies()...)) cnd.Wait(ctx) svc.Runner().Run(eventSink) }(svc) }
62. Service: getting things together for _, svc := range services { go func(svc Service) { cnd := WaitForAll(svc.PreCondition(), ServiceRunning(svc.Dependencies()...)) cnd.Wait(ctx) svc.Runner().Run(eventSink) }(svc) }
63. Service: getting things together for _, svc := range services { go func(svc Service) { cnd := WaitForAll(svc.PreCondition(), ServiceRunning(svc.Dependencies()...)) cnd.Wait(ctx) svc.Runner().Run(eventSink) }(svc) }
64. Building
65. Building OS in 3 minutes Parallel builds (stages, go parallelization over packages) Efficient caching (Go build cache, buildkitd caching) Go compiler is blazingly fast!
66. Build Graph toolchain kernel tools Go sources init container images initramfs rootfs OS
67. Build Graph toolchain kernel tools Go sources init container images initramfs rootfs OS
68. Building with buildkit tl;dr: better docker build Environment-agnostic reproducible builds Efficient caching Internal parallelization Shared cache
69. Slow Go modules download COPY ./ ./
70. Slow Go modules download COPY ./ ./ RUN go mod download
71. Slow Go modules download COPY ./ ./ RUN go mod download RUN go mod verify RUN go build ./...
72. Slow Go modules download COPY ./ ./ RUN go mod download RUN go mod verify RUN go build ./... SLOOOOW! Any source code change invalidates build cache
73. Caching Go modules download COPY ./go.mod ./ COPY ./go.sum ./ RUN go mod download RUN go mod verify
74. Caching Go modules download COPY ./go.mod ./ COPY ./go.sum ./ RUN go mod download RUN go mod verify COPY ./cmd ./cmd COPY ./pkg ./pkg RUN go build ./...
75. Caching Go modules download COPY ./go.mod ./ COPY ./go.sum ./ RUN go mod download RUN go mod verify COPY ./cmd ./cmd COPY ./pkg ./pkg RUN go build ./... cached until Go modules requirements change cache invalidates with source code changes
76. Caching Go modules go.{mod,sum} $ go mod download $ go mod verify ./src $ go build go.{mod,sum} $ go mod download $ go mod verify ./src $ go build
77. Mounting Go build cache Mounted volume doesn’t affect build cache and persists across builds # syntax = docker/dockerfile-upstream:1.1.2-experimental RUN --mount=type=cache,target=/root/.cache/go-build go build ./...
78. Caching tools COPY ./src . RUN go build ./... RUN curl -sfL .../golangci-lint.sh | bash -s -- -b /bin v1.16.0 RUN golangci-lint run
79. Caching tools FROM base AS tools RUN curl -sfL .../golangci-lint.sh | bash -s -- -b /bin v1.16.0
80. Caching tools FROM base AS tools RUN curl -sfL .../golangci-lint.sh | bash -s -- -b /bin v1.16.0 FROM tools AS src COPY ./src .
81. Caching tools FROM base AS tools RUN curl -sfL .../golangci-lint.sh | bash -s -- -b /bin v1.16.0 FROM tools AS src COPY ./src . FROM src AS build RUN go build ./... FROM src AS lint RUN golangci-lint run
82. Caching tools tools src run build run linter image golangci-lint
83. Testing
84. Testing OS • unit-tests: ◦ algorithms, concurrent code (race detector!), hard to hit conditions. • integration: ◦ boot Talos in local mode as set of docker containers. ◦ integration test as part of the build. • Cluster API: ◦ boot inception cluster, create more clusters on different cloud providers via CAPI. • conformance: ◦ run conformance tests against cluster. • upgrade tests: ◦ upgrade existing Talos cluster. • bare-metal tests: ◦ hardware, networking, various configurations.
85. Unit-tests in buildkit ‘Unit-tests’ are not quite ‘unit’ Some tests require containerd Should be same containerd as OS: requires rootfs Run tests in buildkit! Russian doll: buildkit - runc - test - containerd - runc - container
86. Running steps in insecure mode # syntax = docker/dockerfile-upstream:1.1.2-experimental RUN --security=insecure go test ./... Equivalent to “docker run --privileged” Contributed back to buildkit Russian doll now possible!
87. Integration tests 1. Bring up Talos cluster (3 masters + 1 worker, HA control plane) as Docker containers 2. Wait for kubeadm to boot up Kubernetes cluster 3. Deploy Kubernetes addons (CNI, DNS, …) 4. Wait for all the nodes to report ‘Ready’ Overall time: ~5 mins
88. Summary Talos OS - currently in beta, certified K8s installer Go: concurrency, correctness, integration, build speed Fast: seconds to boot Minimal: OS image < 150 MB Immutable: read-only filesystem, easy upgrades Hardened & Secure: KSPP, no ssh, extra bloat, auth via certs (API)
89. Thanks! GitHub: https://github.com/talos-systems/talos Project: https://www.talos-systems.com Contributions are welcome! me: @smira Go gopher by Renee French, licensed under Creative Commons 3.0 Attributions license.