MachineConfigs and GitOps Working machineconfig and machineconfigpools in GitOps GitOps Happy Hour 1 Christian Hernandez Cloud Platforms BU

Future OpenShift Cluster Management 2 Cloud API Machine Deployment Controller MachineDeployment Machine Set Controller MachineSet Machine Controller Machine Cloud Instance NodeLink Controller Node Bootstrap

OpenShift Cluster Management | Machine Configuration 3 OS configuration is stored and applied across the cluster via the Machine Config Operator. ● Subset of ignition modules applicable post provisioning ○ SSH keys ○ Files ○ systemd units ○ kernel arguments ● Standard k8s YAML/JSON manifests ● Desired state of nodes is checked/fixed regularly ● Can be paused to suspend operations Machine Config Operator A Kube-native way to configure hosts # test.yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: test-file spec: config: storage: files: - contents: source: data:,hello%20world%0A verification: {} filesystem: root mode: 420 path: /etc/test

Node Machine Config Daemon Node Machine Config Daemon Operator/Operand Relationships OpenShift Cluster Management | Machine Configuration Node 4 Machine Config Daemon Machine Config Operator Machine Config Controller Machine Config Server

OpenShift Cluster Management | Machine Configuration 5 Machine Config and Machine Config Pool Inheritance-based mapping of configuration to nodes 50-kargs role:worker 5-chrony role:worker 50-motd role:worker Rendered config: rendered-worker-

Pool: role:worker OpenShift Cluster Management | Machine Configuration 6 Custom Machine Config Pools Hierarchical/layered configuration rendering 50-args /etc/args role:worker 5-chrony /etc/ntp.conf role:worker 50-motd /etc/motd role:worker Pool: role:highperf 60-args /etc/args role:highperf 5-other /etc/other.conf role:highperf 51-motd /etc/motd role:worker files: 5-chrony: /etc/ntp.conf 5-other: /etc/other.conf 50-args: /etc/args 50-motd: /etc/motd 51-motd: /etc/motd 60-args: /etc/args rendered-highperf-

OpenShift Cluster Management | Machine Configuration 7 Machine Config Server Providing Ignition configuration for provisioning rendered-worker- {.spec.config} VM / Server Ignition “worker.ign” RHCOS Image Machine Config Server Instance Metadata: https://api-int.xxx.local:22623/config/worker

OpenShift Cluster Management | Machine Configuration 8 Machine Config Server Identical nodes at massive scale New Workers ……. Existing Workers rendered-worker- {.spec.config} Machine Config Server

OpenShift Cluster Management | Machine Configuration 9 Machine Config Daemon Preventing drift Machine Config Daemon 50-registries role:worker 5-chrony role:worker 50-motd role:worker Rendered config: rendered-worker- /etc/containers/registries.conf /etc/chrony.conf /etc/motd

OpenShift Cluster Management | Machine Configuration 10 Machine Config Daemon Acting on drift The MCO coordinates with the MCD to perform the following actions, in a rolling manner, when OS updates and/or configuration changes are applied: ● Cordon / uncordons nodes ● Drain pods ● Stage node changes ○ OS upgrade ○ config changes ○ systemd units ● Reboot 1. Validates node state matches desired state 2. Validate cluster state & policy to apply change 3. Change is rolled across cluster OS_VERSION = != MaxUnavailable = 1

OpenShift Cluster Management | Machine Configuration 11 Transactional updates ensure that RHEL CoreOS is never altered during runtime. Rather it is booted directly into an always “known good” version. ● Each OS update is versioned and tested as a complete image. ● OS binaries (/usr) are read-only ● OS updates encapsulated in container images ● file system and package layering available for hotfixes and debugging Transactional updates with rpm-ostree

OpenShift Cluster Management 12 Over-the-air updates: Cluster Components ... some-component ... ... Cluster Version Operator Machine Config Operator Machine Config Operator Operands Some Operator Release Payload Info Upgrade Process

OpenShift Cluster Management 13 Over-the-air updates: Nodes ... machine-config-operator machine-os-content ... Cluster Version Operator Machine Config Operator Machine Config Operator Machine Config Daemons Machine Config Operator Rolling Machine Config Daemon Download and mount update content into host Machine Config Daemon Update host using mounted content Release Payload Info

Let’s Explore! Hacking away at it until it works. GitOps Happy Hour 14 Keyboard time!