Implementing configuration management primitives in 2024

🎥 https://www.youtube.com/watch?v=yTECEJ2FVW8
🧑 Alexis Mousset
📅 Configuration Management Camp 2024

Configuration management primitives appear like a solved topic now, and current major solutions have converged to pretty similar choices 10+ years ago. However new needs are becoming more prominent, like observability, auditing and self-auditing abilities, in a context of growing attention for security topics. Could we benefit from reconsidering some of these design choices now to better address them?

In this talk, we will navigate through the solution space of configuration management low-level implementations (resource/promise/etc.), and explore what we can modify to provide new promising features. It will also cover implementation and programming language choices, from C to Python, Ruby, and Rust, and how these choices participate in shaping our tools strengths and weaknesses. It will feature some examples from ongoing work in Rudder, as well as other projects (mgmt, Jet, etc.)

Transcript

1. Implementing configuration management primitives (as of 2024) Alexis Mousset 6th February 2024
2. CfgMgmt Camp CfgMgmtCamp 2024 2
3. CfgMgmt Camp YAML Camp CfgMgmtCamp 2024 3
4. CfgMgmt Camp YAML Camp checkApply() Camp CfgMgmtCamp 2024 4
5. checkApply def checkApply(): if is_ok(state): do_nothing() else: fix(state) • Idempotency building block • State is a global variable • This is what infra automation is all about • What’s new in the checkApply world? • And is it still relevant? CfgMgmtCamp 2024 5
6. Who am i? • System developer • Developer at Rudder since 2015 • CFEngine contributor • Ansible & Puppet user CfgMgmtCamp 2024 6
7. Configuration management primitives CfgMgmtCamp 2024 7
8. Raw material? CfgMgmtCamp 2024 8
10. Raw material: interfaces & configuration data • configuration data -> checkApply() -> system APIs • Interact with existing “programming” interfaces • Rather high-level • Glue, really • Configuration data can be declarative or a program • Not different at the lowest level CfgMgmtCamp 2024 10
11. Which interface do we use on the Linux systems? CfgMgmtCamp 2024 11
12. Which interface do we use on the Linux systems? Unix v7, really (actually ~ POSIX) CfgMgmtCamp 2024 11
13. The Unix programming interface • C (too difficult) • Text interfaces • Commands (in shells) • Maybe even sockets • Everything is a file • That’s (pretty much) it! CfgMgmtCamp 2024 12
14. Since Unix 7 • More orientation towards declarative configuration and data • init scripts -> systemd units • dedicated configuration languages -> YAML/JSON/TOML/etc. • Libraries • systemd • package managers CfgMgmtCamp 2024 13
15. What does Cfg Mgmt do? • All CM tools parse commands outputs with regexes and a lot of special cases • The internals are usually messy • Hides this pretty successfully from the user • But the abstractions are leaky CfgMgmtCamp 2024 14
16. Side note: configuration surface What does configuring a “system” mean? CfgMgmtCamp 2024 15
18. Configuration surface of a Linux system • A kernel, including a lot of runtime config • 1629 values in sysctl • nftables rules • eBPF programs • A file system • States everywhere CfgMgmtCamp 2024 17
19. Configuration surface of a Linux system • Config Management only manages a tiny part of it • For the rest, we don’t know • Is it a problem? • Are containers a solution? CfgMgmtCamp 2024 18
20. Has Cfg Mgmt (partially) failed? • Immutable infrastructure • Dockerfiles are shell scripts • Trading flexibility and power for simplicity • Could have Cfg Mgmt provided an interoperable and reusable higher level interface as a commodity? • Was the promise of abstracting system configuration kept? CfgMgmtCamp 2024 19
21. Solution space • Config Mgmgt = an engine passing parameters to resources providers • What are the choices we make when implementing resources? • What are the tradeoffs? • Are there unexplored areas?
22. Engine • Handles data management • Load properties • String substitution • Out of scope here • Calls the resources • Usually a big checkApply • A stack structure usually • A graph sometimes? CfgMgmtCamp 2024 21
23. A basic resource implementation def checkApply(): if is_ok(state): do_nothing() else: fix(state) CfgMgmtCamp 2024 22
24. A first choice def engine(): if not check(): apply() #### def check(): is_ok(state) def apply(): fix(state) CfgMgmtCamp 2024 23
25. Dry-run / audit / check • Allow to see what would change if the agent ran def checkApply(audit): if is_ok(state): do_nothing() else: if audit: error() else: fix(state) CfgMgmtCamp 2024 24
26. Static checks fn checkApply(audit: bool, expectedState: DirectoryResource): if state.is_ok(expectedState) { do_nothing() } else { if audit { error() } else { state.fix(expectedState) } } CfgMgmtCamp 2024 25
27. Specs and typing for parameters • User experience • Reliability • Shifting problems left struct DirectoryResource { path: Path, state: Presence, permissions: Permissions, } CfgMgmtCamp 2024 26
28. Return value • We need a standardization • Required or proper user feedback and conditional actions • Most basic version enum ResourceOutcome { Ok(...), Repaired(...), Error(...), } CfgMgmtCamp 2024 27
29. Generalization • To what extent to we generalize? • Should resources be multi-platform? • Should we cover all options? • Or a subset and provides convenient escape hatches? • A “package” resource or dnf, apt, etc. • A “container” resource or “docker” CfgMgmtCamp 2024 28
30. Agent vs. Agentless • What is actually the difference? • There is always a kind of agent when the software runs • It can be copied over SSH and removed afterwards • At low level, no difference • Mainly the presence of a long running process on the system • Push/pull CfgMgmtCamp 2024 29
31. Engine • To what extent are the engine and the resource implementation interleaved? CfgMgmtCamp 2024 30
32. Connectivity • Are resources connected? • If so, how? • Can resource instances be grouped? • Install several packages at once automatically • Can a resource trigger something? • Can a resource get information from nother resource CfgMgmtCamp 2024 31
33. Programming language • Write the resources in language used for the policies • Better for “dev-oriented” users • Flexibility • Use data format or a DSL, and a different language for resources • Hides the complexity • Can limit user ability to hack the system CfgMgmtCamp 2024 32
34. Side node: Tech stack & values Technological choices matters • Properties given by the tech stack • Fast, Portable, Beginner-friendly, etc. • Values carried with the tech stack • Attracts different people • Participates to shape the community CfgMgmtCamp 2024 33
35. Language in infra software • C • Ruby • Python • Go • C++ • Rust CfgMgmtCamp 2024 34
36. Impact • System administrators are not developers • Some languages are oriented towards for software developers • Prevents extension by users • Worse is better? CfgMgmtCamp 2024 35
37. Add a custom resource to software • Hardcoded resources may be hard to add • Extension APIs • Sometimes a “language” version (library) and a “data” version (YAML, etc.) CfgMgmtCamp 2024 36
38. Ansible • AnsibleModule library for Python • JSON for other CfgMgmtCamp 2024 37
39. CFEngine • JSON protocol over stdin/stdout • Any binary can implement it • Spawn the binary at agent start, call N times, close. { "operation": "evaluate_promise", "log_level": "info", "promise_type": "git", "promiser": "/opt/cfengine/masterfiles", "attributes": {"repo": "https://github.com/cfengine/ masterfiles"} } CfgMgmtCamp 2024 38
40. CFEngine • Logging protocol • Outcome • kept • repaired • not_kept CfgMgmtCamp 2024 39
41. CFEngine • Called like a standard resource bundle agent main { my_custom_type: "promiser" paramerter1 => "value1", paramerter2 => "value2", paramerter3 => "value3"; } CfgMgmtCamp 2024 40
42. mgmt type Res interface { CheckApply(apply bool) (checkOK bool, err error) Default() Res Validate() error Init(*Init) error Close() error Watch() error ... } CfgMgmtCamp 2024 41
43. Jet • Ansible-like in Rust • Speed • Reliability • Interesting choices • Shell as unique system interface • (Now abandoned AFAIK) CfgMgmtCamp 2024 42
44. Windows • Interesting stuff is happening • DSC 3.0 • Multiplatform (Windows, macOS, Linux) • Open-source • No dependency on Powershell • Resources can be written in any language • JSON instead of MOF • YAML policies CfgMgmtCamp 2024 43
45. Observability • Growing infrastructure complexity • Describe state and changes in a structured way • Text-based in not enough • The hardest part in implementation • Extract information • Comprehensive error handling CfgMgmtCamp 2024 44
46. Security/Traceability • Supply-chain security • No news is not good news? CfgMgmtCamp 2024 45
47. Isn’t immutability the solution? • The infrastructures are not immutables • A way to model changes • Move the mutation to a higher abstraction layer • Mutability is light and fast • Immutable means frozen, not understood and observable CfgMgmtCamp 2024 46
48. • Parallel with programming language? • Mutability is a major source of bugs • Immutability is a way to prevent them • Most programs are written with mutability eveywhere • E.g. Rust aims at managing mutability better instead of preventing it CfgMgmtCamp 2024 47
49. In Rudder • We have a Unix agent (cf-agent) and a Windows agent (based on Windows native technologies) • We want to be able to mutualize between agents • We believe in strong typing (Scala, Rust, Elm) • We look for realiability and performance • Our users are (mostly) not devs • No big-bang, transition should be progressive • No new agent! CfgMgmtCamp 2024 48
50. In Rudder Choices • Standalone pluggable resources • Implemented in Rust • Using extensibility APIs • A resource: one or more checkApply functions • Try to enrich existing data models • Independent resources for now • No magic features like mgmt CfgMgmtCamp 2024 49
51. In Rudder CFEngine Custom Promise Types • Almost perfect for our needs • Allows moving the complexity from the DSL/data to a programming language • We hadd to add an intermediate API, permitted by the JSON arbitrary data passed • The first level in the parameters JSON is interpreted by the library • A subkey is passed to the resource implementation • Allows passing metadata (machine ID, public key, temp dir, etc.) • Not a problem as we compile .cf policies from YAML CfgMgmtCamp 2024 50
52. In Rudder Language Server Protocol • A standard for implementing IDE features for a language • From VS Code • JSON communication with a binary • Allows plugging to alot of editors with a single implementation • Could we take a form of inspiration from this? • But different incentives and economy CfgMgmtCamp 2024 51
53. In Rudder Rudder • Rust library allowing to build binaries • Implements several agents extension API • Observability and traceability focus • Integrate with existing engines • Additional static validation • YAML data with additional typing and validation CfgMgmtCamp 2024 52
54. In Rudder pub trait ModuleType { fn metadata(&self) -> ModuleTypeMetadata; fn init(&mut self) -> ProtocolResult; fn validate(&self, _parameters: &Parameters) -> ValidateResult; fn check_apply(&mut self, mode: PolicyMode, parameters: &Parameters) -> CheckApplyResult; fn terminate(&mut self) -> ProtocolResult; } pub type ValidateResult = Result<()>; pub enum Outcome { Success(Option<String>), Repaired(String), } CfgMgmtCamp 2024 53
55. In Rudder Inventory • Module types can be source of inventories • Either targeted or global • Structured output in checkApply • Diff CfgMgmtCamp 2024 54
56. In Rudder Other perspectives • NixOS/NixOps, Guix • Controled mutability • Microkernel • Specialized systems • e.g.: Talos Linux • No SSH • Only an HTTP API • Specialized use case CfgMgmtCamp 2024 55
57. In Rudder Thank you! • Keep exploring • Not over yet • Have fun! These slides will be available at: amousset.me/checkApply CfgMgmtCamp 2024 56