Functional Operations

Maintaining configurations for different kinds of nodes and cloud resources in a [micro]service architecture can be an operational nightmare, especially if not managed with the application codebase. CI and CD job environments diverge from production configuration yielding their results unpredictable at best or produce false positives in the worst case. Code pushes to staging and production can have unintended consequences that can't be reasoned about before deploy and often can’t be inspected thoroughly on a dry run. Leading to unhappy users when problems do arise.

This session will demonstrate the use of the Nix and NixOS ecosystem to define and build packages in a referentially transparent way which can be leveraged as a solid foundation to configure systems and test multiple [virtual] machines with coordinated scenarios. We also look at how reliable packaging allows us to build a consistent CI/CD pipeline where upgrading your version of the JVM doesn't break your CI build servers for days.

Transcript

1. Functional Operations Functional Programming @ Comcast Labs Connect Susan Potter (@SusanPotter) 2018-03-09 Fri
3. Un-Fun Ops 0
4. Core InfraEng Problems 1
5. Core InfraEng Problems • non-repeatable environments • unpredictable machine rollback • out-of-band updating of CI dependencies • lack of muti-node integration testing 1
6. Causes 2
7. Causes • no single source of truth • partial definitions yield inconsistencies (over time) • shared mutable state (file manipulation most Linux/UNIX distros) 2
8. Nix: Reliable Packaging
9. Nix: What is it? 3
10. Nix: What is it? • "The Purely Functional Package Manager" (https://nixos.org/nix) 3
11. Nix: What is it? • "The Purely Functional Package Manager" (https://nixos.org/nix) • (And lazily evaluated expression language) 3
12. Nix: Source Example This Nix lambda denotes the hello package # Named keyword arguments: all non -optional { stdenv , fetchurl , perl }: let version = "2.10"; in stdenv.mkDerivation { inherit perl; name = "hello -${version}"; builder = ./builder.sh; src = fetchurl { url = "mirror://gnu/hello/${name}.tar.gz"; sha256 = "0ssi1wpaf7plaswqqjw...l7c9lng89ndq1i"; }; } 4
13. Nix: Derivation drvPath Inspecting hello package drvPath $ nix -repl ’<nixpkgs >’ nix -repl > hello.drvPath "/nix/store /3mdx...-hello -2.10. drv" nix -repl > hello.outPath "/nix/store/kmwd ...-hello -2.10" nix -repl > :q $ stat -t /nix/store /3mdx...-hello -2.10. drv /nix/store /3mdx...-hello -2.10. drv 1219 8 8124 0 0 fe00 13940452 ... 5
14. Nix: Package outPath Lazy evaluation means we only build the packages we need $ stat -t /nix/store/kmwd ...-hello -2.10 stat: cannot stat ’/nix/store/kmwd ...-hello -2.10 ’: No such file or ... To evaluate or realize the package we nix build $ nix build nixpkgs.hello [1 copied (0.2 MiB), 0.0 MiB DL] Now we can see the hello package outPath in the Nix store $ stat -t /nix/store/kmwd ...-hello -2.10 /nix/store/kmwd ...-hello -2.10 4096 8 416d 0 0 fe00 14115111 4 0 0 ... 6
15. Nix: Package outPath The high-level layout of the hello.outPath $ tree --du -hugFDL 2 /nix/store/kmwd ...-hello -2.10 /nix/store/ kmwd1hq55akdb9sc7l3finr175dajlby -hello -2.10 |-- [root root 33K Dec 31 1969] bin/ | |-- [root root 29K Dec 31 1969] hello* |-- [root root 16K Dec 31 1969] share/ |-- [root root 4.0K Dec 31 1969] info/ |-- [root root 4.0K Dec 31 1969] locale/ |-- [root root 4.0K Dec 31 1969] man/ 53K used in 5 directories , 1 file 7
16. Nix: Package in user profile Since it is built, surely we can run the binary? (No, not yet) $ hello The program ’hello ’ is currently not installed. It is provided by ... To link into the user profile we install ("Look Ma, no hands/sudo") $ nix -env -iA nixos.hello installing ’hello -2.10 ’ building ’/nix/store /6 s28kd3hdnv5cpd ...1yi5 -user -environment.drv ’... created 6852 symlinks in user environment $ hello -v hello (GNU Hello) 2.10 8
17. Nix: Properties 9
18. Nix: Properties • Lazily evaluated • Nix store is immutable! • nixpkgs is an expression: snapshot of community pkgs plus overlayed packages 9
19. Nix: Properties • Lazily evaluated • Nix store is immutable! • nixpkgs is an expression: snapshot of community pkgs plus overlayed packages • When packages sandboxed: • chroot -ed • private /proc • disabled network/IPC/etc • no env var inheritance 9
20. Nix: Properties • Lazily evaluated • Nix store is immutable! • nixpkgs is an expression: snapshot of community pkgs plus overlayed packages • When packages sandboxed: • chroot -ed • private /proc • disabled network/IPC/etc • no env var inheritance • Same inputs and src and definition yields same result (outPath and content) • Different inputs or src or definition yields different result (outPath and/or content) • Equational reasoning allows binary substitution 9
21. Nix Shell: Consistent Envs
22. nix-shell: Getting started • CI scripts can use nix-shell as shebang #!/usr/bin/env nix -shell #!nix -shell -p python27Full python27Packages .boto3 #!nix -shell -I /path/to/pinned/nixpkgs #!nix -shell -i python import boto3 ec2 = boto3.client(’ec2’) response = ec2. describe_instances () print(response) Note: Approach requires Nix/NixOS installed on builders 10
23. nix-shell: Running (first time) $ chmod u+x boto.py $ ./ boto.py these paths will be fetched (7.70 MiB download , 45.59 MiB unpacked ): /nix/store/mj8vmgxff9m ...-python -2.7.14 /nix/store/q6js3yms8xd ...-tix -8.4.3 /nix/store/yqgj43ancc0 ...- python2 .7-boto3 -1.4.8 copying path ’/nix/store/yqg...- python2 .7-boto3 -1.4.8 ’ from ’https :// cache.n copying path ’/nix/store/q6j...-tix -8.4.3 ’ from ’https :// cache.nixos.org ’... copying path ’/nix/store/mj8...-python -2.7.14 ’ from ’https :// cache.nixos.org OUTPUT HERE 11
24. nix-shell: Running (subsequent times) $ ./ boto.py OUTPUT HERE 12
25. Nix Shell: Recap • Claim: we can get consistent environment, assuming: • our core Nix expression pins version of nixpkgs • devs reload shell at the git revisions to pick up dependency changes • CI server runs tests using shell.nix referenced by nix-shell at that revision • infrastructure for all environments is built using pinned Nix expression • devenv, CI, production-like system builds share the same core expression 13
26. NixOS
27. What is NixOS? • "The Purely Functional Linux Distribution" (https://nixos.org/) • Provides congruent system-level configuration • System-level rollbacks 14
28. NixOS: Example (nginx) # ./machines/nginx.nix { pkgs , ... }: { boot.kernelPackages = pkgs.linuxPackages_4_15; time.timeZone = "UTC"; services.nginx = { enable = true; virtualHosts."my.domain.tld" = { forceSSL = true; enableACME = true; locations."/assets".root = "/var/www/assets"; locations."/".proxyPass = "http://backend -lb:3000"; }; }; } 15
29. NixOS: Example (Scala microservice) # ./machines/scala -services.nix { pkgs , ... }: let inherit (builtins) listToAttrs map; inherit (pkgs.myorg) scalaServices mulib; in { # Interal custom myorg module; overlayed into pkgs myorg.defaults.enable = true; users.extraUsers = map mulib.mkSystemUser scalaServices; systemd.services = map mulib.mkServiceUnit scalaServices; } 16
30. NixOS: Example (memcached) # ./machines/memcached.nix { config , pkgs , ... }: { services.memcached.enable = true; services.memcached.maxConnections = 2048; services.memcached.maxMemory = 256; services.memcached.listen = config.networking.eth1.ipAddress; } 17
31. NixOS: Containers (composition) { config , pkgs , ... }: { containers.cache.config = import ./memcached.nix; containers.proxy.config = import ./nginx.nix; containers.services = { config = import ./scala -services.nix; bindMounts."/webapp" = { hostPath = pkgs.myorg.scalaServices.webapp.outPath; isReadOnly = true; }; }; } 18
32. NixOS: Multi-node testing (node setup) import ./make -test.nix ({ pkgs , lib , ... }: let kafkaPackage = pkgs.apacheKafka_1_0; in { name = "kafka -1.0"; nodes = { zookeeper = import ./machines/zookeeper.nix; kafka = import ./machines/kafka.nix; }; testScript = import ./tests/kafka.nix { package = kafkaPackage; util = pkgs.myorg.testing.util; }; }) 19
33. NixOS: Multi-node testing (test script) { pkg , kPort ? 9092 , zkPort ? 2181 , topic ? "test" , util }: let inherit (util) createTopicCmd producerPipe consumerPipe; in ’’ startAll; $zookeeper ->waitForUnit("zookeeper"); $zookeeper ->waitForOpenPort(${zkPort}); $kafka ->waitForUnit("apache -kafka"); $kafka ->waitForOpenPort(${kPort}); $kafka ->waitUntilSucceeds("${createTopicCmd}"); $kafka ->mustSucceed("echo ’test 1’ | ${producerPipe}"); $kafka ->mustSucceed("${consumerPipe} | grep ’test 1 ’"); ’’ 20
34. NixOS & nixpkgs: Properties • modular (See nixpkgs overlays) • extensible (see nixos modules) • everything (almost) is a package in the Nix store • system rollbacks (with system profile snapshots saved as generations) • builds on purity of Nix packaging • NO FHS mutation-in-place (links into FHS to immutable Nix store) 21
35. Reliability
36. Reliable Software "Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result the programming process will become cheaper." – EWD 22
37. Why do properties/laws matter? 23
38. Why do properties/laws matter? • Because it allows us humble programmers to: 23
39. Why do properties/laws matter? • Because it allows us humble programmers to: • compose our reasoning • know what we cannot reason about • break down big problems into smaller parts • accomplish the same result with less effort 23
40. Questions? @SusanPotter (heckle me, ask me questions) 24