Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Functional Operations

Susan Potter
March 28, 2018
320

Functional Operations

Maintaining configurations for different kinds of nodes and cloud resources in a [micro]service architecture can be an operational nightmare, especially if not managed with the application codebase. CI and CD job environments diverge from production configuration yielding their results unpredictable at best or produce false positives in the worst case. Code pushes to staging and production can have unintended consequences that can't be reasoned about before deploy and often can’t be inspected thoroughly on a dry run. Leading to unhappy users when problems do arise.

This session will demonstrate the use of the Nix and NixOS ecosystem to define and build packages in a referentially transparent way which can be leveraged as a solid foundation to configure systems and test multiple [virtual] machines with coordinated scenarios. We also look at how reliable packaging allows us to build a consistent CI/CD pipeline where upgrading your version of the JVM doesn't break your CI build servers for days.

Susan Potter

March 28, 2018
Tweet

Transcript

  1. Core InfraEng Problems • non-repeatable environments • unpredictable machine rollback

    • out-of-band updating of CI dependencies • lack of muti-node integration testing 1
  2. Causes • no single source of truth • partial definitions

    yield inconsistencies (over time) • shared mutable state (file manipulation most Linux/UNIX distros) 2
  3. Nix: What is it? • "The Purely Functional Package Manager"

    (https://nixos.org/nix) • (And lazily evaluated expression language) 3
  4. Nix: Source Example This Nix lambda denotes the hello package

    # Named keyword arguments: all non -optional { stdenv , fetchurl , perl }: let version = "2.10"; in stdenv.mkDerivation { inherit perl; name = "hello -${version}"; builder = ./builder.sh; src = fetchurl { url = "mirror://gnu/hello/${name}.tar.gz"; sha256 = "0ssi1wpaf7plaswqqjw...l7c9lng89ndq1i"; }; } 4
  5. Nix: Derivation drvPath Inspecting hello package drvPath $ nix -repl

    ’<nixpkgs >’ nix -repl > hello.drvPath "/nix/store /3mdx...-hello -2.10. drv" nix -repl > hello.outPath "/nix/store/kmwd ...-hello -2.10" nix -repl > :q $ stat -t /nix/store /3mdx...-hello -2.10. drv /nix/store /3mdx...-hello -2.10. drv 1219 8 8124 0 0 fe00 13940452 ... 5
  6. Nix: Package outPath Lazy evaluation means we only build the

    packages we need $ stat -t /nix/store/kmwd ...-hello -2.10 stat: cannot stat ’/nix/store/kmwd ...-hello -2.10 ’: No such file or ... To evaluate or realize the package we nix build $ nix build nixpkgs.hello [1 copied (0.2 MiB), 0.0 MiB DL] Now we can see the hello package outPath in the Nix store $ stat -t /nix/store/kmwd ...-hello -2.10 /nix/store/kmwd ...-hello -2.10 4096 8 416d 0 0 fe00 14115111 4 0 0 ... 6
  7. Nix: Package outPath The high-level layout of the hello.outPath $

    tree --du -hugFDL 2 /nix/store/kmwd ...-hello -2.10 /nix/store/ kmwd1hq55akdb9sc7l3finr175dajlby -hello -2.10 |-- [root root 33K Dec 31 1969] bin/ | |-- [root root 29K Dec 31 1969] hello* |-- [root root 16K Dec 31 1969] share/ |-- [root root 4.0K Dec 31 1969] info/ |-- [root root 4.0K Dec 31 1969] locale/ |-- [root root 4.0K Dec 31 1969] man/ 53K used in 5 directories , 1 file 7
  8. Nix: Package in user profile Since it is built, surely

    we can run the binary? (No, not yet) $ hello The program ’hello ’ is currently not installed. It is provided by ... To link into the user profile we install ("Look Ma, no hands/sudo") $ nix -env -iA nixos.hello installing ’hello -2.10 ’ building ’/nix/store /6 s28kd3hdnv5cpd ...1yi5 -user -environment.drv ’... created 6852 symlinks in user environment $ hello -v hello (GNU Hello) 2.10 8
  9. Nix: Properties • Lazily evaluated • Nix store is immutable!

    • nixpkgs is an expression: snapshot of community pkgs plus overlayed packages 9
  10. Nix: Properties • Lazily evaluated • Nix store is immutable!

    • nixpkgs is an expression: snapshot of community pkgs plus overlayed packages • When packages sandboxed: • chroot -ed • private /proc • disabled network/IPC/etc • no env var inheritance 9
  11. Nix: Properties • Lazily evaluated • Nix store is immutable!

    • nixpkgs is an expression: snapshot of community pkgs plus overlayed packages • When packages sandboxed: • chroot -ed • private /proc • disabled network/IPC/etc • no env var inheritance • Same inputs and src and definition yields same result (outPath and content) • Different inputs or src or definition yields different result (outPath and/or content) • Equational reasoning allows binary substitution 9
  12. nix-shell: Getting started • CI scripts can use nix-shell as

    shebang #!/usr/bin/env nix -shell #!nix -shell -p python27Full python27Packages .boto3 #!nix -shell -I /path/to/pinned/nixpkgs #!nix -shell -i python import boto3 ec2 = boto3.client(’ec2’) response = ec2. describe_instances () print(response) Note: Approach requires Nix/NixOS installed on builders 10
  13. nix-shell: Running (first time) $ chmod u+x boto.py $ ./

    boto.py these paths will be fetched (7.70 MiB download , 45.59 MiB unpacked ): /nix/store/mj8vmgxff9m ...-python -2.7.14 /nix/store/q6js3yms8xd ...-tix -8.4.3 /nix/store/yqgj43ancc0 ...- python2 .7-boto3 -1.4.8 copying path ’/nix/store/yqg...- python2 .7-boto3 -1.4.8 ’ from ’https :// cache.n copying path ’/nix/store/q6j...-tix -8.4.3 ’ from ’https :// cache.nixos.org ’... copying path ’/nix/store/mj8...-python -2.7.14 ’ from ’https :// cache.nixos.org OUTPUT HERE 11
  14. Nix Shell: Recap • Claim: we can get consistent environment,

    assuming: • our core Nix expression pins version of nixpkgs • devs reload shell at the git revisions to pick up dependency changes • CI server runs tests using shell.nix referenced by nix-shell at that revision • infrastructure for all environments is built using pinned Nix expression • devenv, CI, production-like system builds share the same core expression 13
  15. What is NixOS? • "The Purely Functional Linux Distribution" (https://nixos.org/)

    • Provides congruent system-level configuration • System-level rollbacks 14
  16. NixOS: Example (nginx) # ./machines/nginx.nix { pkgs , ... }:

    { boot.kernelPackages = pkgs.linuxPackages_4_15; time.timeZone = "UTC"; services.nginx = { enable = true; virtualHosts."my.domain.tld" = { forceSSL = true; enableACME = true; locations."/assets".root = "/var/www/assets"; locations."/".proxyPass = "http://backend -lb:3000"; }; }; } 15
  17. NixOS: Example (Scala microservice) # ./machines/scala -services.nix { pkgs ,

    ... }: let inherit (builtins) listToAttrs map; inherit (pkgs.myorg) scalaServices mulib; in { # Interal custom myorg module; overlayed into pkgs myorg.defaults.enable = true; users.extraUsers = map mulib.mkSystemUser scalaServices; systemd.services = map mulib.mkServiceUnit scalaServices; } 16
  18. NixOS: Example (memcached) # ./machines/memcached.nix { config , pkgs ,

    ... }: { services.memcached.enable = true; services.memcached.maxConnections = 2048; services.memcached.maxMemory = 256; services.memcached.listen = config.networking.eth1.ipAddress; } 17
  19. NixOS: Containers (composition) { config , pkgs , ... }:

    { containers.cache.config = import ./memcached.nix; containers.proxy.config = import ./nginx.nix; containers.services = { config = import ./scala -services.nix; bindMounts."/webapp" = { hostPath = pkgs.myorg.scalaServices.webapp.outPath; isReadOnly = true; }; }; } 18
  20. NixOS: Multi-node testing (node setup) import ./make -test.nix ({ pkgs

    , lib , ... }: let kafkaPackage = pkgs.apacheKafka_1_0; in { name = "kafka -1.0"; nodes = { zookeeper = import ./machines/zookeeper.nix; kafka = import ./machines/kafka.nix; }; testScript = import ./tests/kafka.nix { package = kafkaPackage; util = pkgs.myorg.testing.util; }; }) 19
  21. NixOS: Multi-node testing (test script) { pkg , kPort ?

    9092 , zkPort ? 2181 , topic ? "test" , util }: let inherit (util) createTopicCmd producerPipe consumerPipe; in ’’ startAll; $zookeeper ->waitForUnit("zookeeper"); $zookeeper ->waitForOpenPort(${zkPort}); $kafka ->waitForUnit("apache -kafka"); $kafka ->waitForOpenPort(${kPort}); $kafka ->waitUntilSucceeds("${createTopicCmd}"); $kafka ->mustSucceed("echo ’test 1’ | ${producerPipe}"); $kafka ->mustSucceed("${consumerPipe} | grep ’test 1 ’"); ’’ 20
  22. NixOS & nixpkgs: Properties • modular (See nixpkgs overlays) •

    extensible (see nixos modules) • everything (almost) is a package in the Nix store • system rollbacks (with system profile snapshots saved as generations) • builds on purity of Nix packaging • NO FHS mutation-in-place (links into FHS to immutable Nix store) 21
  23. Reliable Software "Those who want really reliable software will discover

    that they must find means of avoiding the majority of bugs to start with, and as a result the programming process will become cheaper." – EWD 22
  24. Why do properties/laws matter? • Because it allows us humble

    programmers to: • compose our reasoning • know what we cannot reason about • break down big problems into smaller parts • accomplish the same result with less effort 23