Lambda the Ultimate Devops v2

Slide 1

Slide 1 text

Lambda the Ul,mate Devops Vlad Ki / @darkproger

Slide 2

Slide 2 text

• Pla%orm So+ware / SRE at Zalora, CS undergrad at NTUU KPI

Slide 3

Slide 3 text

The Goal

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

Demonstrate: • core func*onal programming values applied to pla5orm engineering: • transforma*on-driven programing • ADTs • types 1. increase development agility and ﬂexibility 2. prevent from focusing on boring/obvious bugs

Slide 6

Slide 6 text

Microgram The Zalora Pla+orm

Slide 7

Slide 7 text

Microgram • everything is an API • framework for composing (and running) alien so=ware ┌───────────┐ ┌───────────┬─────────┐ ┌───> app │ │ │ kernel │ ┌───────┐ │ └───────────┘ │ ├─────────┤ │infra? ├────────┐ │ ┌───────────┐ │ sdk │ openssl │ └────┬──┘realm ├──┴┬──> app │ │(nix-based)├─────────┤ └───────────┘ │ └───────────┘ │ │ nginx │ │ ┌───────────┐ │ ├─────────┤ └──> app │ │ │ myprog │ └───────────┘ └───────────┴─────────┘

Slide 8

Slide 8 text

DSLs everywhere

Slide 9

Slide 9 text

/etc/passwd nobody:*:-2:-2:Unprivileged User:/var/empty:/bin/false root:*:0:0:System Administrator:/var/root:/bin/sh daemon:*:1:1:System Services:/var/root:/bin/false _taskgated:*:13:13:Task Gate Daemon:/var/empty:/bin/false _networkd:*:24:24:Network Services:/var/networkd:/bin/false _lp:*:26:26:Printing Services:/var/spool/cups:/bin/false • nothing special, just TSV data

Slide 10

Slide 10 text

/etc/pf.conf block quick from pass in on $ext_if proto tcp to $webserver port www \ (max-src-conn-rate 100/10, \ overload flush global) pass out on em0 inet proto tcp \ from $developerhosts to any port 80 \ set queue developers pass out on em0 inet proto tcp \ from any to any port 25 • macros, quick implies ﬂow control

Slide 11

Slide 11 text

sendmail.mc define(`confTO_CONNECT', `1m')dnl define(`confTO_IDENT', `0')dnl define(`confTO_COMMAND', `2m')dnl LOCAL_NET_CONFIG # This rule ensures that all local mail is delivered using the # smtp transport, everything else will go via the smart host. R$* < @ $* .$m. > $* $#smtp $@ $2.$m. $: $1 < @ $2.$m. > $3 • no comments

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

JSON { "domain": "www.example.com", "mongodb": { "host": "localhost", "port": 27017 } }

Slide 14

Slide 14 text

"Programming" with JSON (CloudForma8on) { "Type": "AWS::ElasticLoadBalancing::LoadBalancer", "Properties": { "Listeners": [{ "SSLCertificateId": {"Fn::FindInMap": ["SSLMAP", "au", {"Ref": "EnvironmentType"}]}, }], "LoadBalancerName": {"Fn::Join": ["", ["app-web-", {"Ref": "EnvironmentName"}, ""]]} } } • people are actually expected to write that • lisp users can weep now

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

YAML / Ansible tasks: - name: take out of load balancer pool command: /usr/bin/take_out_of_pool {{ inventory_hostname }} delegate_to: 127.0.0.1 • string interpola.on, apparently

Slide 17

Slide 17 text

"Programming" with YAML - command: /opt/application/upgrade_db.py when: inventory_hostname == webservers[0] • implicit ﬂow control? python syntax?

Slide 18

Slide 18 text

Puppet: resource-oriented DSL, isolated eﬀects Although Puppet’s language is built around describing resources (and the rela6onships between them) in a declara6ve way, several parts of the language do depend on evalua6on order case $operatingsystem { centos, redhat: { $service_name = 'ntpd' } debian, ubuntu: { $service_name = 'ntp' } } package { $service_name: ensure => installed, }

Slide 19

Slide 19 text

Puppet + Hiera Apparently trea+ng code and data separately is cool again! --- mysql::server::root_password: 'strongpassword' databases: gotcms: user: 'got' password: 'super_secret_db_password' host: 'localhost' grant: 'ALL'

Slide 20

Slide 20 text

Chef: DSL + side eﬀects ruby_block 'sleep30s' do block do sleep 30 end action :nothing end

Slide 21

Slide 21 text

Building good Dockerﬁles is mostly folklore FROM debian:jessie RUN apt-get update && \ apt-get install -y --no-install-recommends openjdk-7-jre-headless ADD http://someplace/zookeeper-3.4.6.tar.gz /opt/ RUN cd /opt && \ tar -zxvf zookeeper-3.4.6.tar.gz && \ mv zookeeper-3.4.6 zookeeper && \ rm -rf ./zookeeper-*tar.gz && \ mkdir -p /var/lib/zookeeper ADD entry.sh /entry.sh WORKDIR /opt/zookeeper EXPOSE 2181 2888 3888 VOLUME ["/var/lib/zookeeper", "/opt/zookeeper/conf", "/tmp/zookeeper"] ENTRYPOINT ["/entry.sh"]

Slide 22

Slide 22 text

7 -1 -1 -1 false true false false false

Slide 23

Slide 23 text

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

DSLs are like drugs

Slide 26

Slide 26 text

Case study: MariaDB mul+-source replica+on

Slide 27

Slide 27 text

% cat databass.yaml sg: db-alpha.infra.zalora.io my: db-psi.infra.zalora.io id: db-psi.infra.zalora.io hk: db-psi.infra.zalora.io th: db-theta.infra.zalora.io ph: db-beta.infra.zalora.io vn: db-beta.infra.zalora.io

Slide 28

Slide 28 text

• MariaDB channels are server-centric • our model is country-centric • countries can share servers have to aggregate!

Slide 29

Slide 29 text

Any insuﬃciently expressive DSL leads to the necessity of using an ul:mate systems integra:on tool.

Slide 30

Slide 30 text

(drumroll)

Slide 31

Slide 31 text

Which is bash. With some awk, sed and perl

Slide 32

Slide 32 text

% cat databass.yaml | \ awk -F': ' '{print $2, $1}' | \ awk '{a[$1] = a[$1] (a[$1] ? "," : "") $2; } END { for(k in a) print k, a[k]; }' db-beta.infra.zalora.io ph,vn db-psi.infra.zalora.io my,id,hk db-alpha.infra.zalora.io sg db-theta.infra.zalora.io th

Slide 33

Slide 33 text

Bash • a pi&all trap that looks easy • somewhat ubiquitous and mostly unportable • hard to scale • strings • more strings • unexpected EOF while looking for matching `"' • also strings

Slide 34

Slide 34 text

Devops Success Stories 101 • when in doubt, "do it manually just this once" • when repeated 5 9mes, wrap with a string interpola9on library • if not enough, try to add ﬂow control as a "declara9ve" construct • when failed, embed a random PL with unclear boundaries • of course, error handling is ad-hoc trial-and-error • weep

Slide 35

Slide 35 text

why not use an expressive language in the ﬁrst place?

Slide 36

Slide 36 text

Scaling Bash

Slide 37

Slide 37 text

Scaling Bash with Nix

Slide 38

Slide 38 text

db-slave-channels = let mapper = _: { db-name, masterhost, ... }: nameValuePair (to-key master-host) { inherit master-host; databases = [ db-name ]; }; reducer = { name, value }: all: all // { ${name} = value // { databases = all.${name}.databases or [] ++ value.databases; }; }; in fold reducer {} (mapAttrsToList mapper conf);

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

bash = "${pkgs.bash}/bin/bash"; base64 = "${pkgs.coreutils}/bin/base64"; jq = "${pkgs.jq}/bin/jq"; curl = "${pkgs.curl}/bin/curl -s --retry --fail"; awk = "${pkgs.gawk}/bin/awk"; openssl = "${pkgs.openssl}/bin/openssl"; • automa'c dependency tracking!

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

Nix • func&onal language • dynamically typed • lightweight "schema valida&on" (hard to say typing) • one side-eﬀect (derivation, used to build a package manager framework)

Slide 43

Slide 43 text

Case study: conﬁguring Jenkins • string templa.ng again? • click through all the forms? • can we build a DSL?

Slide 44

Slide 44 text

eris-sdk = erisJob { branch = "master"; shell = '' export SLACK_CHANNEL='#eris-facepalm' export SLACK_TIMEOUT=5m bin/slack make sdk ''; ssh-keys = [ credentials.hydrabot ]; triggers = [ (ghprb-trigger "eris-sdk") (github-push-trigger) ]; };

Slide 45

Slide 45 text

mapAttrs' (realm-name: spec: deployJob "deploy-${realm-name}" { inherit realm-name; scm = eris_master; permissions = with spec; { build = humans-can-build ++ others-can-build; }; ssh-keys = attrValues credentials; })) (filterAttrs is-deployable realms);

Slide 46

Slide 46 text

choice-parameter = { name , description ? "" , choices ? [ "this is a list" ] }: (term "hudson.model.ChoiceParameterDefinition" null [ (term "name" null name) (term "description" null description) (term "choices" { class = "java.util.Arrays$ArrayList"; } [ (term "a" { class = "string-array"; } (map (term "string" null) choices)) ]) ]);

Slide 47

Slide 47 text

Run$mes • implementa*on of Apps APIs to linux services • same APIs - mul*ple backends • allows rapid experimenta*on: mesos/docker/diego/ ares

Slide 48

Slide 48 text

apps runtimes targets ┌─────────┐ ┌──────────┐┌%%%%%%%%%%%%%%%%%%%┐ ┌───> ASG │ ┌───> mysql ││ nixos/systemd │──┼┐ └─────────┘ │ └──────────┘└───────────────────┘ ││ ┌───────────┐ │ ┌──────────┐┌###################┐ ││ ┌─────────┐ │ realm ├──┴┬──>strongswan││ adhoc │──┼┼───>baremetal│ └───────────┘ │ └──────────┘└───────────────────┘ ││ └─────────┘ │ ┌──────────┐┌&&&&&&&&&&&&&&&&&&&┐ ││ └──> someweb ││ ami ├──┘│ ┌─────────┐ └──────────┘└───────────────────┘ └──> ec2 │ └─────────┘

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

Case study: containers • adhoc run+me deployed to a ﬂeet of old RedHat 6 servers • adhoc apps were exposed to global state • adop+ng runc • +me to PoC - 3 hours • +me to produc+on - 12 hours (8 wasted tracing kernel code) • running on 2.6.32.602 with minor libcontainer patches

Slide 51

Slide 51 text

Infrastructure Specs

Slide 52

Slide 52 text

infra = { ec2-instance = { web1 = m3large; web2 = m3large; db-master = { infra, ... }: r3xlarge' { blockDeviceMapping."/dev/xvdm".disk = infra.ebs.database; }; cron = instance "m3.large"; }; elb = elb.defaults; ebs.database = { inherit (realm.ec2-args) region zone; size = 200; volumeType.gp2 = true; }; };

Slide 53

Slide 53 text

Infrastructure Specs • nixops-inspired -> reimplemented in upcast • terraform implementa5on as an evening hack • autoscaling PoC was wri=en bash :O

Slide 54

Slide 54 text

Pluggable sta+c veriﬁca+on

Slide 55

Slide 55 text

writeBashScript = name: script: let prelude = '' #!${pkgs.bash}/bin/bash set -e -o pipefail ''; in pkgs.runCommand name { inherit prelude script; } '' echo "$prelude" >> "$out" echo "$script" >> "$out" chmod +x "$out" ${ShellCheck}/bin/shellcheck \ "$out" '';

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

Nix: try explaining a user what not to do

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

• zalora/upcast - declara1ve infra provisioning (like nixops/terraform/fugue) • zalora/replicator - automated MySQL replica1on • zalora/sproxy - proxy that handles OAuth2 + ACL interface • zalora/aws-ec2 - EC2 extensions for aris1db/aws • unicron, a single-user cron • a lot more on Zalora's GitHub

Slide 60

Slide 60 text

Correct by construc-on: safe Bash • harden your interoperability with type-safe APIs data Expr :: * -> * where E :: Executable -> e -> Expr e Pipe :: Expr e -> Expr e -> Expr e Seq :: Expr e -> Expr e -> Expr e Or :: Expr e -> Expr e -> Expr e Redir :: Expr e -> FilePath -> Expr e Env :: [Pair] -> Expr e -> Expr e Sudo :: Expr e -> Expr e SSH :: Hostname -> e -> Expr e -> Expr e

Slide 61

Slide 61 text

Correct by construc-on: safe Nix data Nix :: * -> * where Null :: Nix () Lit :: Literal -> Nix a -- ... Var :: Index -> Nix a Lam :: Index -> Nix a -> Nix (Nix input -> Nix a) Ap :: Nix (Nix input -> Nix a) -> Nix input -> Nix a -- ... List1 :: Nix a -> Nix [Nix a]

Slide 62

Slide 62 text

Typing the standard library listToAttrs :: Nix [Nix (Nix String, Nix a)] -> Nix (Attrset a) listToAttrs = Ap (Lit "builtins.listToAttrs") concatLists :: Nix [Nix [a]] -> Nix [a] concatLists = Ap (Lit "builtins.concatLists") map' :: Nix (a -> b) -> Nix [a] -> Nix [b] map' f xs = Ap (Ap (Lit "map") f) xs attrNames :: Nix (Attrset a) -> Nix [Nix String] attrNames = Ap (Lit "builtins.attrNames")

Slide 63

Slide 63 text

Induc&on Composi&on! λ> let concatMapToAttrs = (\f -> listToAttrs . concatLists . map' f) λ> :t concatMapToAttrs concatMapToAttrs :: Nix (a1 -> Nix [Nix (Nix String, Nix a)]) -> Nix [a1] -> Nix (Attrset a)

Slide 64

Slide 64 text

λ> :t null null :: Nix () λ> :t concatMapToAttrs null :1:18: Couldn't match type ‘()’ with ‘a1 -> Nix [Nix (Nix String, Nix a)]’ Expected type: Nix (a1 -> Nix [Nix (Nix String, Nix a)]) Actual type: Nix ()

Slide 65

Slide 65 text

Correct by construc-on: web APIs data DeploymentParams = -- ... data DeploymentStatus = -- ... type Deployments = "deployments.xml" :> Header "x-api-key" ApiKey :> ReqBody '[FormUrlEncoded] DeploymentParams :> Post '[XML] DeploymentStatus h"ps:/ /haskell-servant.github.io Type-safe generic programming!

Slide 66

Slide 66 text

Interpre'ng eﬀects wtf :: (Member Spec r, Member Upcast r) => Eff r (Map Machine ExitCode) wtf = infra >>= machines >>= traverse (`ssh` cmd) where cmd :: Commandline cmd = exec "wtf" [] • notable implementa-on: Eﬀ-lang

Slide 67

Slide 67 text

-- | Idempotent effects based on evaluating the spec (ok to cache). data Spec v where NixQuery :: FromJSON json => Query json -> Spec json Infra :: Spec Infras Stub :: Show a => a -> Spec () -- | Effects that require networking. data Upcast v where Machines :: Infras -> Upcast (Attrs Machine) NixBuild :: Query a -> Upcast [StorePath] NixInstantiate :: Query a -> Upcast [StorePath] MachineExec :: (Machine, Commandline) -> Upcast ExitCode SystemInstalls :: (Machine, StorePath) -> Upcast ExitCode NRNotification :: ApiKey -> [DNS] -> Upcast ()

Slide 68

Slide 68 text

Integra(ng with the rest of the stringy world • UNIX is a mineﬁeld for experimen5ng with parser combinators! • a lot of perf analysis or systems explora5on is done by analysing streams of text • use haskell if lost in awk+perl+sed • see proger/lxkit and zalora/gctuner

Slide 69

Slide 69 text

finally { • infrastructure tools come and go • they're ﬁlled with adhoc interfaces with poor composablity and/or expressiveness • "tradi=onal" composi=on prac=ces are too error- prone and divert your precious a?en=on to ﬁxing irrelevant bugs • basic PLT tools like transforma=onal programming and type systems keep you sane