Tweet
Tweet

Slide 1

Slide 1 text

Lambda the Ul,mate Devops Vlad Ki / @darkproger

Slide 2

Slide 2 text

• Pla%orm So+ware / SRE at Zalora, CS undergrad at NTUU KPI • We're hiring to our distributed team!

Slide 3

Slide 3 text

• config mgmt • running systems • monitoring • deployment / CI / CD • immutable infrastructure • supervising failures • ... • system integra

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

systems as code

Slide 7

Slide 7 text

Microgram The Zalora Pla+orm

Slide 8

Slide 8 text

Microgram • Applica(on realms (infra + configs) • Applica(on defini(ons (how to run / scale / monitor) • User APIs • Run(me converts defini(ons to real things (like infra) • Run(me handles opera(ons & automates labor • Fail

Slide 9

Slide 9 text

DSLs everywhere

Slide 10

Slide 10 text

/etc/passwd nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false root:*:0:0:System Administrator:/var/root:/bin/sh daemon:*:1:1:System Services:/var/root:/usr/bin/false _uucp:*:4:4:Unix to Unix Copy Protocol:/var/spool/uucp:/usr/sbin/uucico _taskgated:*:13:13:Task Gate Daemon:/var/empty:/usr/bin/false _networkd:*:24:24:Network Services:/var/networkd:/usr/bin/false _installassistant:*:25:25:Install Assistant:/var/empty:/usr/bin/false _lp:*:26:26:Printing Services:/var/spool/cups:/usr/bin/false _postfix:*:27:27:Postfix Mail Server:/var/spool/postfix:/usr/bin/false _scsd:*:31:31:Service Configuration Service:/var/empty:/usr/bin/false

Slide 11

Slide 11 text

/etc/pf.conf block quick from pass in on $ext_if proto tcp to $webserver port www \ (max-src-conn-rate 100/10, \ overload flush global) pass out on em0 inet proto tcp \ from $developerhosts to any port 80 \ set queue developers pass out on em0 inet proto tcp \ from any to any port 25 \ set queue mail

Slide 12

Slide 12 text

sendmail.mc define(`confTO_CONNECT', `1m')dnl define(`confTO_IDENT', `0')dnl define(`confTO_COMMAND', `2m')dnl LOCAL_NET_CONFIG # This rule ensures that all local mail is delivered using the # smtp transport, everything else will go via the smart host. R$* < @ $* .$m. > $* $#smtp $@ $2.$m. $: $1 < @ $2.$m. > $3

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

JSON { "domain": "www.example.com", "mongodb": { "host": "localhost", "port": 27017 } }

Slide 15

Slide 15 text

YAML / Ansible tasks: - name: take out of load balancer pool command: /usr/bin/take_out_of_pool {{ inventory_hostname }} delegate_to: 127.0.0.1

Slide 16

Slide 16 text

"Programming" with YAML - command: /opt/application/upgrade_db.py when: inventory_hostname == webservers[0]

Slide 17

Slide 17 text

"Programming" with JSON (CloudForma8on) "Outputs" : { "MyOutput" : { "Value" : { "Fn::Join" : [ "%", [ "A-string", {"Ref" : "AWS::StackName" } ] ] } } }

Slide 18

Slide 18 text

Puppet: resource-oriented DSL, isolated effects Although Puppet’s language is built around describing resources (and the rela6onships between them) in a declara6ve way, several parts of the language do depend on evalua6on order case $operatingsystem { centos, redhat: { $service_name = 'ntpd' } debian, ubuntu: { $service_name = 'ntp' } } package { $service_name: ensure => installed, }

Slide 19

Slide 19 text

Puppet + Hiera Apparently trea+ng code and data separately is cool again! --- mysql::server::root_password: 'strongpassword' databases: gotcms: user: 'got' password: 'super_secret_db_password' host: 'localhost' grant: 'ALL'

Slide 20

Slide 20 text

Chef: DSL + side effects ruby_block 'sleep30s' do block do sleep 30 end action :nothing end

Slide 21

Slide 21 text

7 -1 -1 -1 false true false false false

Slide 22

Slide 22 text

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

systems as code systems are code

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

Case study: MariaDB mul+-source replica+on

Slide 27

Slide 27 text

% cat databass.yaml sg: db-alpha.infra.zalora.io my: db-psi.infra.zalora.io id: db-psi.infra.zalora.io hk: db-psi.infra.zalora.io th: db-theta.infra.zalora.io ph: db-beta.infra.zalora.io vn: db-beta.infra.zalora.io

Slide 28

Slide 28 text

• MariaDB channels are server-centric • our model is country-centric • countries can share servers go#a aggregate!

Slide 29

Slide 29 text

Any insufficiently expressive DSL leads to the necessity of using an ul:mate systems integra:on tool.

Slide 30

Slide 30 text

(drumroll)

Slide 31

Slide 31 text

Which is bash. With some awk, sed and perl

Slide 32

Slide 32 text

% cat databass.yaml | \ awk -F': ' '{print $2, $1}' | \ awk '{a[$1] = a[$1] (a[$1] ? "," : "") $2; } END { for(k in a) print k, a[k]; }' db-beta.infra.zalora.io ph,vn db-psi.infra.zalora.io my,id,hk db-alpha.infra.zalora.io sg db-theta.infra.zalora.io th

Slide 33

Slide 33 text

Bash • somewhat ubiquitous and mostly unportable • hard to scale • strings • more strings • unexpected EOF while looking for matching `"' • also strings

Slide 34

Slide 34 text

Other ways to tackle refactoring • rewrite manually? • use a random script to do it during preprocessing? • write a puppet plugin to do that? • AbstractAggregationVirtualMethodFactoryFactory? • find a be8er template engine? • like, embed python into your tags? • why not use an expressive language in the first place?

Slide 35

Slide 35 text

Scaling Bash

Slide 36

Slide 36 text

Scaling Bash with Nix

Slide 37

Slide 37 text

db-slave-channels = let mapper = _: { db-name, masterhost, ... }: nameValuePair (to-key master-host) { inherit master-host; databases = [ db-name ]; }; reducer = { name, value }: all: all // { ${name} = value // { databases = all.${name}.databases or [] ++ value.databases; }; }; in fold reducer {} (mapAttrsToList mapper conf);

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

bash = "${pkgs.bash}/bin/bash"; base64 = "${pkgs.coreutils}/bin/base64"; jq = "${pkgs.jq}/bin/jq"; curl = "${pkgs.curl}/bin/curl -s --retry --fail"; awk = "${pkgs.gawk}/bin/awk"; openssl = "${pkgs.openssl}/bin/openssl";

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

Nix • func&onal language • dynamically typed • lightweight "schema valida&on" (hard to say typing) • one side-effect (derivation, used to build a package manager framework)

Slide 42

Slide 42 text

Case study: configuring Jenkins • click through all the forms? • can we build a DSL?

Slide 43

Slide 43 text

eris-sdk = erisJob { branch = "master"; shell = '' export SLACK_CHANNEL='#eris-facepalm' export SLACK_TIMEOUT=5m bin/slack make sdk ''; ssh-keys = [ credentials.hydrabot ]; triggers = [ (ghprb-trigger "eris-sdk") (github-push-trigger) ]; };

Slide 44

Slide 44 text

mapAttrs' (realm-name: spec: deployJob "deploy-${realm-name}" { inherit realm-name; scm = eris_master; permissions = with spec; { build = humans-can-build ++ others-can-build; }; ssh-keys = attrValues credentials; })) (filterAttrs is-deployable realms);

Slide 45

Slide 45 text

choice-parameter = { name , description ? "" , choices ? [ "this is a list" ] }: (term "hudson.model.ChoiceParameterDefinition" null [ (term "name" null name) (term "description" null description) (term "choices" { class = "java.util.Arrays$ArrayList"; } [ (term "a" { class = "string-array"; } (map (term "string" null) choices)) ]) ]);

Slide 46

Slide 46 text

NixOS • Nix packages + Linux kernel + systemd • Immutable system images • Full-stack

Slide 47

Slide 47 text

Infrastructure Specs

Slide 48

Slide 48 text

infra = { ec2-instance = { web1 = m3large; web2 = m3large; db-master = { infra, ... }: r3xlarge' { blockDeviceMapping."/dev/xvdm".disk = infra.ebs.database; }; cron = instance "m3.large"; }; elb = elb.defaults; ebs.database = { inherit (realm.ec2-args) region zone; size = 200; volumeType.gp2 = true; }; };

Slide 49

Slide 49 text

Pluggable sta+c verifica+on

Slide 50

Slide 50 text

writeBashScript = name: script: let prelude = '' #!${pkgs.bash}/bin/bash set -e -o pipefail ''; in pkgs.runCommand name { inherit prelude script; } '' echo "$prelude" >> "$out" echo "$script" >> "$out" chmod +x "$out" ${ShellCheck}/bin/shellcheck \ "$out" '';

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

Experiences with Nix

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

• zalora/upcast - declara1ve infra provisioning (like nixops/ terraform/fugue) • zalora/replicator - automated MySQL replica1on • zalora/sproxy - proxy that handles OAuth2 + ACL interface • zalora/aws-ec2 - EC2 extensions for aris1db/aws • unicron, a single-user cron • a lot more on Zalora's GitHub

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

Correct by construc-on • harden your interoperability with type-safe APIs data Expr :: * -> * where E :: Executable -> e -> Expr e Pipe :: Expr e -> Expr e -> Expr e Seq :: Expr e -> Expr e -> Expr e Or :: Expr e -> Expr e -> Expr e Redir :: Expr e -> FilePath -> Expr e Env :: [Pair] -> Expr e -> Expr e Sudo :: Expr e -> Expr e SSH :: Hostname -> e -> Expr e -> Expr e

Slide 57

Slide 57 text

Correct by construc-on: web APIs data DeploymentParams = -- ... data DeploymentStatus = -- ... type Deployments = "deployments.xml" :> Header "x-api-key" ApiKey :> ReqBody '[FormUrlEncoded] DeploymentParams :> Post '[XML] DeploymentStatus

Slide 58

Slide 58 text

Interpre'ng effects wtf :: (Member Spec r, Member Upcast r) => Eff r (Map Machine ExitCode) wtf = infra >>= machines >>= traverse (`ssh` cmd) where cmd :: Commandline cmd = exec "wtf" []

Slide 59

Slide 59 text

-- | Idempotent effects based on evaluating the spec (ok to cache). data Spec v where NixQuery :: FromJSON json => Query json -> Spec json Infra :: Spec Infras Stub :: Show a => a -> Spec () -- | Effects that require networking. data Upcast v where Machines :: Infras -> Upcast (Attrs Machine) NixBuild :: Query a -> Upcast [StorePath] NixInstantiate :: Query a -> Upcast [StorePath] MachineExec :: (Machine, Commandline) -> Upcast ExitCode SystemInstalls :: (Machine, StorePath) -> Upcast ExitCode NRNotification :: ApiKey -> [DNS] -> Upcast ()

Slide 60

Slide 60 text

Integra(ng with the rest of the stringy world • UNIX is a minefield for experimen5ng with parser combinators! • a lot of perf analysis or systems explora5on is done by analysing streams of text • use haskell if lost in awk+perl+sed • see proger/lxkit and zalora/gctuner

Slide 61

Slide 61 text

Notable tools • propellor • shake/bake • language-puppet • literate haskell (idea from org-mode with emacs)

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

Why not just take a random PL again? • DSLs are about isola0ng effects • (like calling system() during a pure graph traversal) • hard to work in a language where you can't protect from shoo0ng self in the foot

Slide 64

Slide 64 text

Don't sacrifice expressivity for cheap wins. You'll pay the cost later.

Slide 65

Slide 65 text

There are infinitely many tools to do cool things.

Slide 66

Slide 66 text

There are not enough languages that are powerful enough to map those domains.

Slide 67

Slide 67 text

Keep the unicorn happy!