Lambda the Ultimate Devops v2

Lambda the Ultimate Devops from Build Stuff Ukraine 2015

Transcript

1. Lambda the Ul,mate Devops Vlad Ki / @darkproger
2. • Pla%orm So+ware / SRE at Zalora, CS undergrad at NTUU KPI
3. The Goal
5. Demonstrate: • core func*onal programming values applied to pla5orm engineering: • transforma*on-driven programing • ADTs • types 1. increase development agility and flexibility 2. prevent from focusing on boring/obvious bugs
6. Microgram The Zalora Pla+orm
7. Microgram • everything is an API • framework for composing (and running) alien so=ware ┌───────────┐ ┌───────────┬─────────┐ ┌───> app │ │ │ kernel │ ┌───────┐ │ └───────────┘ │ ├─────────┤ │infra? ├────────┐ │ ┌───────────┐ │ sdk │ openssl │ └────┬──┘realm ├──┴┬──> app │ │(nix-based)├─────────┤ └───────────┘ │ └───────────┘ │ │ nginx │ │ ┌───────────┐ │ ├─────────┤ └──> app │ │ │ myprog │ └───────────┘ └───────────┴─────────┘
8. DSLs everywhere
9. /etc/passwd nobody:*:-2:-2:Unprivileged User:/var/empty:/bin/false root:*:0:0:System Administrator:/var/root:/bin/sh daemon:*:1:1:System Services:/var/root:/bin/false _taskgated:*:13:13:Task Gate Daemon:/var/empty:/bin/false _networkd:*:24:24:Network Services:/var/networkd:/bin/false _lp:*:26:26:Printing Services:/var/spool/cups:/bin/false • nothing special, just TSV data
10. /etc/pf.conf block quick from <bad_hosts> pass in on $ext_if proto tcp to $webserver port www \ (max-src-conn-rate 100/10, \ overload <bad_hosts> flush global) pass out on em0 inet proto tcp \ from $developerhosts to any port 80 \ set queue developers pass out on em0 inet proto tcp \ from any to any port 25 • macros, quick implies flow control
11. sendmail.mc define(`confTO_CONNECT', `1m')dnl define(`confTO_IDENT', `0')dnl define(`confTO_COMMAND', `2m')dnl LOCAL_NET_CONFIG # This rule ensures that all local mail is delivered using the # smtp transport, everything else will go via the smart host. R$* < @ $* .$m. > $* $#smtp $@ $2.$m. $: $1 < @ $2.$m. > $3 • no comments
13. JSON { "domain": "www.example.com", "mongodb": { "host": "localhost", "port": 27017 } }
14. "Programming" with JSON (CloudForma8on) { "Type": "AWS::ElasticLoadBalancing::LoadBalancer", "Properties": { "Listeners": [{ "SSLCertificateId": {"Fn::FindInMap": ["SSLMAP", "au", {"Ref": "EnvironmentType"}]}, }], "LoadBalancerName": {"Fn::Join": ["", ["app-web-", {"Ref": "EnvironmentName"}, ""]]} } } • people are actually expected to write that • lisp users can weep now
16. YAML / Ansible tasks: - name: take out of load balancer pool command: /usr/bin/take_out_of_pool {{ inventory_hostname }} delegate_to: 127.0.0.1 • string interpola.on, apparently
17. "Programming" with YAML - command: /opt/application/upgrade_db.py when: inventory_hostname == webservers[0] • implicit flow control? python syntax?
18. Puppet: resource-oriented DSL, isolated effects Although Puppet’s language is built around describing resources (and the rela6onships between them) in a declara6ve way, several parts of the language do depend on evalua6on order case $operatingsystem { centos, redhat: { $service_name = 'ntpd' } debian, ubuntu: { $service_name = 'ntp' } } package { $service_name: ensure => installed, }
19. Puppet + Hiera Apparently trea+ng code and data separately is cool again! --- mysql::server::root_password: 'strongpassword' databases: gotcms: user: 'got' password: 'super_secret_db_password' host: 'localhost' grant: 'ALL'
20. Chef: DSL + side effects ruby_block 'sleep30s' do block do sleep 30 end action :nothing end
21. Building good Dockerfiles is mostly folklore FROM debian:jessie RUN apt-get update && \ apt-get install -y --no-install-recommends openjdk-7-jre-headless ADD http://someplace/zookeeper-3.4.6.tar.gz /opt/ RUN cd /opt && \ tar -zxvf zookeeper-3.4.6.tar.gz && \ mv zookeeper-3.4.6 zookeeper && \ rm -rf ./zookeeper-*tar.gz && \ mkdir -p /var/lib/zookeeper ADD entry.sh /entry.sh WORKDIR /opt/zookeeper EXPOSE 2181 2888 3888 VOLUME ["/var/lib/zookeeper", "/opt/zookeeper/conf", "/tmp/zookeeper"] ENTRYPOINT ["/entry.sh"]
22. <?xml version='1.0' encoding='UTF-8'?> <project> <actions/> <description></description> <logRotator> <daysToKeep>7</daysToKeep> <numToKeep>-1</numToKeep> <artifactDaysToKeep>-1</artifactDaysToKeep> <artifactNumToKeep>-1</artifactNumToKeep> </logRotator> <keepDependencies>false</keepDependencies> <properties/> <scm class="hudson.scm.NullSCM"/> <canRoam>true</canRoam> <disabled>false</disabled> <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding> <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding> <triggers class="vector"/>
23. <tr> <xsl:attribute name="style"> <xsl:choose> <xsl:when test="CONDITION"> <xsl:value-of select="'visibility: visible'"> </xsl:when> <xsl:otherwise> <xsl:value-of select="'visibility: collapse'"> </xsl:otherwise> </xsl:choose> </xsl:attribute> </tr>
25. DSLs are like drugs
26. Case study: MariaDB mul+-source replica+on
27. % cat databass.yaml sg: db-alpha.infra.zalora.io my: db-psi.infra.zalora.io id: db-psi.infra.zalora.io hk: db-psi.infra.zalora.io th: db-theta.infra.zalora.io ph: db-beta.infra.zalora.io vn: db-beta.infra.zalora.io
28. • MariaDB channels are server-centric • our model is country-centric • countries can share servers have to aggregate!
29. Any insufficiently expressive DSL leads to the necessity of using an ul:mate systems integra:on tool.
30. (drumroll)
31. Which is bash. With some awk, sed and perl
32. % cat databass.yaml | \ awk -F': ' '{print $2, $1}' | \ awk '{a[$1] = a[$1] (a[$1] ? "," : "") $2; } END { for(k in a) print k, a[k]; }' db-beta.infra.zalora.io ph,vn db-psi.infra.zalora.io my,id,hk db-alpha.infra.zalora.io sg db-theta.infra.zalora.io th
33. Bash • a pi&all trap that looks easy • somewhat ubiquitous and mostly unportable • hard to scale • strings • more strings • unexpected EOF while looking for matching `"' • also strings
34. Devops Success Stories 101 • when in doubt, "do it manually just this once" • when repeated 5 9mes, wrap with a string interpola9on library • if not enough, try to add flow control as a "declara9ve" construct • when failed, embed a random PL with unclear boundaries • of course, error handling is ad-hoc trial-and-error • weep
35. why not use an expressive language in the first place?
36. Scaling Bash
37. Scaling Bash with Nix
38. db-slave-channels = let mapper = _: { db-name, masterhost, ... }: nameValuePair (to-key master-host) { inherit master-host; databases = [ db-name ]; }; reducer = { name, value }: all: all // { ${name} = value // { databases = all.${name}.databases or [] ++ value.databases; }; }; in fold reducer {} (mapAttrsToList mapper conf);
40. bash = "${pkgs.bash}/bin/bash"; base64 = "${pkgs.coreutils}/bin/base64"; jq = "${pkgs.jq}/bin/jq"; curl = "${pkgs.curl}/bin/curl -s --retry --fail"; awk = "${pkgs.gawk}/bin/awk"; openssl = "${pkgs.openssl}/bin/openssl"; • automa'c dependency tracking!
42. Nix • func&onal language • dynamically typed • lightweight "schema valida&on" (hard to say typing) • one side-effect (derivation, used to build a package manager framework)
43. Case study: configuring Jenkins • string templa.ng again? • click through all the forms? • can we build a DSL?
44. eris-sdk = erisJob { branch = "master"; shell = '' export SLACK_CHANNEL='#eris-facepalm' export SLACK_TIMEOUT=5m bin/slack make sdk ''; ssh-keys = [ credentials.hydrabot ]; triggers = [ (ghprb-trigger "eris-sdk") (github-push-trigger) ]; };
45. mapAttrs' (realm-name: spec: deployJob "deploy-${realm-name}" { inherit realm-name; scm = eris_master; permissions = with spec; { build = humans-can-build ++ others-can-build; }; ssh-keys = attrValues credentials; })) (filterAttrs is-deployable realms);
46. choice-parameter = { name , description ? "" , choices ? [ "this is a list" ] }: (term "hudson.model.ChoiceParameterDefinition" null [ (term "name" null name) (term "description" null description) (term "choices" { class = "java.util.Arrays$ArrayList"; } [ (term "a" { class = "string-array"; } (map (term "string" null) choices)) ]) ]);
47. Run$mes • implementa*on of Apps APIs to linux services • same APIs - mul*ple backends • allows rapid experimenta*on: mesos/docker/diego/ ares
48. apps runtimes targets ┌─────────┐ ┌──────────┐┌%%%%%%%%%%%%%%%%%%%┐ ┌───> ASG │ ┌───> mysql ││ nixos/systemd │──┼┐ └─────────┘ │ └──────────┘└───────────────────┘ ││ ┌───────────┐ │ ┌──────────┐┌###################┐ ││ ┌─────────┐ │ realm ├──┴┬──>strongswan││ adhoc │──┼┼───>baremetal│ └───────────┘ │ └──────────┘└───────────────────┘ ││ └─────────┘ │ ┌──────────┐┌&&&&&&&&&&&&&&&&&&&┐ ││ └──> someweb ││ ami ├──┘│ ┌─────────┐ └──────────┘└───────────────────┘ └──> ec2 │ └─────────┘
50. Case study: containers • adhoc run+me deployed to a fleet of old RedHat 6 servers • adhoc apps were exposed to global state • adop+ng runc • +me to PoC - 3 hours • +me to produc+on - 12 hours (8 wasted tracing kernel code) • running on 2.6.32.602 with minor libcontainer patches
51. Infrastructure Specs
52. infra = { ec2-instance = { web1 = m3large; web2 = m3large; db-master = { infra, ... }: r3xlarge' { blockDeviceMapping."/dev/xvdm".disk = infra.ebs.database; }; cron = instance "m3.large"; }; elb = elb.defaults; ebs.database = { inherit (realm.ec2-args) region zone; size = 200; volumeType.gp2 = true; }; };
53. Infrastructure Specs • nixops-inspired -> reimplemented in upcast • terraform implementa5on as an evening hack • autoscaling PoC was wri=en bash :O
54. Pluggable sta+c verifica+on
55. writeBashScript = name: script: let prelude = '' #!${pkgs.bash}/bin/bash set -e -o pipefail ''; in pkgs.runCommand name { inherit prelude script; } '' echo "$prelude" >> "$out" echo "$script" >> "$out" chmod +x "$out" ${ShellCheck}/bin/shellcheck \ "$out" '';
57. Nix: try explaining a user what not to do
59. • zalora/upcast - declara1ve infra provisioning (like nixops/terraform/fugue) • zalora/replicator - automated MySQL replica1on • zalora/sproxy - proxy that handles OAuth2 + ACL interface • zalora/aws-ec2 - EC2 extensions for aris1db/aws • unicron, a single-user cron • a lot more on Zalora's GitHub
60. Correct by construc-on: safe Bash • harden your interoperability with type-safe APIs data Expr :: * -> * where E :: Executable -> e -> Expr e Pipe :: Expr e -> Expr e -> Expr e Seq :: Expr e -> Expr e -> Expr e Or :: Expr e -> Expr e -> Expr e Redir :: Expr e -> FilePath -> Expr e Env :: [Pair] -> Expr e -> Expr e Sudo :: Expr e -> Expr e SSH :: Hostname -> e -> Expr e -> Expr e
61. Correct by construc-on: safe Nix data Nix :: * -> * where Null :: Nix () Lit :: Literal -> Nix a -- ... Var :: Index -> Nix a Lam :: Index -> Nix a -> Nix (Nix input -> Nix a) Ap :: Nix (Nix input -> Nix a) -> Nix input -> Nix a -- ... List1 :: Nix a -> Nix [Nix a]
62. Typing the standard library listToAttrs :: Nix [Nix (Nix String, Nix a)] -> Nix (Attrset a) listToAttrs = Ap (Lit "builtins.listToAttrs") concatLists :: Nix [Nix [a]] -> Nix [a] concatLists = Ap (Lit "builtins.concatLists") map' :: Nix (a -> b) -> Nix [a] -> Nix [b] map' f xs = Ap (Ap (Lit "map") f) xs attrNames :: Nix (Attrset a) -> Nix [Nix String] attrNames = Ap (Lit "builtins.attrNames")
63. Induc&on Composi&on! λ> let concatMapToAttrs = (\f -> listToAttrs . concatLists . map' f) λ> :t concatMapToAttrs concatMapToAttrs :: Nix (a1 -> Nix [Nix (Nix String, Nix a)]) -> Nix [a1] -> Nix (Attrset a)
64. λ> :t null null :: Nix () λ> :t concatMapToAttrs null <interactive>:1:18: Couldn't match type ‘()’ with ‘a1 -> Nix [Nix (Nix String, Nix a)]’ Expected type: Nix (a1 -> Nix [Nix (Nix String, Nix a)]) Actual type: Nix ()
65. Correct by construc-on: web APIs data DeploymentParams = -- ... data DeploymentStatus = -- ... type Deployments = "deployments.xml" :> Header "x-api-key" ApiKey :> ReqBody '[FormUrlEncoded] DeploymentParams :> Post '[XML] DeploymentStatus h"ps:/ /haskell-servant.github.io Type-safe generic programming!
66. Interpre'ng effects wtf :: (Member Spec r, Member Upcast r) => Eff r (Map Machine ExitCode) wtf = infra >>= machines >>= traverse (`ssh` cmd) where cmd :: Commandline cmd = exec "wtf" [] • notable implementa-on: Eff-lang
67. -- | Idempotent effects based on evaluating the spec (ok to cache). data Spec v where NixQuery :: FromJSON json => Query json -> Spec json Infra :: Spec Infras Stub :: Show a => a -> Spec () -- | Effects that require networking. data Upcast v where Machines :: Infras -> Upcast (Attrs Machine) NixBuild :: Query a -> Upcast [StorePath] NixInstantiate :: Query a -> Upcast [StorePath] MachineExec :: (Machine, Commandline) -> Upcast ExitCode SystemInstalls :: (Machine, StorePath) -> Upcast ExitCode NRNotification :: ApiKey -> [DNS] -> Upcast ()
68. Integra(ng with the rest of the stringy world • UNIX is a minefield for experimen5ng with parser combinators! • a lot of perf analysis or systems explora5on is done by analysing streams of text • use haskell if lost in awk+perl+sed • see proger/lxkit and zalora/gctuner
69. finally { • infrastructure tools come and go • they're filled with adhoc interfaces with poor composablity and/or expressiveness • "tradi=onal" composi=on prac=ces are too error- prone and divert your precious a?en=on to fixing irrelevant bugs • basic PLT tools like transforma=onal programming and type systems keep you sane