Share
Embed
Lambda the Ultimate Devops v2
Lambda the Ultimate Devops from Build Stuff Ukraine 2015
Volodymyr Kyrylov
November 23, 2015
Transcript
71 slides
-
Demonstrate: • core func*onal programming values applied to pla5orm engineering:
• transforma*on-driven programing • ADTs • types 1. increase development agility and flexibility 2. prevent from focusing on boring/obvious bugs -
Microgram • everything is an API • framework for composing
(and running) alien so=ware ┌───────────┐ ┌───────────┬─────────┐ ┌───> app │ │ │ kernel │ ┌───────┐ │ └───────────┘ │ ├─────────┤ │infra? ├────────┐ │ ┌───────────┐ │ sdk │ openssl │ └────┬──┘realm ├──┴┬──> app │ │(nix-based)├─────────┤ └───────────┘ │ └───────────┘ │ │ nginx │ │ ┌───────────┐ │ ├─────────┤ └──> app │ │ │ myprog │ └───────────┘ └───────────┴─────────┘ -
/etc/passwd nobody:*:-2:-2:Unprivileged User:/var/empty:/bin/false root:*:0:0:System Administrator:/var/root:/bin/sh daemon:*:1:1:System Services:/var/root:/bin/false _taskgated:*:13:13:Task Gate Daemon:/var/empty:/bin/false
_networkd:*:24:24:Network Services:/var/networkd:/bin/false _lp:*:26:26:Printing Services:/var/spool/cups:/bin/false • nothing special, just TSV data -
/etc/pf.conf block quick from <bad_hosts> pass in on $ext_if proto
tcp to $webserver port www \ (max-src-conn-rate 100/10, \ overload <bad_hosts> flush global) pass out on em0 inet proto tcp \ from $developerhosts to any port 80 \ set queue developers pass out on em0 inet proto tcp \ from any to any port 25 • macros, quick implies flow control -
sendmail.mc define(`confTO_CONNECT', `1m')dnl define(`confTO_IDENT', `0')dnl define(`confTO_COMMAND', `2m')dnl LOCAL_NET_CONFIG # This
rule ensures that all local mail is delivered using the # smtp transport, everything else will go via the smart host. R$* < @ $* .$m. > $* $#smtp $@ $2.$m. $: $1 < @ $2.$m. > $3 • no comments -
"Programming" with JSON (CloudForma8on) { "Type": "AWS::ElasticLoadBalancing::LoadBalancer", "Properties": { "Listeners":
[{ "SSLCertificateId": {"Fn::FindInMap": ["SSLMAP", "au", {"Ref": "EnvironmentType"}]}, }], "LoadBalancerName": {"Fn::Join": ["", ["app-web-", {"Ref": "EnvironmentName"}, ""]]} } } • people are actually expected to write that • lisp users can weep now -
YAML / Ansible tasks: - name: take out of load
balancer pool command: /usr/bin/take_out_of_pool {{ inventory_hostname }} delegate_to: 127.0.0.1 • string interpola.on, apparently -
"Programming" with YAML - command: /opt/application/upgrade_db.py when: inventory_hostname == webservers[0]
• implicit flow control? python syntax? -
Puppet: resource-oriented DSL, isolated effects Although Puppet’s language is built
around describing resources (and the rela6onships between them) in a declara6ve way, several parts of the language do depend on evalua6on order case $operatingsystem { centos, redhat: { $service_name = 'ntpd' } debian, ubuntu: { $service_name = 'ntp' } } package { $service_name: ensure => installed, } -
Puppet + Hiera Apparently trea+ng code and data separately is
cool again! --- mysql::server::root_password: 'strongpassword' databases: gotcms: user: 'got' password: 'super_secret_db_password' host: 'localhost' grant: 'ALL' -
Chef: DSL + side effects ruby_block 'sleep30s' do block do
sleep 30 end action :nothing end -
Building good Dockerfiles is mostly folklore FROM debian:jessie RUN apt-get
update && \ apt-get install -y --no-install-recommends openjdk-7-jre-headless ADD http://someplace/zookeeper-3.4.6.tar.gz /opt/ RUN cd /opt && \ tar -zxvf zookeeper-3.4.6.tar.gz && \ mv zookeeper-3.4.6 zookeeper && \ rm -rf ./zookeeper-*tar.gz && \ mkdir -p /var/lib/zookeeper ADD entry.sh /entry.sh WORKDIR /opt/zookeeper EXPOSE 2181 2888 3888 VOLUME ["/var/lib/zookeeper", "/opt/zookeeper/conf", "/tmp/zookeeper"] ENTRYPOINT ["/entry.sh"] -
<?xml version='1.0' encoding='UTF-8'?> <project> <actions/> <description></description> <logRotator> <daysToKeep>7</daysToKeep> <numToKeep>-1</numToKeep> <artifactDaysToKeep>-1</artifactDaysToKeep>
<artifactNumToKeep>-1</artifactNumToKeep> </logRotator> <keepDependencies>false</keepDependencies> <properties/> <scm class="hudson.scm.NullSCM"/> <canRoam>true</canRoam> <disabled>false</disabled> <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding> <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding> <triggers class="vector"/> -
<tr> <xsl:attribute name="style"> <xsl:choose> <xsl:when test="CONDITION"> <xsl:value-of select="'visibility: visible'"> </xsl:when>
<xsl:otherwise> <xsl:value-of select="'visibility: collapse'"> </xsl:otherwise> </xsl:choose> </xsl:attribute> </tr> -
% cat databass.yaml sg: db-alpha.infra.zalora.io my: db-psi.infra.zalora.io id: db-psi.infra.zalora.io hk:
db-psi.infra.zalora.io th: db-theta.infra.zalora.io ph: db-beta.infra.zalora.io vn: db-beta.infra.zalora.io -
• MariaDB channels are server-centric • our model is country-centric
• countries can share servers have to aggregate! -
Any insufficiently expressive DSL leads to the necessity of using
an ul:mate systems integra:on tool. -
% cat databass.yaml | \ awk -F': ' '{print $2,
$1}' | \ awk '{a[$1] = a[$1] (a[$1] ? "," : "") $2; } END { for(k in a) print k, a[k]; }' db-beta.infra.zalora.io ph,vn db-psi.infra.zalora.io my,id,hk db-alpha.infra.zalora.io sg db-theta.infra.zalora.io th -
Bash • a pi&all trap that looks easy • somewhat
ubiquitous and mostly unportable • hard to scale • strings • more strings • unexpected EOF while looking for matching `"' • also strings -
Devops Success Stories 101 • when in doubt, "do it
manually just this once" • when repeated 5 9mes, wrap with a string interpola9on library • if not enough, try to add flow control as a "declara9ve" construct • when failed, embed a random PL with unclear boundaries • of course, error handling is ad-hoc trial-and-error • weep -
db-slave-channels = let mapper = _: { db-name, masterhost, ...
}: nameValuePair (to-key master-host) { inherit master-host; databases = [ db-name ]; }; reducer = { name, value }: all: all // { ${name} = value // { databases = all.${name}.databases or [] ++ value.databases; }; }; in fold reducer {} (mapAttrsToList mapper conf); -
bash = "${pkgs.bash}/bin/bash"; base64 = "${pkgs.coreutils}/bin/base64"; jq = "${pkgs.jq}/bin/jq"; curl
= "${pkgs.curl}/bin/curl -s --retry --fail"; awk = "${pkgs.gawk}/bin/awk"; openssl = "${pkgs.openssl}/bin/openssl"; • automa'c dependency tracking! -
Nix • func&onal language • dynamically typed • lightweight "schema
valida&on" (hard to say typing) • one side-effect (derivation, used to build a package manager framework) -
Case study: configuring Jenkins • string templa.ng again? • click
through all the forms? • can we build a DSL? -
eris-sdk = erisJob { branch = "master"; shell = ''
export SLACK_CHANNEL='#eris-facepalm' export SLACK_TIMEOUT=5m bin/slack make sdk ''; ssh-keys = [ credentials.hydrabot ]; triggers = [ (ghprb-trigger "eris-sdk") (github-push-trigger) ]; }; -
mapAttrs' (realm-name: spec: deployJob "deploy-${realm-name}" { inherit realm-name; scm =
eris_master; permissions = with spec; { build = humans-can-build ++ others-can-build; }; ssh-keys = attrValues credentials; })) (filterAttrs is-deployable realms); -
choice-parameter = { name , description ? "" , choices
? [ "this is a list" ] }: (term "hudson.model.ChoiceParameterDefinition" null [ (term "name" null name) (term "description" null description) (term "choices" { class = "java.util.Arrays$ArrayList"; } [ (term "a" { class = "string-array"; } (map (term "string" null) choices)) ]) ]); -
Run$mes • implementa*on of Apps APIs to linux services •
same APIs - mul*ple backends • allows rapid experimenta*on: mesos/docker/diego/ ares -
apps runtimes targets ┌─────────┐ ┌──────────┐┌%%%%%%%%%%%%%%%%%%%┐ ┌───> ASG │ ┌───> mysql
││ nixos/systemd │──┼┐ └─────────┘ │ └──────────┘└───────────────────┘ ││ ┌───────────┐ │ ┌──────────┐┌###################┐ ││ ┌─────────┐ │ realm ├──┴┬──>strongswan││ adhoc │──┼┼───>baremetal│ └───────────┘ │ └──────────┘└───────────────────┘ ││ └─────────┘ │ ┌──────────┐┌&&&&&&&&&&&&&&&&&&&┐ ││ └──> someweb ││ ami ├──┘│ ┌─────────┐ └──────────┘└───────────────────┘ └──> ec2 │ └─────────┘ -
Case study: containers • adhoc run+me deployed to a fleet
of old RedHat 6 servers • adhoc apps were exposed to global state • adop+ng runc • +me to PoC - 3 hours • +me to produc+on - 12 hours (8 wasted tracing kernel code) • running on 2.6.32.602 with minor libcontainer patches -
infra = { ec2-instance = { web1 = m3large; web2
= m3large; db-master = { infra, ... }: r3xlarge' { blockDeviceMapping."/dev/xvdm".disk = infra.ebs.database; }; cron = instance "m3.large"; }; elb = elb.defaults; ebs.database = { inherit (realm.ec2-args) region zone; size = 200; volumeType.gp2 = true; }; }; -
Infrastructure Specs • nixops-inspired -> reimplemented in upcast • terraform
implementa5on as an evening hack • autoscaling PoC was wri=en bash :O -
writeBashScript = name: script: let prelude = '' #!${pkgs.bash}/bin/bash set
-e -o pipefail ''; in pkgs.runCommand name { inherit prelude script; } '' echo "$prelude" >> "$out" echo "$script" >> "$out" chmod +x "$out" ${ShellCheck}/bin/shellcheck \ "$out" ''; -
• zalora/upcast - declara1ve infra provisioning (like nixops/terraform/fugue) • zalora/replicator
- automated MySQL replica1on • zalora/sproxy - proxy that handles OAuth2 + ACL interface • zalora/aws-ec2 - EC2 extensions for aris1db/aws • unicron, a single-user cron • a lot more on Zalora's GitHub -
Correct by construc-on: safe Bash • harden your interoperability with
type-safe APIs data Expr :: * -> * where E :: Executable -> e -> Expr e Pipe :: Expr e -> Expr e -> Expr e Seq :: Expr e -> Expr e -> Expr e Or :: Expr e -> Expr e -> Expr e Redir :: Expr e -> FilePath -> Expr e Env :: [Pair] -> Expr e -> Expr e Sudo :: Expr e -> Expr e SSH :: Hostname -> e -> Expr e -> Expr e -
Correct by construc-on: safe Nix data Nix :: * ->
* where Null :: Nix () Lit :: Literal -> Nix a -- ... Var :: Index -> Nix a Lam :: Index -> Nix a -> Nix (Nix input -> Nix a) Ap :: Nix (Nix input -> Nix a) -> Nix input -> Nix a -- ... List1 :: Nix a -> Nix [Nix a] -
Typing the standard library listToAttrs :: Nix [Nix (Nix String,
Nix a)] -> Nix (Attrset a) listToAttrs = Ap (Lit "builtins.listToAttrs") concatLists :: Nix [Nix [a]] -> Nix [a] concatLists = Ap (Lit "builtins.concatLists") map' :: Nix (a -> b) -> Nix [a] -> Nix [b] map' f xs = Ap (Ap (Lit "map") f) xs attrNames :: Nix (Attrset a) -> Nix [Nix String] attrNames = Ap (Lit "builtins.attrNames") -
Induc&on Composi&on! λ> let concatMapToAttrs = (\f -> listToAttrs .
concatLists . map' f) λ> :t concatMapToAttrs concatMapToAttrs :: Nix (a1 -> Nix [Nix (Nix String, Nix a)]) -> Nix [a1] -> Nix (Attrset a) -
λ> :t null null :: Nix () λ> :t concatMapToAttrs
null <interactive>:1:18: Couldn't match type ‘()’ with ‘a1 -> Nix [Nix (Nix String, Nix a)]’ Expected type: Nix (a1 -> Nix [Nix (Nix String, Nix a)]) Actual type: Nix () -
Correct by construc-on: web APIs data DeploymentParams = -- ...
data DeploymentStatus = -- ... type Deployments = "deployments.xml" :> Header "x-api-key" ApiKey :> ReqBody '[FormUrlEncoded] DeploymentParams :> Post '[XML] DeploymentStatus h"ps:/ /haskell-servant.github.io Type-safe generic programming! -
Interpre'ng effects wtf :: (Member Spec r, Member Upcast r)
=> Eff r (Map Machine ExitCode) wtf = infra >>= machines >>= traverse (`ssh` cmd) where cmd :: Commandline cmd = exec "wtf" [] • notable implementa-on: Eff-lang -
-- | Idempotent effects based on evaluating the spec (ok
to cache). data Spec v where NixQuery :: FromJSON json => Query json -> Spec json Infra :: Spec Infras Stub :: Show a => a -> Spec () -- | Effects that require networking. data Upcast v where Machines :: Infras -> Upcast (Attrs Machine) NixBuild :: Query a -> Upcast [StorePath] NixInstantiate :: Query a -> Upcast [StorePath] MachineExec :: (Machine, Commandline) -> Upcast ExitCode SystemInstalls :: (Machine, StorePath) -> Upcast ExitCode NRNotification :: ApiKey -> [DNS] -> Upcast () -
Integra(ng with the rest of the stringy world • UNIX
is a minefield for experimen5ng with parser combinators! • a lot of perf analysis or systems explora5on is done by analysing streams of text • use haskell if lost in awk+perl+sed • see proger/lxkit and zalora/gctuner -
finally { • infrastructure tools come and go • they're
filled with adhoc interfaces with poor composablity and/or expressiveness • "tradi=onal" composi=on prac=ces are too error- prone and divert your precious a?en=on to fixing irrelevant bugs • basic PLT tools like transforma=onal programming and type systems keep you sane