Herding cattle with Clojure at MixRadio

A talk given at Clojure eXchange 2014.

Video available here: https://skillsmatter.com/skillscasts/6057-herding-cattle-with-clojure-at-mixradio

Transcript

1. Herding cattle with Clojure at @neilprosser
2. Who are we?
3. Who am I?
4. Cattle?
5. "Treat your servers like cattle, not pets." @noahcampbell
6. Our platform
7. RESTful microservices (﴾pragmatic)﴿
8. Clojure
9. mr-‐clojure https://github.com/mixradio/mr-‐clojure
10. Legacy Java
11. PHP
12. And others...
13. Problems
14. Snowflake servers
15. Slow provisioning
16. Slow deployment time
17. Configuration confusion
18. Time for AWS
19. Environments
20. Dev account Prod account poke betapoke newsearch prod betaprod
21. mr-‐*
22. http://www.penguin.com.au/products/9781846462955/mr-‐men-‐and-‐little-‐miss-‐mr-‐tickle
23. mr-‐klink mr-‐lister mr-‐baker mr-‐tyrant mr-‐pedantic mr-‐maestro AWS
24. mr-‐klink Command-‐line Swiss Army knife
25. mr-‐klink mr-‐baker mr-‐pedantic AWS mr-‐lister mr-‐tyrant mr-‐maestro
26. Written in Go
27. Cross-‐platform
28. Uses our RESTful interfaces
29. Create application > klink create-‐app clojurex -‐E me@somewhere.com
30. > klink images clojurex Find machine images
31. > klink deploy clojurex prod ami-‐deadbeef Deploy
32. > klink boxes clojurex prod List boxes
33. mr-‐lister Application and environment list https://github.com/mixradio/mr-‐lister
34. mr-‐klink mr-‐baker mr-‐pedantic AWS mr-‐lister mr-‐tyrant mr-‐maestro
35. { "name": "search", "metadata": { "bakeType": "hvm", "contact": "Fusion", "jobsPath": "http://jenkins/view/Search/", "releasePath": "http://jenkins/job/search-‐release/", "srcRepo": "git@github:helios/search.git", "statusPath": "/search-‐query-‐rest/2.x/ping" } }
36. Backed by DynamoDB
37. Faraday https://github.com/ptaoussanis/faraday
38. (ns applications (:require [taoensso.faraday :as far])) (def table :applications) (defn create-‐application [name] (far/put-‐item table {:name name :some "info"})) (defn get-‐application [name] (far/get-‐item table {:name name}))
39. Demo
40. mr-‐baker Machine image bakery
41. mr-‐klink mr-‐baker mr-‐pedantic AWS mr-‐lister mr-‐tyrant mr-‐maestro
42. Bake once, deploy everywhere
43. Phoenix servers
44. No upgrades
45. Amazon Linux base image
46. Amazon Linux base image MixRadio base image
47. Amazon Linux base image MixRadio base image Service image
48. Amazon Linux base image MixRadio base image Service image Ad-‐hoc image
49. Shells out to Packer https://packer.io/
50. Conch https://github.com/Raynes/conch
51. (ns application (:require [me.raynes.conch :as conch])) (defn packer-‐build [template-‐path] (conch/let-‐programs [packer "/opt/packer/packer"] (let [out-‐stream (PipedOutputStream.) in-‐stream (PipedInputStream. out-‐stream)] (future (packer "build" template-‐path {:out out-‐stream :timeout (* 1000 60 30)})) in-‐stream)))
52. Demo
53. mr-‐tyrant Per-‐environment configuration https://github.com/mixradio/mr-‐tyrant
54. mr-‐klink mr-‐baker mr-‐pedantic AWS mr-‐lister mr-‐tyrant mr-‐maestro
55. Backed by Github
56. Tentacles https://github.com/Raynes/tentacles
57. { "anotherService.baseUrl": "https://somewhere:8080/api", "database.connectionString": "mysql://db:3306/database", "service.port": 8080 } Application properties
58. { "desiredCapacity": 2, "max": 2, "min": 2, "selectedLoadBalancers": [ "application" ], "selectedSecurityGroups": [ "internal-‐8080", "internal-‐ssh" ] } Deployment parameters
59. [ "/usr/bin/yes | mdadm -‐C /dev/md0 -‐v -‐n 2 -‐l 0 /dev/sdb /dev/sdc", "mkfs.ext4 /dev/md0", "mkdir -‐p /var/lib/solr", "mount /var/lib/solr", "chown -‐R solr:solr /var/lib/solr" ] Launch data
60. #!/bin/bash cat > /etc/profile.d/asgard.sh <<EOF export CLOUD_APP=application export CLOUD_CLUSTER=application-‐environment export CLOUD_STACK=environment EOF cat > /etc/application.properties <<EOF anotherService.baseUrl=https://somewhere:8080/api database.connectionString=mysql://db:3306/database service.port=8080 EOF "/usr/bin/yes | mdadm -‐C /dev/md0 -‐v -‐n 2 -‐l 0 /dev/sdb /dev/sdc" "mkfs.ext4 /dev/md0" "mkdir -‐p /var/lib/solr" "mount /var/lib/solr" "chown -‐R solr:solr /var/lib/solr"
61. No outbound connectivity on launch
62. mr-‐pedantic Keeping our infrastructure in sync https://github.com/mixradio/mr-‐pedantic
63. mr-‐klink mr-‐baker mr-‐pedantic AWS mr-‐lister mr-‐tyrant mr-‐maestro
64. "Puppet for cloud infrastructure"
65. Lets us to roll out in another region
66. Backed by Github
67. Clojure for configuration
68. (def $sg-‐http-‐80 "internal-‐http") (def $sg-‐egress-‐all [{:IpRanges "0.0.0.0/0" :IpProtocol -‐1}]) (def $app-‐subnets ["subnet-‐a123321a" "subnet-‐b456654b" "subnet-‐c789987c"]) (def $private-‐ips ["192.168.0.0/16"]) (def $vpc-‐id "vpc-‐d123321d") {:SecurityGroups [{:GroupName $sg-‐http-‐80 :GroupDescription "internal-‐http" :VpcId $vpc-‐id :Ingress [{:IpRanges $private-‐ips :IpProtocol "tcp" :FromPort 80 :ToPort 80}] :Egress $sg-‐egress-‐all}]}
69. (def $sg-‐http-‐80 "internal-‐http") (def $sg-‐egress-‐all [{:IpRanges "0.0.0.0/0" :IpProtocol -‐1}]) (def $app-‐subnets ["subnet-‐a123321a" "subnet-‐b456654b" "subnet-‐c789987c"]) (def $private-‐ips ["192.168.0.0/16"]) (def $vpc-‐id "vpc-‐d123321d") {:SecurityGroups [{:GroupName "internal-‐http" :GroupDescription "internal-‐http" :VpcId "vpc-‐d123321d" :Ingress [{:IpRanges ["192.168.0.0/16"] :IpProtocol "tcp" :FromPort 80 :ToPort 80}] :Egress [{:IpRanges "0.0.0.0/0" :IpProtocol -‐1}]}]}
70. (def $app-‐name "something") (def $env "prod")
71. (def bucket (if (= $env "prod") "prod-‐bucket" "dev-‐bucket")) {:LoadBalancer {:LoadBalancerName $app-‐name :Listeners [{:LoadBalancerPort 80 :InstancePort 8080 :Protocol "http" :InstanceProtocol "http"}] :SecurityGroups [$sg-‐http-‐80] :Subnets $app-‐subnets :Scheme "internal" :HealthCheck {:Target "HTTP:8080/healthcheck" :HealthyThreshold 2 :UnhealthyThreshold 4 :Interval 12 :Timeout 5}} :Role {:RoleName $app-‐name} :S3 [{:BucketName bucket}]}
72. (def $sg-‐http-‐80 "internal-‐http") (def $sg-‐egress-‐all [{:IpRanges "0.0.0.0/0" :IpProtocol -‐1}]) (def $app-‐subnets ["subnet-‐a123321a" "subnet-‐b456654b" "subnet-‐c789987c"]) (def $private-‐ips ["192.168.0.0/16"]) (def $vpc-‐id "vpc-‐d123321d") (def bucket (if (= $env "prod") "prod-‐bucket" "dev-‐bucket")) {:LoadBalancer {:LoadBalancerName $app-‐name :Listeners [{:LoadBalancerPort 80 :InstancePort 8080 :Protocol "http" :InstanceProtocol "http"}] :SecurityGroups [$sg-‐http-‐80] :Subnets $app-‐subnets :Scheme "internal" :HealthCheck {:Target "HTTP:8080/healthcheck" :HealthyThreshold 2 :UnhealthyThreshold 4 :Interval 12 :Timeout 5}} :Role {:RoleName $app-‐name} :S3 [{:BucketName bucket}]} (def $app-‐name "something") (def $env "prod")
73. Danger!
74. (ns clojure.core) (defn not [x] true)
75. People do silly things
76. Clojail https://github.com/Raynes/clojail
77. (evaluate-‐string "(ns clojure.core) (defn not [x] true)") > SecurityException You tripped the alarm! class clojure.lang.Var is bad!
78. mr-‐maestro Deployment orchestration https://github.com/mixradio/mr-‐maestro
79. mr-‐klink mr-‐baker mr-‐pedantic AWS mr-‐lister mr-‐tyrant mr-‐maestro
80. (def deployment (+ configuration image))
81. Asgard https://github.com/Netflix/asgard
82. Trigger Asgard via APIs
83. Watch the tasks
84. Store the data
85. Red/black deployment
86. Red/black deployment load balancer v1
87. Red/black deployment load balancer v1 v2
88. Red/black deployment load balancer v1 v2
89. Red/black deployment load balancer v1 v2
90. Red/black deployment load balancer v2
91. Great for finding out what we wanted
92. Eventually we wanted more
93. Rolled our own
94. Break deployment into tasks
97. Redis message-‐queue https://github.com/ptaoussanis/carmine
98. (defn some-‐task [message] ...) {:id "abc" :attempt 1 :parameters {:value "one"}} {:parameters {:value "one" :something "else"} :status :success}
99. (defn populate-‐vpc-‐zone-‐identifier [{:keys [parameters]}] (let [{:keys [environment region]} parameters state (:new-‐state parameters) {:keys [selected-‐subnets]} state] (log/write "Populating VPC zone identifier.") {:status :success :parameters (assoc-‐in parameters [:new-‐state :vpc-‐zone-‐identifier] (str/join "," selected-‐subnets))}))
100. (defn send-‐completion-‐notification [{:keys [parameters]}] (try (n/send-‐completion-‐message parameters) {:status :success :parameters parameters} (catch Exception e {:status :error :throwable e})))
101. (defn wait-‐for-‐old-‐auto-‐scaling-‐group-‐deletion [{:keys [attempt parameters]}] (let [old-‐asg-‐name (old-‐asg-‐name parameters)] (try (if (asg old-‐asg-‐name environment region) (do (when (= 1 attempt) (log/write "Waiting for auto scaling group deletion.")) (capped-‐retry-‐after 10000 attempt max-‐attempts)) (do (log/write "Auto scaling group deleted.") (success parameters))) (catch Exception e (error-‐with e)))))
102. (ns maestro.actions (:require [linked.set :refer [linked-‐set]] [maestro.messages [asg :as asg] [data :as data] [health :as health] [notification :as notification]])) (def ^:private action-‐ordering (linked-‐set ... :maestro.messages.data/generate-‐user-‐data :maestro.messages.data/complete-‐deployment-‐preparation :maestro.messages.data/start-‐deployment :maestro.messages.asg/create-‐launch-‐configuration :maestro.messages.asg/create-‐auto-‐scaling-‐group :maestro.messages.asg/disable-‐adding-‐instances :maestro.messages.asg/add-‐scaling-‐notifications ...))
103. (defn to-‐function [action-‐kw] (when action-‐kw (let [ns (namespace action-‐kw) name (name action-‐kw)] (resolve (symbol (format "%s/%s" ns name))))))
104. Can easily add new tasks
105. { "application": "app", "environment": "env", "id": "dccffd4c-‐af6a-‐4bed-‐b49a-‐92935e85479c", "message": "Make things better", "new-‐state": { "auto-‐scaling-‐group-‐name": "app-‐env-‐v030", "auto-‐scaling-‐group-‐tags": [...], "availability-‐zones": [...], "block-‐device-‐mappings": [...], "created": true, "hash": "de41a5d44f2248e13c591e717fd26e916dc3c577", "image-‐details": { "application": "app", "bake-‐date": "2014-‐11-‐25T17:54:36.000Z", "id": "ami-‐0c5dedeb", "image-‐name": "ent-‐app-‐1.2-‐1-‐hvm-‐2014-‐11-‐25_17-‐54-‐36", "iteration": "1", "version": "1.2", "virt-‐type": "hvm" },
106. "launch-‐configuration-‐name": "app-‐env-‐v030-‐20141125182609", "lister": {...}, "selected-‐security-‐group-‐ids": [...], "selected-‐subnets": [...], "termination-‐policies": [...], "tyrant": { "application-‐properties": {...}, "deployment-‐params": {...}, "launch-‐data": [...] }, "user-‐data": ..., "vpc-‐zone-‐identifier": ... }, "phase": "deployment", "previous-‐state": {...}, "region": "eu-‐west-‐1", "silent": false, "start": "2014-‐11-‐25T18:25:52.548Z", "status": "running", "user": "nprosser" }
107. Elastisch https://github.com/clojurewerkz/elastisch
108. Deployments are scalable
109. Maestro can deploy itself
110. Amazonica https://github.com/mcohen01/amazonica
111. (ns application (:require [amazonica.aws.ec2 :as ec2])) (ec2/describe-‐instances) => {:reservations [{:instances [...]}]}
112. Demo
113. mr-‐klink mr-‐baker mr-‐pedantic AWS mr-‐lister mr-‐tyrant mr-‐maestro
114. Automatic process
115. Over 5000 deployments
116. Wrapping up
117. Clojure makes things simple
118. Functional code is easier to reason about
119. Deployment tooling is personal
120. Work in the open
121. https://github.com/mixradio
122. Work like you’ll open-‐source everything
123. Questions? We are hiring http://dev.mixrad.io/jobs