My Background Based in London, previously a consultant ➤ Started with JS, C & Objective-C Took a more operationally involved role in 2013–2014 ➤ Learned a whole bunch of other stuff Started working with Erlang/Elixir in 2015
Caveat: Advice Applicability This presentation consists of solely my personal opinion; any resemblance of facts should be considered a mere coincidence. Depending on your current scale and cultural inclination, they may or may not be applicable to your product.
Details There’s lots of prior art on deploying Elixir / Phoenix ➤ Dockerisation, OTP Releases, etc. already covered numerous times ➤ Distillery is clearly the way forward Starting from scratch is a different story ➤ Let’s try to build up a decision framework
What I’ll Cover Today Theory: design principles and evaluation criteria for your deployment Requirements: things that are important to have in your deployment Design: How we currently address this issue Implementation: SSL/TLS steering, clustering, configuration, etc Demo: (if time permits) a full demonstration of what we’re using Then try to answer any question
Reality as I Perceive It #1: Knowledge Transfer Takes Non-Zero Time ➤ Erlang/OTP is a beautiful and pioneering tool ➤ Many concepts have found their way outside the Erlang/OTP ecosystem ➤ Load balancing, clustering, service discovery, etc ➤ Your colleagues probably already know how to use them ➤ Key: Your colleagues are knowledgeable: ensure they are useful
Reality as I Perceive It #2: Opportunity Cost: Platform vs Product Development ➤ You wrote it — You maintain it! ➤ Any code written by your team will need to be maintained for the entire lifetime of its existence by your team, using your team’s budget ➤ Consider opportunity cost of custom platform engineering ➤ Seek maximum leverage from commercial off-the-shelf solutions ➤ Key: Fully exploit well-known solutions created and maintained by others
Reality as I Perceive It #3: You’re Probably in Vendor Lock-In Anyway ➤ “Portability” is a damned lie; it’s vendor-specific shims all the way down ➤ Open Source Languages/Frameworks — Always! ➤ Closed Source Third-Party Services — Some? Many? All? ➤ Post-porting activities also important: fine-tune, measure, monitor, re-adjust… ➤ Might as well design and exploit each platform to their fullest ➤ Key: When possible, spend other people’s R&D money
Guiding Opinions Use existing non-Erlang/OTP solutions when appropriate ➤ Exploit existing knowledge; avoid forced re-learning Use as many pre-built solutions as possible ➤ Build upon existing solutions by your community Properly integrate with your primary platform ➤ An existing and adequate solution is still better than nothing Also: try to follow https://12factor.net as closely as possible
Infrastructure Design Goals There are only 2 metrics you must not compromise: ➤ Minimum hours spent away from work per day, per team member ➤ Number of involuntary interventions per week/month/year Do not optimise for “developer happiness” or “developer productivity” ➤ These are by-products of a system correctly designed for perpetual operation ➤ Focus on operational stability and sustainability instead
2014: WhatsApp https://github.com/reedr/reedr/blob/master/slides/efsf2014- whatsapp-scaling.pdf 1 That's Billion with a B: Scaling to the next level at WhatsApp Rick Reed WhatsApp Erlang Factory SF March 7, 2014
Requirements Predictable: all important bits automated; least surprises Resilient: benign issues do not require human intervention Efficient: quick to deploy/rollback, quick to start, etc Secure: proper role/net segregation, minimised public footprint Observable: easy to monitor/intervene (say, with console)
“Predictable” Provide assurance that any change will… ➤ run to its completion, ➤ in a timely manner, ➤ make changes atomically only, and ➤ provide an actionable, reasonable result, ➤ on either release or rollback
“Predictable” Scenario 1: typo in production code ➤ Good: production push stopped automatically (and serenely) ➤ Bad: site brought down, alerts triggered, change rolled back Scenario 2: lots of moving parts / dependencies ➤ Good: flaky bits designed away, deployment always works ➤ Bad: constantly wiggling the snowflakes
“Resilient” Common failure modes are accounted for. ➤ Within reason: you can probably lose an AZ but not an entire region Intermittent errors can be recovered from automatically. ➤ Let your Supervision Tree sort out the easy bits. ➤ Have your infrastructure pick up the hard bits. ➤ NB: they operate at different levels and are actually complementary
“Resilient” Scenario 1: Disk filling up? ➤ Good: EC2 instance fails health check and is replaced automatically. New containers launched on the replacement instance; service continues. ➤ Bad: Intermittent 500s start to appear. Malaised server generates errors quicker than everything else. Everything is broken. The site is down.
“Resilient” Scenario 2: You’ve lost your DB primary? ➤ Good: Site goes read-only as read replica gets promoted to primary. Important events queued for deferred processing. Impacted Erlang/OTP processes restart. You may or may not have lost a couple transactions (depending on how your replication is built). ➤ Bad: Site goes down. Restoration from yesterday’s backup will take at least 4 hours. Third party events dropped. R.T.O. now on the floor.
“Efficient” Strong argument: deployment speed is a feature ➤ 30 seconds: good; 1 minute: OK, longer than 5 minutes: insufferable Helps reduce ceremonial role of deployments ➤ Much lower cost of error correction if deployments are fast Also helps you fulfil the inevitable “build a new stack” requests ➤ Staging / Test / Support / One-Off Odds & Ends / Big Migration / DR… ➤ You will probably get asked to “run a copy of PROD” with miracles expected
“Efficient” Scenario: “We’d like to test this big feature in a safe manner” Good: New stack created and traffic split at DNS / LB level. Everybody carries on working/testing. Eventually, rollout completed. Bad: Big-bang rollout which inevitably fails. After much gnashing of teeth and lots of finger-pointing, the operators were blamed (!) Probably OK: use feature flags…
“Secure” Put the RDBMS, Web Servers and Bastion Servers in separate places ➤ Utilise VPC capabilities to their fullest ➤ Additional layers around your systems Separate development/deployment/infrastructure roles ➤ Resource access/creation/mutation limited with dedicated roles ➤ Also allows you to deploy customer-hosted services as a vendor
“Secure” Scenario 1: CEO signed a managed services deal with TVBCOA… ➤ Good: “Run this template, then give us keys from its output, which are limited to deployment/maintenance for this service only.” ➤ Bad: “Yeah we need root access to these servers just to deploy…” Scenario 2: Lots of frenemies in one room ➤ Good: each division gets their own VPC ➤ Bad: everything can see everything else…
“Observable” Retain ability to kick the tires from time to time ➤ Have a mental picture of what “normal” looks like ➤ Different tools for Application and Infrastructure level needs Maintain ability to intervene quickly if required ➤ Little can be done by pure operators apart from scaling horizontally ➤ Developer access is still crucial as systems mature
“Observable” Scenario: Service is “wonky”, no further information available ➤ Good: Console access is available. Node connected to, root cause identified, and ad-hoc patch seems to alleviate the problem. New release created and rolled out. ➤ Bad: Insufficient access, so a new release was needed, just to add logging statements…
Our Current Setup — Networking One custom (non-default) VPC for Production, One for Development ➤ Some older AWS EC2 customers actually do not have a “Default VPC”! ➤ Even if they do, it would be poor form to put everything in there Three Subnets per Availability Zone ➤ One each for Public, Private and Data services: Public → Private, Private → Data ➤ For Internet Access (NTP, etc): Private → Public, Data → Public ➤ You can put in a S3 Gateway if you wish: cheaper S3 access!
Our Current Setup — Application Fully utilise AWS Container Service ➤ AWS-managed Kubernetes would be great. ➤ ECS hosts spread around all available AZs Fully utilise Docker for production builds ➤ We’re all using Macs anyway ➤ Development on macOS, Release/Production on Linux ➤ Helps catch system-specific issues
Our Current Setup — Management AWS CloudFormation for nearly everything ➤ This is much simpler than writing an Infrastructure Deployment Guide ➤ 100% repeatable, no churn, no faffing about ➤ Checked into Git and Version Controlled Locally run Bash scripts for everything else ➤ The choice is simple: either run Bash locally or trust/run Lambda Functions remotely. I’d rather do it locally ➤ Checked into Git and Version Controlled
Script: Create Stack Grab Input and Parameters ➤ Establish and pin dependency against upstream stacks ➤ Pinned resources can not change upstream Run Stack (as Administrator) ➤ You can make more Administrator accounts using IAM ➤ Meta: you could also build a self-service administrator account in IAM, which can only update the stack which created it.
Script: Update Resources Validate Environment Open a PostgreSQL connection to RDS Create Database with the right encoding/collation Create Role with the right permissions Run Seed Script/Migrations
Script: Deploy Release Validate Environment Build Containers with Docker Compose ➤ The same images previously used for validation are reused Push Images to ECS Container Repository Revise ECS Task Definition Revise ECS Service Definition
Script: Retrieve Platform Logs Validate Environment Query CFN/EC2/ECS for environmental particulars Retrieve latest events from ECS ➤ Has the cluster entered a stable state? Retrieve EC2 instances and their particulars Print metrics of particular interest (RAM/CPU utilisation, etc)
In the meantime, we use the awslogs tool to print CloudWatch Logs ➤ $ awslogs get log-group ALL --watch --start='15m ago' ➤ jorgebastida/awslogs Future Script: Application Logs
In the meantime, run docker exec… ➤ $ docker ps # find a container ➤ $ docker exec -it (container) iex -S mix Also for later consideration: attach a local Erlang node to VPC infra ➤ This is a straightforward matter of lining the ports up ➤ You could even do an ad-hoc task if you wish, but it’d be slower Future Script: Attach Console
Docker Everywhere Docker is basically a way to quickly provision servers ➤ Essential for automatic failover ➤ You can attach volumes for stuff you wish to keep Alternative: Cloud Init, Custom IAM, Ansible / Puppet ➤ Either slower (re-provisioning everything dynamically takes minutes to hours), or less efficient (entire IAMs need to be rebuilt: is your build infrastructure also reproducible?)
Docker Everywhere: Notes If you attach a volume, you could get the artefacts out. ➤ This gets you the underlying Erlang/OTP release ➤ You could then wrap it in another delivery mechanism ➤ Erlang on Xen, perhaps?
Ecto + SSL + Postgres Ecto allows you to enable encryption ➤ Heroku enforces it ➤ Other security-minded people like it ➤ You may have it turned on right now Now how do you test it…
#!/bin/bash set -e # # Copy the key pair to $PGDATA. The key pair has been # generated manually beforehand, as the container running # PostgreSQL does not have OpenSSL exposed anyway. # # https://www.postgresql.org/docs/9.1/static/ssl-tcp.html # See: 17.9.3. Creating a Self-signed Certificate # cp /docker-entrypoint-initdb.d/server.{crt,key} "$PGDATA" chown postgres:postgres "$PGDATA"/server.{crt,key} chmod 0600 "$PGDATA"/server.key # # Given that this is a development container, # we do not wish to play with PostgreSQL configuration too much # therefore a simple line appended to the end of the configuration # file will suffice. # echo "ssl = on" >> "$PGDATA/postgresql.conf"
SSL/TLS Serving + Steering When a customer accesses your HTTP endpoint, redirect to HTTPS ➤ Much better for applications: better than throwing an error ➤ Not really needed for programmatic access: just fail straight away X-Forwarded-Proto header ➤ Emitted by Heroku and AWS Elastic Load Balancer ➤ Supported by Plug.SSL and therefore Phoenix’s Endpoint ➤ config :app, Endpoint, force_ssl: [rewrite_on: [:x_forwarded_proto]]
Be Mindful of ELB Health Checks With TLS offloaded to the Elastic Load Balancer, all ELB / App interaction will be conducted in HTTP, including health checks. With HTTPS enforcement, health checks by default get 301. Adjust your Matcher accordingly to avoid knocking your site offline.
Clustering De facto entry point: libCluster ➤ https://github.com/bitwalker/libcluster ➤ Out-of-the-Box Support for: ➤ EPMD ([email protected] predefined) ➤ Gossip (UDP Multicast), ➤ Kubernetes ➤ You can provide a custom strategy
Clustering: Gossip in VPC? Multicast is not really supported in AWS ➤ https://aws.amazon.com/articles/6234671078671125 ➤ Establish VPN tunnels between each host., then fake it… You could emulate UDP multicast in AWS if you really want to ➤ A lot of work, though! ➤ Consider what the actual reward
Clustering: Roll Your Own? Run a state service behind a load-balancer / proxy ➤ Perhaps etcd or riak, but you could use Redis/Postgres if you wish Each Erlang/OTP node puts in an expiring “heartbeat” ➤ {nodename, host, port, cookie, memo}, TTL= 30 seconds Each Erlang/OTP node then tries to connect with each other ➤ Custom libCluster strategy needed
Clustering: Work Partitioning Once you have a cluster, you may wish to have work partitioned among your nodes. ➤ You can utilise Riak Core once you get a cluster going ➤ Alternatively: you can take a distributed lock and allocate from there ➤ Many other ways to distribute work — find the best for your application
Node Name / Node Host in ECS We ended up customising vm.args ➤ Pass a “placeholder” Erlang cookie, so we can deploy many revisions of something together without them talking to each other ➤ Actual Erlang Cookie kept in an AWS SSM Parameter Also customised the startup script ➤ Dynamically generate the Node Name to avoid clashes
# rel/config.exs environment :prod do set include_erts: false set include_src: false set cookie: :placeholder set vm_args: "./rel/vm.args" end # rel/vm.args ## Name of the node -name ${NODE_NAME} ## Cookie for distributed erlang -setcookie ${NODE_COOKIE}
# actual startup script run by ECS #!/usr/bin/env sh if [ -z $NODE_NAME ]; then nonce=$(cat /dev/urandom | \ LC_ALL=C tr -dc ‘a-zA-Z0-9' | \ fold -w 32 | head -n 1) hostname=$(hostname -I) export NODE_NAME="[email protected]$hostname" fi if [ -z $NODE_COOKIE ]; then export NODE_COOKIE="app-docker" fi export REPLACE_OS_VARS=true cd /app && ./releases/app/bin/app foreground
Secure SSH Tunneling + Proxying Actually open a PostgreSQL connection to RDS ➤ ssh -L local_port:remote_host:remote_port ➤ Tunnel from Laptop to Bastion ➤ Tunnel from Bastion to any ECS Host ➤ You must have at least one host in there anyway ➤ Result: localhost:5436 maps to RDS:5432 — success! ➤ Postgres should be configured to use SSL only
Seeding Data This should be part of infrastructure provisioning ➤ Goal: once the stack is handed off to the development team, it is already running the application in a fully functional fashion off the master branch You could tunnel to RDS and build up the state yourself ➤ Good way to consistently exercise the “Seed” file ➤ Good way to expose incorrect migrations too
Migrations There’s a script available in Distillery documentation ➤ https://hexdocs.pm/distillery/running-migrations.html#content It is a good starting point ➤ However, your migrations should always run with only Repo running ➤ This means no application dependencies ➤ iex -S mix --no-start ➤ Treat Ecto migrations as a stable Elixir to SQL transformation
Migrations: Sanity Check? If you run this on your application right now, will they work? ➤ mix ecto.drop ➤ mix ecto.create ➤ mix ecto.migrate ➤ mix run priv/repo/seeds.exs ➤ mix ecto.rollback --all ➤ mix ecto.migrate ➤ mix run priv/repo/seeds.exs
Interactive Elixir: Pro Tip ➤ You now have a supervision tree! ➤ However, you may not want to run everything when starting iex. ➤ Wouldn’t be good if a remote node started sharing production workloads ➤ You can check :init.get_arguments[:user] ➤ Does it have ’Elixir.IEx.CLI’? ➤ Optionally skip certain bits of your supervision tree
Observe Nodes There are some tools available ➤ http://www.brendangregg.com/linuxperf.html ➤ https://github.com/utkarshkukreti/ex_top ➤ http://zhongwencool.github.io/observer_cli/ ➤ https://github.com/shinyscorpion/wobserver ➤ https://github.com/IanLuites/wobserver-elixirconf-2017
evadne
okuzawats
moyheen
jeromevdl
oracle4engineer
minorun365
hazumirr
irof
y_iwai
taishin
okada_fuutass
skrb
mclloyd
mongodb
addyosmani
jmmastey
bkeepers
philhawksworth
zakiwarfel
scottboms
marcduiker
danielanewman
zakiyama
maltzj