Scripting creating AWS based servers

Scripting your servers
@coldclimate CANDDi.com

View Slide

Put your pens down

View Slide

There once was a man...

View Slide

And then there was a document...

View Slide

Which became a wiki...

View Slide

Which finally because a script

View Slide

Which mostly worked...

View Slide

And then the madness had to stop

View Slide

You wouldn’t treat code like this
(hopefully)

View Slide

Man => Document => Wiki => Script =>
Images

View Slide

Encoding
Version controlling
(automating)

View Slide

Man => Document
Removing the bus factor
“The man we can’t fire”

View Slide

Document => Wiki
The fucker *might* stay up to date

View Slide

Wiki => Script
Because script in documentation
is stupid

View Slide


View Slide

Why are you not talking about
chef, puppet, ansible etc?

View Slide

http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html

View Slide

Build once or build twice?

View Slide

Build once
Raw OS
Base
Instance
Script
Working
Instance

View Slide

Kick-off instance
Install packages
Configure them
Install your software
Hope it all works

View Slide

Build Twice
Raw OS
Base
Instance
Script “Base Box” Make AMI
“Base Box”
“Instance
Box”
Script
Base AMI

View Slide

Build once or build twice?

View Slide

Kick-off instance
Install packages
--------------------------------------------
Configure them

View Slide

Kick-off instance
Install packages
--------------------------------------------
Configure them
Changes rarely
Changes frequently

View Slide

Kick-off instance
Install packages
--------------------------------------------
Configure them
AMI
Instance

View Slide

Stability

View Slide

Speed

View Slide

The game:
always runs, no manual fixes

View Slide

A script to built an AMI
https://gist.github.com/coldclimate/5ff56c96dac116f3234e

View Slide

ec2-run-instances
--region 'eu-west-1'
-g TEMP
-k TEMP
-f install.sh
-t t1.micro
ami-ce7b6fba

View Slide

install.sh
limited size (16kb)
runs as root
cock to debug

View Slide

/var/log/cloud-init.log

View Slide

set -ex

View Slide

Add any sources
Update the APT repository
Install core packages

View Slide

Add any sources

View Slide

echo "deb http://archive.canonical.com/ precise
partner" | sudo tee -a /etc/apt/sources.list

View Slide

python-setuptools
add-apt-repository -y ppa:ondrej/php5

View Slide

apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist
10gen' | sudo tee /etc/apt/sources.list.d/10gen.list

View Slide

Update the APT repository

View Slide

apt-get update

View Slide

Install core packages

View Slide

apt-get -y install zip unzip xmlstarlet
sysstat iftop ntp sysvbanner build-
essential python-setuptools nginx
nodejs

View Slide

easy_install -U boto

View Slide

wget https://github.com/nicolasff/phpredis/archive/2.2.4.zip -P /tmp/
mkdir -p /opt/code
unzip /tmp/2.2.4.zip -d /opt/code/
cd /opt/code/phpredis-2.2.4
phpize
./configure
make
make install

View Slide

banner “*** DONE BABY ***”

View Slide

ec2-create-image
-n
instance_id

View Slide

Making running instance

View Slide

ec2-run-instances
--region 'eu-west-1'
-g sg-bb18b66c
-k LIVE-PEM-KEY
-p Instance_Profile
-f install.sh
-t m1.small
--availability-zone eu-west-1a
--disable-api-termination
ami-XXXXXX

View Slide

Roles
http://docs.aws.amazon.com/cli/latest/reference/iam/create-role.html
Instance Profiles
http://docs.aws.amazon.com/cli/latest/reference/iam/create-instance-profile.html
Adding a Role to an Instance Profile
http://docs.aws.amazon.com/cli/latest/reference/iam/add-role-to-instance-profile.html

View Slide

Download from S3
Setup directories
Setup config
Install code
Start it up
Register it

View Slide

Download from S3
python <import boto
import sys
from boto.s3.key import Key
def get_wonka(bucket_name, file_name, destination_file):
conn = boto.connect_s3()
bucket = conn.get_bucket(bucket_name)
k = Key(bucket)
k.name = file_name
destination = "/tmp/%s" % (destination_file)
k.get_contents_to_filename(destination)
get_wonka("BUCKET","SOURCEFILE","DESTINATION")
PY

View Slide

Setup directories
Your script is running as root
so remember to chown

View Slide

Setup config
Copy it from the download into place
Restart the service

View Slide

Install code
https://github.com/jordansissel/fpm

View Slide

Start it up
Supervisord
Ubuntu service
Avoid own scripts

View Slide

Register it

View Slide

Monitoring (zabbix)

View Slide

Load Balancer (ec2 call)

View Slide

Logging (papertrail)

View Slide

DNS (R53 call)

View Slide

And it works!

View Slide


View Slide

Need to change the code or config?

View Slide

JFDI and roll a fresh one

View Slide

You need to change the base AMI

View Slide

It’s simple, it just doesn’t feel it

View Slide

Update the scripts until you can
“launch clean”

View Slide

Lastly

View Slide


View Slide

speakerdeck.
com/coldclimate/scripting-creating-
aws-based-servers
@coldclimate
canddi.com/blog

View Slide

Scripting creating AWS based servers

Scripting creating AWS based servers

coldclimate

More Decks by coldclimate

Other Decks in Technology

Featured

Transcript