AUTOMATING AWS 

WHY AUTOMATE? Reproducibility Functionality Treat UI as read-only 

ACCOUNT MANAGEMENT SPOILER Billing-only account No root access to other accounts: Production, Build, Staging, Demo, Dev. 

ROOT ACCOUNT Synonomous with AWS account. 

IAM ACCOUNT Belongs to an AWS account Cannot access billing Follows security policy 

IAM PASSWORD Optional Can be used with 2FA For human users 

API KEY Optional Can have 0-2 For robots users 

POLICIES Root can do anything "Admin" is almost anything Infinitely customizable 

EXAMPLE: EVERYTHING EXCEPT IAM {"Version": "2012-10-17", "Statement": [ {"Effect": "Deny", "Action": "iam:*", "Resource": "*"}, {"Effect": "Allow", "Action": "*", "Resource": "*"}]} 

CONSOLIDATED BILLING Create master account Create sub-accounts (It used to be harder) Bill easy to separate out 

CORPORATE AWESOMENESS Only admin/finance have access to master account No resources in master account Production account for user-facing Staging account Build/Test account Pets account 

CORPORATE AWESOMENESS: POLICIES Prod/stage/build -- all resources automated, tagged with "reason" Pets -- all resources tagged with dev name, GC by default 

BOTO3 BASICS 

AMAZON ACCOUNTS Dedicated bot accounts Tag with IAM policy 

GENERATING API KEY Admin permission needed Saving secrets is hard Keys have publicly shareable part 

ROTATING API KEYS Up to two keys per user Generate 2nd, deploy, invalidate old A er invalidation causes no problems, delete 

REGIONS AND AVAILABILITY ZONES Regions: geographical areas: Oregon (us-west-2) Availability zones: data centers (us-west-2a: depends on account) 

EC2 INSTANCES Virtual machines Basic "compute" unit Size, cost varies 

CLOUDINIT Python package Grabs instance metadata Configures machine: e.g. s s h - a u t h o r i z e d - k e y s 

KEYPAIR MANAGEMENT Generate pair on AWS Upload public Per-region name 

SECURING CONNECTION client = boto3.client('ec2', region_name='us-west-2') resource = boto3.resource('ec2', region_name='us-west-2') output = client.get_console_output( InstanceId='i-04376443919eb0f22') result = output['Output'] rsa = [line for line in result.splitlines() if line.startswith('ssh-rsa')][0] instance = resource.Instance('i-04376443919eb0f22') known_hosts = '{},{} {}\n'.format(instance.public_dns_name, instance.public_ip_address, rsa) with open(os.path.expanduser('~/.ssh/known_hosts'), 'a') as fp: fp.write(known_hosts) 

SECURING CONNECTION Last line of ~ / . s s h / k n o w n _ h o s t s , broken for readability. ec2-52-26-144-73.us-west-2.compute.amazonaws.com,52.26.144.73 ssh-rsa AAAA....S6vzCZG3gSh root@ip-172-31-18-116 

AMI BUILDS 

WHAT ARE AMIS? Amazon Machine Image Ready made ones Your own Can be shared across accounts Avoid secrets 

WHY BUILD YOUR OWN Security updates Specific so ware Configuration 

BUILDING AMIS WITH SSH $ ssh $USER@$IP sudo yum update -y $ python ... >>> client.create_image(....) 

BUILDING AMIS WITH SALT $ pex -o salt-call -c salt-call salt-ssh $ scp -r salt-call salt-files $USER@$IP:/ $ ssh $USER@$IP /salt-call --local --file-root /salt-files $ python ... >>> client.create_image(....) 

CLOUD FORMATION 

DEFINING CLOUD FORMATION TEMPLATES Templates are JSON files Can be parameterized Can be self-referential 

CLOUD FORMATION TEMPLATE EXAMPLE {"AWSTemplateFormatVersion" : "2010-09-09", "Parameters": {"KeyName":{"Type":"AWS::EC2::KeyPair::KeyName"}}, "Resources": {"EC2Instance": {"Type": "AWS::EC2::Instance", "Properties": {"InstanceType": {"t2.micro"}, "SecurityGroups": [{"Ref" : "InstanceSecurityGroup"}], "KeyName": {"Ref": "KeyName"}, "ImageId": {"ami-XXXXX" }}}, "InstanceSecurityGroup": {"Type": "AWS::EC2::SecurityGroup", "Properties": {"SecurityGroupIngress": [{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0" "ToPort": "22", "FromPort": "22"}]}}}, "Outputs": {"InstanceId": {"Value": {"Ref": "EC2Instance"}}}} 

DEFINING CLOUD FORMATION STACKS j s o n . d u m p s Ad-hoc creation of dicts/lists 

USING TROPOSPHERE from troposphere import Ref, Template, ec2 t = Template() inst = t.add_resource(ec2.Instance("inst", ImageId="ami-951945d0" InstanceType="t2.micro")) vol = template.add_resource(ec2.Volume("myvol, Size="8")) template.add_resource(ec2.VolumeAttachment("mtpt", InstanceId=Ref(inst), VolumeId=Ref(vol), Device="/dev/xdc")) t.add_resource(instance) cft = t.to_json() 

BUILDING A FRESH STACK client = boto3.client('cloudformation') client.create_stack("MyStack", TemplateBody=cft) in_progress = True while in_progress: desc = client.describe_stacks("MyStack") status = desc['Stacks'][0]['StackStatus'] in_progress = status == 'CREATE_IN_PROGRESS' time.sleep(1) print("Status: {}".format(status)) 

UPDATING A STACK New JSON Create a change-set Optional: eyeball check Apply a change-set 

VPC IN CLOUD FORMATION Separating stacks with VPCs Easy to duplicate -- staging, dev, etc. Create entire network (VPC, Subnets, Security groups, etc.) 

SUMMARY No root logins Cloud formation everything Automate everyting 

QUESTIONS? 

BONUS MATERIAL 

S3 BUCKETS 

WHAT IS A BUCKET? Collection of objects Globally unique name Objects map names to content and metadata 

UPLOADING SMALL FILES TO A BUCKET client = boto3.client('s3') client.upload_fileobj(BytesIO(b'important'), bucketname, 'important.html', ExtraArgs={'ContentType': 'text/html'}) 

UPLOADING LARGE FILES TO A BUCKET uploaded = functools.partial(print, "uploaded bytes:") client = boto3.client('s3') with open('pic.jpg') as fp: client.upload_fileobj(fp, bucketname, 'pic.jpg', ExtraArgs={'ContentType': 'image/jpeg'}, Callback=uploaded) 

SNAPSHOT MANAGEMENT 

TAKING SNAPSHOTS resource = boto3.resource('ec2', region_name='us-west-2') instance = resource.Instance('i-0c1df06045ea65840') volumes = list(instance.volumes.all()) sdb, = [volume for volume in volumes if volume.attachments[0]['Device']=='/dev/sdb'] result = sdb.create_snapshot(Description='important-snapshot') result.create_tags(Tags=[dict(Key='importance', Value='high')]) 

CREATING VOLUMES FROM SNAPSHOTS resource = boto3.resource('ec2', region_name='us-west-2') snapshot, = list(resource.snapshots.filter( Filters=[dict(Name='tag:importance', Values=['high'])])) mapping = [dict(DeviceName='/dev/sdb', Ebs=dict(SnapshotId=snapshot.id))] resource.create_instances(ImageId='ami-8ca83fec', KeyName='talk-us-west-2', InstanceType='t2.micro', BlockDeviceMappings=mapping, MinCount=1, MaxCount=1)