Moshe Zadka - Automate AWS With Python

AWS is one of the best-known cloud vendors.
Using the Web UI is fine when starting out,
but automating cloud operations is important.
Boto3 provides a great Pythonic API to AWS,
but using it correctly can be subtle.

The talk will cover how to automate AMI builds,
building Cloud Formation Templates and
automating S3 bucket management.

https://us.pycon.org/2017/schedule/presentation/128/

Transcript

1. AUTOMATING AWS
2. WHY AUTOMATE? Reproducibility Functionality Treat UI as read-only
3. ACCOUNT MANAGEMENT SPOILER Billing-only account No root access to other accounts: Production, Build, Staging, Demo, Dev.
4. ROOT ACCOUNT Synonomous with AWS account.
5. IAM ACCOUNT Belongs to an AWS account Cannot access billing Follows security policy
6. IAM PASSWORD Optional Can be used with 2FA For human users
7. API KEY Optional Can have 0-2 For robots users
8. POLICIES Root can do anything "Admin" is almost anything Infinitely customizable
9. EXAMPLE: EVERYTHING EXCEPT IAM {"Version": "2012-10-17", "Statement": [ {"Effect": "Deny", "Action": "iam:*", "Resource": "*"}, {"Effect": "Allow", "Action": "*", "Resource": "*"}]}
10. CONSOLIDATED BILLING Create master account Create sub-accounts (It used to be harder) Bill easy to separate out
11. CORPORATE AWESOMENESS Only admin/finance have access to master account No resources in master account Production account for user-facing Staging account Build/Test account Pets account
12. CORPORATE AWESOMENESS: POLICIES Prod/stage/build -- all resources automated, tagged with "reason" Pets -- all resources tagged with dev name, GC by default
13. BOTO3 BASICS
14. AMAZON ACCOUNTS Dedicated bot accounts Tag with IAM policy
15. GENERATING API KEY Admin permission needed Saving secrets is hard Keys have publicly shareable part
16. ROTATING API KEYS Up to two keys per user Generate 2nd, deploy, invalidate old A er invalidation causes no problems, delete
17. REGIONS AND AVAILABILITY ZONES Regions: geographical areas: Oregon (us-west-2) Availability zones: data centers (us-west-2a: depends on account)
18. EC2 INSTANCES Virtual machines Basic "compute" unit Size, cost varies
19. CLOUDINIT Python package Grabs instance metadata Configures machine: e.g. s s h - a u t h o r i z e d - k e y s
20. KEYPAIR MANAGEMENT Generate pair on AWS Upload public Per-region name
21. SECURING CONNECTION client = boto3.client('ec2', region_name='us-west-2') resource = boto3.resource('ec2', region_name='us-west-2') output = client.get_console_output( InstanceId='i-04376443919eb0f22') result = output['Output'] rsa = [line for line in result.splitlines() if line.startswith('ssh-rsa')][0] instance = resource.Instance('i-04376443919eb0f22') known_hosts = '{},{} {}\n'.format(instance.public_dns_name, instance.public_ip_address, rsa) with open(os.path.expanduser('~/.ssh/known_hosts'), 'a') as fp: fp.write(known_hosts)
22. SECURING CONNECTION Last line of ~ / . s s h / k n o w n _ h o s t s , broken for readability. ec2-52-26-144-73.us-west-2.compute.amazonaws.com,52.26.144.73 ssh-rsa AAAA....S6vzCZG3gSh root@ip-172-31-18-116
23. AMI BUILDS
24. WHAT ARE AMIS? Amazon Machine Image Ready made ones Your own Can be shared across accounts Avoid secrets
25. WHY BUILD YOUR OWN Security updates Specific so ware Configuration
26. BUILDING AMIS WITH SSH $ ssh $USER@$IP sudo yum update -y $ python ... >>> client.create_image(....)
27. BUILDING AMIS WITH SALT $ pex -o salt-call -c salt-call salt-ssh $ scp -r salt-call salt-files $USER@$IP:/ $ ssh $USER@$IP /salt-call --local --file-root /salt-files $ python ... >>> client.create_image(....)
28. CLOUD FORMATION
29. DEFINING CLOUD FORMATION TEMPLATES Templates are JSON files Can be parameterized Can be self-referential
30. CLOUD FORMATION TEMPLATE EXAMPLE {"AWSTemplateFormatVersion" : "2010-09-09", "Parameters": {"KeyName":{"Type":"AWS::EC2::KeyPair::KeyName"}}, "Resources": {"EC2Instance": {"Type": "AWS::EC2::Instance", "Properties": {"InstanceType": {"t2.micro"}, "SecurityGroups": [{"Ref" : "InstanceSecurityGroup"}], "KeyName": {"Ref": "KeyName"}, "ImageId": {"ami-XXXXX" }}}, "InstanceSecurityGroup": {"Type": "AWS::EC2::SecurityGroup", "Properties": {"SecurityGroupIngress": [{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0" "ToPort": "22", "FromPort": "22"}]}}}, "Outputs": {"InstanceId": {"Value": {"Ref": "EC2Instance"}}}}
31. DEFINING CLOUD FORMATION STACKS j s o n . d u m p s Ad-hoc creation of dicts/lists
32. USING TROPOSPHERE from troposphere import Ref, Template, ec2 t = Template() inst = t.add_resource(ec2.Instance("inst", ImageId="ami-951945d0" InstanceType="t2.micro")) vol = template.add_resource(ec2.Volume("myvol, Size="8")) template.add_resource(ec2.VolumeAttachment("mtpt", InstanceId=Ref(inst), VolumeId=Ref(vol), Device="/dev/xdc")) t.add_resource(instance) cft = t.to_json()
33. BUILDING A FRESH STACK client = boto3.client('cloudformation') client.create_stack("MyStack", TemplateBody=cft) in_progress = True while in_progress: desc = client.describe_stacks("MyStack") status = desc['Stacks'][0]['StackStatus'] in_progress = status == 'CREATE_IN_PROGRESS' time.sleep(1) print("Status: {}".format(status))
34. UPDATING A STACK New JSON Create a change-set Optional: eyeball check Apply a change-set
35. VPC IN CLOUD FORMATION Separating stacks with VPCs Easy to duplicate -- staging, dev, etc. Create entire network (VPC, Subnets, Security groups, etc.)
36. SUMMARY No root logins Cloud formation everything Automate everyting
37. QUESTIONS?
38. BONUS MATERIAL
39. S3 BUCKETS
40. WHAT IS A BUCKET? Collection of objects Globally unique name Objects map names to content and metadata
41. UPLOADING SMALL FILES TO A BUCKET client = boto3.client('s3') client.upload_fileobj(BytesIO(b'<em>important</em>'), bucketname, 'important.html', ExtraArgs={'ContentType': 'text/html'})
42. UPLOADING LARGE FILES TO A BUCKET uploaded = functools.partial(print, "uploaded bytes:") client = boto3.client('s3') with open('pic.jpg') as fp: client.upload_fileobj(fp, bucketname, 'pic.jpg', ExtraArgs={'ContentType': 'image/jpeg'}, Callback=uploaded)
43. SNAPSHOT MANAGEMENT
44. TAKING SNAPSHOTS resource = boto3.resource('ec2', region_name='us-west-2') instance = resource.Instance('i-0c1df06045ea65840') volumes = list(instance.volumes.all()) sdb, = [volume for volume in volumes if volume.attachments[0]['Device']=='/dev/sdb'] result = sdb.create_snapshot(Description='important-snapshot') result.create_tags(Tags=[dict(Key='importance', Value='high')])
45. CREATING VOLUMES FROM SNAPSHOTS resource = boto3.resource('ec2', region_name='us-west-2') snapshot, = list(resource.snapshots.filter( Filters=[dict(Name='tag:importance', Values=['high'])])) mapping = [dict(DeviceName='/dev/sdb', Ebs=dict(SnapshotId=snapshot.id))] resource.create_instances(ImageId='ami-8ca83fec', KeyName='talk-us-west-2', InstanceType='t2.micro', BlockDeviceMappings=mapping, MinCount=1, MaxCount=1)