Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Moshe Zadka - Automate AWS With Python

Moshe Zadka - Automate AWS With Python

AWS is one of the best-known cloud vendors.
Using the Web UI is fine when starting out,
but automating cloud operations is important.
Boto3 provides a great Pythonic API to AWS,
but using it correctly can be subtle.

The talk will cover how to automate AMI builds,
building Cloud Formation Templates and
automating S3 bucket management.

https://us.pycon.org/2017/schedule/presentation/128/

PyCon 2017

May 21, 2017
Tweet

More Decks by PyCon 2017

Other Decks in Programming

Transcript

  1. ACCOUNT MANAGEMENT SPOILER Billing-only account No root access to other

    accounts: Production, Build, Staging, Demo, Dev.
  2. EXAMPLE: EVERYTHING EXCEPT IAM {"Version": "2012-10-17", "Statement": [ {"Effect": "Deny",

    "Action": "iam:*", "Resource": "*"}, {"Effect": "Allow", "Action": "*", "Resource": "*"}]}
  3. CORPORATE AWESOMENESS Only admin/finance have access to master account No

    resources in master account Production account for user-facing Staging account Build/Test account Pets account
  4. CORPORATE AWESOMENESS: POLICIES Prod/stage/build -- all resources automated, tagged with

    "reason" Pets -- all resources tagged with dev name, GC by default
  5. ROTATING API KEYS Up to two keys per user Generate

    2nd, deploy, invalidate old A er invalidation causes no problems, delete
  6. SECURING CONNECTION client = boto3.client('ec2', region_name='us-west-2') resource = boto3.resource('ec2', region_name='us-west-2')

    output = client.get_console_output( InstanceId='i-04376443919eb0f22') result = output['Output'] rsa = [line for line in result.splitlines() if line.startswith('ssh-rsa')][0] instance = resource.Instance('i-04376443919eb0f22') known_hosts = '{},{} {}\n'.format(instance.public_dns_name, instance.public_ip_address, rsa) with open(os.path.expanduser('~/.ssh/known_hosts'), 'a') as fp: fp.write(known_hosts)
  7. SECURING CONNECTION Last line of ~ / . s s

    h / k n o w n _ h o s t s , broken for readability. ec2-52-26-144-73.us-west-2.compute.amazonaws.com,52.26.144.73 ssh-rsa AAAA....S6vzCZG3gSh root@ip-172-31-18-116
  8. WHAT ARE AMIS? Amazon Machine Image Ready made ones Your

    own Can be shared across accounts Avoid secrets
  9. BUILDING AMIS WITH SSH $ ssh $USER@$IP sudo yum update

    -y $ python ... >>> client.create_image(....)
  10. BUILDING AMIS WITH SALT $ pex -o salt-call -c salt-call

    salt-ssh $ scp -r salt-call salt-files $USER@$IP:/ $ ssh $USER@$IP /salt-call --local --file-root /salt-files $ python ... >>> client.create_image(....)
  11. CLOUD FORMATION TEMPLATE EXAMPLE {"AWSTemplateFormatVersion" : "2010-09-09", "Parameters": {"KeyName":{"Type":"AWS::EC2::KeyPair::KeyName"}}, "Resources":

    {"EC2Instance": {"Type": "AWS::EC2::Instance", "Properties": {"InstanceType": {"t2.micro"}, "SecurityGroups": [{"Ref" : "InstanceSecurityGroup"}], "KeyName": {"Ref": "KeyName"}, "ImageId": {"ami-XXXXX" }}}, "InstanceSecurityGroup": {"Type": "AWS::EC2::SecurityGroup", "Properties": {"SecurityGroupIngress": [{ "IpProtocol": "tcp", "CidrIp": "0.0.0.0" "ToPort": "22", "FromPort": "22"}]}}}, "Outputs": {"InstanceId": {"Value": {"Ref": "EC2Instance"}}}}
  12. DEFINING CLOUD FORMATION STACKS j s o n . d

    u m p s Ad-hoc creation of dicts/lists
  13. USING TROPOSPHERE from troposphere import Ref, Template, ec2 t =

    Template() inst = t.add_resource(ec2.Instance("inst", ImageId="ami-951945d0" InstanceType="t2.micro")) vol = template.add_resource(ec2.Volume("myvol, Size="8")) template.add_resource(ec2.VolumeAttachment("mtpt", InstanceId=Ref(inst), VolumeId=Ref(vol), Device="/dev/xdc")) t.add_resource(instance) cft = t.to_json()
  14. BUILDING A FRESH STACK client = boto3.client('cloudformation') client.create_stack("MyStack", TemplateBody=cft) in_progress

    = True while in_progress: desc = client.describe_stacks("MyStack") status = desc['Stacks'][0]['StackStatus'] in_progress = status == 'CREATE_IN_PROGRESS' time.sleep(1) print("Status: {}".format(status))
  15. VPC IN CLOUD FORMATION Separating stacks with VPCs Easy to

    duplicate -- staging, dev, etc. Create entire network (VPC, Subnets, Security groups, etc.)
  16. WHAT IS A BUCKET? Collection of objects Globally unique name

    Objects map names to content and metadata
  17. UPLOADING LARGE FILES TO A BUCKET uploaded = functools.partial(print, "uploaded

    bytes:") client = boto3.client('s3') with open('pic.jpg') as fp: client.upload_fileobj(fp, bucketname, 'pic.jpg', ExtraArgs={'ContentType': 'image/jpeg'}, Callback=uploaded)
  18. TAKING SNAPSHOTS resource = boto3.resource('ec2', region_name='us-west-2') instance = resource.Instance('i-0c1df06045ea65840') volumes

    = list(instance.volumes.all()) sdb, = [volume for volume in volumes if volume.attachments[0]['Device']=='/dev/sdb'] result = sdb.create_snapshot(Description='important-snapshot') result.create_tags(Tags=[dict(Key='importance', Value='high')])
  19. CREATING VOLUMES FROM SNAPSHOTS resource = boto3.resource('ec2', region_name='us-west-2') snapshot, =

    list(resource.snapshots.filter( Filters=[dict(Name='tag:importance', Values=['high'])])) mapping = [dict(DeviceName='/dev/sdb', Ebs=dict(SnapshotId=snapshot.id))] resource.create_instances(ImageId='ami-8ca83fec', KeyName='talk-us-west-2', InstanceType='t2.micro', BlockDeviceMappings=mapping, MinCount=1, MaxCount=1)