Tweet
Tweet

Slide 1

Slide 1 text

MASTER & MINIONS MASTER & MINIONS OR THE DREAM OF PYTHON AUTOMATION OR THE DREAM OF PYTHON AUTOMATION CALVIN HENDRYX-PARKER, CTO CALVIN HENDRYX-PARKER, CTO SIX FEET UP SIX FEET UP

Slide 2

Slide 2 text

CERTAIN THINGS SHOULD BE HAND CERTAIN THINGS SHOULD BE HAND CRAFTED. CRAFTED.

Slide 3

Slide 3 text

BUT NOT YOUR INFRASTRUCTURE BUT NOT YOUR INFRASTRUCTURE

Slide 4

Slide 4 text

BEAUTIFUL UNIQUE SNOWFLAKES ARE BEAUTIFUL UNIQUE SNOWFLAKES ARE NOT REPRODUCIBLE. NOT REPRODUCIBLE.

Slide 5

Slide 5 text

PETS PETS

Slide 6

Slide 6 text

CATTLE CATTLE

Slide 7

Slide 7 text

RULES OF DEVOPS CLUB RULES OF DEVOPS CLUB » The rst rule of DevOps Club is: You do not log into servers » The second rule of DevOps Club is: You do not log into servers » Third rule of DevOps Club: If your deployment fails, rollback » Fourth rule: All artifacts will be stored in source control » Fifth rule: Only one deployment at a time » Sixth rule: No one offs, No special cases » Seventh rule: Deployments will go on as long as they have to » And the eighth and nal rule: If this is your rst night at DevOps Club, you have to push to prod.

Slide 8

Slide 8 text

TOOLS TOOLS To enter the DevOps world, you need to know what tools are available to you. Python has many great tools available to use such as SaltStack and the AWS Boto3 library.

Slide 9

Slide 9 text

FROM THE CLOSET TO THE CLOUD FROM THE CLOSET TO THE CLOUD

Slide 10

Slide 10 text

SINGLE SERVER MONOLITH SINGLE SERVER MONOLITH ┌────────────────────────────────────┐ │ Web/App/Database │ │ ┌───────────────────────────┐ │ │ │ Nginx │ │ │ └───────────────────────────┘ │ │ │ │ ┌───────────────────────────┐ │ │ │ Varnish │ │ │ └───────────────────────────┘ │ │ │ │ ┌───────────────────────────┐ │ │ │ HAProxy │ │ │ └───────────────────────────┘ │ │ │ │ ┌───────────────────────────┐ │ │ │ Plone Instances x 4 │ │ │ └───────────────────────────┘ │ │ │ │ ┌───────────────────────────┐ │ │ │ ZEO Storage Server │ │ │ └───────────────────────────┘ │ │ │ └────────────────────────────────────┘

Slide 11

Slide 11 text

CLOUD OPTIMIZED CLOUD OPTIMIZED

Slide 12

Slide 12 text

AUTOMATION AUTOMATION

Slide 13

Slide 13 text

ENTER SALTSTACK AND BOTO3 ENTER SALTSTACK AND BOTO3

Slide 14

Slide 14 text

NETWORK END GOAL NETWORK END GOAL ┌─────────────────────────────────────────┐ ┌────────────────────────────────────┐ │ control-vpc │ │ prod-vpc │ │ ┌───────────────────────────────────┐ │ │ ┌───────────────────────────┐ │ │ │ control-public-az1-subnet │ │ │ │ ├─┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ prod-public-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ ┌───────────────────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ Security │ └┬──────────────────────────┘ │ │ │ │ │ salt.esc.nd.edu │ │ │────Group ──────│ └────────────────────────────┘ │ │ │ │ │ │ │ Rules │ ┌────────────────────────────┐ │ │ │ │ Salt Master/VPN │ │ │ │ │ ├─┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───────────────────┘ │ │ │ │ prod-private-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───────────────────────────────────┘ │ │ └┬───────────────────────────┘ │ │ └─────────────────────────────────────────┘ │ └─────────────────────────────┘ │ │ └────────────────────────────────────┘ │ │ │ ┌────────────────────────────────────┐ │ │ test-vpc │ │ │ ┌───────────────────────────┐ │ │ │ │ ├─┐ │ │ │ │ │ │ │ │ │ │ test-public-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ Security │ └┬──────────────────────────┘ │ │ └──────────────Group ─────────────────│ └────────────────────────────┘ │ Rules │ ┌────────────────────────────┐ │ │ │ ├─┐ │ │ │ │ │ │ │ │ test-private-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ └┬───────────────────────────┘ │ │ │ └─────────────────────────────┘ │ └────────────────────────────────────┘

Slide 15

Slide 15 text

BIT OF A CHICKEN AND EGG PROBLEM BIT OF A CHICKEN AND EGG PROBLEM # VPC c = boto.vpc.connect_to_region(region) vpc = c.create_vpc('172.20.0.0/16') vpc.add_tag('Name', 'control-vpc') c.modify_vpc_attribute(vpc.id,enable_dns_hostnames=True)

Slide 16

Slide 16 text

TOOL TIP TOOL TIP KEEP YOUR SECRETS… SECRET KEEP YOUR SECRETS… SECRET $ lpass login # LastPass $ $(lpass-env export 'AWS SFU Calvin') $ eval $(op signin sixfeetup) # 1Password $ awsenv "AWS SFU Calvin" $ env | grep AWS AWS_ACCESS_KEY_ID=DEADBEEFCAFE AWS_SECRET_ACCESS_KEY=123jasdfadsOof9akayo5peey0cow AWS_DEFAULT_REGION=us-east-1

Slide 17

Slide 17 text

BOOTSTRAP CONTINUED BOOTSTRAP CONTINUED # EC2 data_path = os.path.join(os.path.dirname(__file__), 'bootstrap-master.sh') with open(data_path) as script: bootstrap = script.read() master = ec2.run_instances( 'ami-55ef662f', subnet_id=subnet.id, security_group_ids=[group.id], key_name=ssh_key, instance_type='t2.micro', user_data=bootstrap ) master.instances[0].add_tag('Name', 'Salt Master')

Slide 18

Slide 18 text

WHAT IS SALTSTACK? WHAT IS SALTSTACK?

Slide 19

Slide 19 text

WHAT SETS SALT APART? WHAT SETS SALT APART? » Remote Execution » Event-Driven Orchestration » Agent or Agent-less Operation » Cloud Provisioning » Speed and Scalability

Slide 20

Slide 20 text

LONELY MINIONS LONELY MINIONS

Slide 21

Slide 21 text

MASTER AND MINIONS MASTER AND MINIONS

Slide 22

Slide 22 text

NOW WE ORCHESTRATE NOW WE ORCHESTRATE And build a test environment $ salt-run state.orchestrate orch.deploy-environment pillarenv=prod $ salt-run state.orchestrate orch.deploy-environment pillarenv=test

Slide 23

Slide 23 text

INFRASTRUCTURE (SANS HUMANS) INFRASTRUCTURE (SANS HUMANS) ┌─────────────────────────────────────────┐ ┌────────────────────────────────────┐ │ control-vpc │ │ prod-vpc │ │ ┌───────────────────────────────────┐ │ │ ┌───────────────────────────┐ │ │ │ control-public-az1-subnet │ │ │ │ ├─┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ prod-public-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ ┌───────────────────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ Security │ └┬──────────────────────────┘ │ │ │ │ │ salt.esc.nd.edu │ │ │────Group ──────│ └────────────────────────────┘ │ │ │ │ │ │ │ Rules │ ┌────────────────────────────┐ │ │ │ │ Salt Master/VPN │ │ │ │ │ ├─┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───────────────────┘ │ │ │ │ prod-private-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───────────────────────────────────┘ │ │ └┬───────────────────────────┘ │ │ └─────────────────────────────────────────┘ │ └─────────────────────────────┘ │ │ └────────────────────────────────────┘ │ │ │ ┌────────────────────────────────────┐ │ │ test-vpc │ │ │ ┌───────────────────────────┐ │ │ │ │ ├─┐ │ │ │ │ │ │ │ │ │ │ test-public-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ Security │ └┬──────────────────────────┘ │ │ └──────────────Group ─────────────────│ └────────────────────────────┘ │ Rules │ ┌────────────────────────────┐ │ │ │ ├─┐ │ │ │ │ │ │ │ │ test-private-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ └┬───────────────────────────┘ │ │ │ └─────────────────────────────┘ │ └────────────────────────────────────┘

Slide 24

Slide 24 text

ORCHESTRATING NEW CODE RELEASES ORCHESTRATING NEW CODE RELEASES

Slide 25

Slide 25 text

ZERO DOWNTIME RELEASES ZERO DOWNTIME RELEASES {%- for app_server in app_servers %} ... # Stop Varnish, indirectly removing this app server from the NetScaler stop-{{ app_server }}-varnish: salt.function: - name: service.stop - arg: - {{ varnish.service }} - tgt: {{ app_server }} ...

Slide 26

Slide 26 text

PREPARE FOR THE RELEASE PREPARE FOR THE RELEASE ... # Stop Instances stop-{{ app_server }}-instances: salt.function: - name: supervisord.stop - arg: - all - tgt: {{ app_server }}

Slide 27

Slide 27 text

OPTIONAL AD-HOC CODE RELEASES OPTIONAL AD-HOC CODE RELEASES ... # Check out a specific revision {% set branch = salt['pillar.get']('plone:branch', 'dev') %} # to specify on the command-line: pillar='{"plone": {"branch": "f5d9859"}}' checkout-code-{{ app_server }}: salt.function: - tgt: {{ app_server }} - name: git.checkout - kwarg: user: webuser rev: {{ branch }} - arg: - {{ buildout_dir }} ...

Slide 28

Slide 28 text

HANDLE RELEASE TASKS HANDLE RELEASE TASKS # Run the buildout on the app server, but only run the generic setup on one server run-buildout-{{ app_server }}: salt.function: - tgt: {{ app_server }} - name: cmd.run - kwarg: cwd: {{ buildout_dir }} runas: webuser - arg: - {{ buildout_dir }}/env/bin/buildout -N

Slide 29

Slide 29 text

HANDLE POST RELEASE TASKS HANDLE POST RELEASE TASKS # Install Gulp bits install-gulp-{{ app_server }}: salt.function: - name: npm.install - tgt: {{ app_server }} - kwarg: runas: webuser dir: {{ theme_dir }} # Run the Gulp Build run-gulp-{{ app_server }}: salt.function: - tgt: {{ app_server }} - name: cmd.run - arg: - gulp build - kwarg:

Slide 30

Slide 30 text

NOT JUST FOR RELEASES NOT JUST FOR RELEASES » Migrating Production Data back to Testing » Scheduled tasks such as Database Backups

Slide 31

Slide 31 text

GRABBING PRODUCTION DATA GRABBING PRODUCTION DATA # Find the dev master db for use later {% set ns = namespace(dev_master=None) %} {% for dev_master in dev_db_servers if salt.saltutil.cmd(dev_master, 'file.search', arg=['/var/run/keepalive.state', 'MASTER']).get(dev_master).get('ret') %} {% if loop.first %} {% set ns.dev_master = dev_master %} {% endif %} {% endfor %} # Transfer Prod database {% for prod_slave in prod_db_servers if salt.saltutil.cmd(prod_slave, 'file.search', arg=['/var/run/keepalive.state', 'BACKUP']).get(prod_slave).get('ret') %} {% if loop.last %} sync prod database to dev:

Slide 32

Slide 32 text

SOUNDS TOO EASY SOUNDS TOO EASY The road was bumpy for sure. Satisfying the rules for no special cases was tricky.

Slide 33

Slide 33 text

MINDFULNESS MINDFULNESS There should be one, and preferably only one, obvious way to do it. Although that way may not be obvious at rst unless you’re Dutch. – Tim Peters, The Zen of Python

Slide 34

Slide 34 text

THE JOURNEY OPERATING SYSTEMS THE JOURNEY OPERATING SYSTEMS » 2a67758 Editing requirements to run properly on amazon linux » 472d844 Refactoring to run CentOS 7 machines » dd67b7a Refactoring for FreeBSD What happened here?

Slide 35

Slide 35 text

OTHER RECOMMENDATIONS OTHER RECOMMENDATIONS » Implement a CI strategy to test your infrastructure with tools like » Create or use tools to help you trace requests in the cloud Kitchen-Salt https://github.com/sixfeetup/aws-log-tools

Slide 36

Slide 36 text

LIVING THE DREAM LIVING THE DREAM

Slide 37

Slide 37 text

QUESTIONS? QUESTIONS? [email protected] [email protected] @calvinhp