Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Designing a Project Lifecycle using Contracts and Constraints

Designing a Project Lifecycle using Contracts and Constraints

Talk from Don Luchini at Boston DevOps meetup 3/23/2016

Boston DevOps

March 23, 2016
Tweet

More Decks by Boston DevOps

Other Decks in Technology

Transcript

  1. What We're Doing Roughly once per sprint, DevOps takes on

    a new project which puts code through the following CD pipeline: 1. Compile, unit test, and publish code. 2. Run Packer to create an image. – The image is provisioned using Chef. 3. Run CloudFormation to make a stack. 4. Run integration tests. Problem: every hour where no deployment exists equates to lost dev / QA productivity.
  2. Guiding Principles • Assume the defaults are sane. • Componentize

    everything. • Every deployment should look and feel consistent with every other deployment. – This is NOT “lowest common denominator”. • Users (devs) should be able to understand and write the deployment code.
  3. Problem: Packer templates are ugly { "builders": [ { "ami_name":

    "Packer myapp_development_32", "instance_type": "t2.micro", "launch_block_device_mappings": [{ "delete_on_termination": true, "device_name": "/dev/sda1", "volume_size": 8 }], "region": "us-east-1", "run_tags": { "build_number": "32", "chef_environment": "myapp-development", "date": "2016-03-03T18:40:36+00:00", "git_branch": "development", "git_commit": "3c87d6dfc1058ad3b08143fdce54f5bcd20adbde", "Name": "Packer myapp 32", "packer_project": "myapp" }, "security_group_id": "sg-8e06a7eb", "source_ami": "ami-57dbe53d", "ssh_username": "root", "subnet_id": "subnet-add89185", "tags": { "build_number": "32", "chef_environment": "myapp-development", "date": "2016-03-03T18:40:36+00:00", "git_branch": "development", "git_commit": "3c87d6dfc1058ad3b08143fdce54f5bcd20adbde", "Name": "Packer myapp development 32", "packer_project": "myapp" }, "type": "amazon-ebs", "vpc_id": "vpc-8f4db9ea", "name": "amazon-ebs" } ], "provisioners": [ { "inline": "mkdir -p /tmp/packer-chef-client/trusted_certs", "type": "shell" }, { "destination": "/tmp/packer-chef-client/trusted_certs/enernoc_root_ca.pem", "source": "/etc/pki/ca-trust/source/anchors/enernoc_root_ca.pem", "type": "file" }, { "execute_command": "chef-client --no-color -c {{.ConfigPath}} -j {{.JsonPath}}", "install_command": "if [ $(rpm -q chef | grep -c \\\"is not installed\\\" ) -gt 0 ]; then rpm -Uvh https://artifactory.release.enernoc.net/centos-6chef-12.4.1-1.el6.x86_64.rpm; fi && rm /etc/chef/client.pem | true", "node_name": "packer_myapp_development_32", "prevent_sudo": true, "server_url": "https://chef.infra.enoc.cc/organizations/enoc", "type": "chef-client", "validation_client_name": "enoc-validator", "validation_key_path": "/var/lib/jenkins/validator.pem", "chef_environment": "myapp-development", "run_list": [ "my_cookbook" ]
  4. Solution: templatize them! require 'base64' require 'json' require 'net/http' userdata

    = Base64.encode64( JSON.parse( Net::HTTP.get('security.service.local', '/api/v1/token') ).token ) Racker::Processor.register_template do |t| t.builders['amazon-ebs']['source_ami'] = 'ami-xxxxxxxx' t.builders['amazon-ebs']['user_data'] = userdata t.provisioners[100]['chef-client']['run_list'] = ['my_cookbook'] end
  5. Deploying Code include_recipe 'wildfly' wildfly_deploy 'my application' do artifact_id 'myapp'

    group_id 'org.my' version '1.0.0' end wildfly_logger 'org.apache' do level 'DEBUG' end wildfly_datasource 'myDS' do username 'user' password 'user' url 'jdbc:postgresql://some-server/some-database' type 'XA' end wildfly_splunk 'logs' do index 'myapp'
  6. Deploying Code include_recipe 'nodejs' nodejs_deploy 'my node app' do app_start

    'start_server.js' artifact_id 'mynodeapp' group_id 'org.my' version '1.0.0' end
  7. Problem: CF templates are ugly {"AWSTemplateFormatVersion":"2010-09-09","Description":"Defines the full stack for

    the Stampede development environment.","Resources":{"SecurityGroup":{"Type":"AWS::EC2::SecurityGroup","Properties":{"GroupDescription":"Enable HTTP access via port 80 locked down to the load balancer + SSH access","SecurityGroupIngress":[{"IpProtocol":"tcp","FromPort":"80","ToPort":"80","CidrIp":"10.0.0.0/8"},{"IpProtocol":"tcp","FromPort":"80","ToPort":"80","CidrIp":"172.16.0.0/12"},{"IpProtocol":"tcp","FromPort":"22","ToPort":"22","CidrIp":"10.0.0.0/8"}, {"IpProtocol":"tcp","FromPort":"22","ToPort":"22","CidrIp":"172.16.0.0/12"},{"IpProtocol":"tcp","FromPort":"6379","ToPort":"6379","CidrIp":"172.16.0.0/12"},{"IpProtocol":"tcp","FromPort":"6379","ToPort":"6379","CidrIp":"10.0.0.0/8"}, {"IpProtocol":"tcp","FromPort":"8080","ToPort":"8080","CidrIp":"172.16.0.0/12"},{"IpProtocol":"tcp","FromPort":"8080","ToPort":"8080","CidrIp":"10.0.0.0/8"}],"VpcId":"vpc-8f4db9ea"}},"CacheRefresherELB":{"Type":"AWS::ElasticLoadBalancing::LoadBalancer","Properties": {"ConnectionSettings":{"IdleTimeout":25},"Listeners":[{"LoadBalancerPort":"80","InstancePort":"8080","Protocol":"HTTP"}],"HealthCheck":{"Target":"HTTP:8080/ping","HealthyThreshold":"2","UnhealthyThreshold":"5","Interval":"10","Timeout":"5"},"Scheme":"internal","SecurityGroups":["sg- 8e06a7eb"],"Subnets":["subnet-5f0b2319"]}},"ConfigELB":{"Type":"AWS::ElasticLoadBalancing::LoadBalancer","Properties":{"ConnectionSettings":{"IdleTimeout":25},"Listeners":[{"LoadBalancerPort":"80","InstancePort":"8080","Protocol":"HTTP"}],"HealthCheck": {"Target":"HTTP:8080/ping","HealthyThreshold":"2","UnhealthyThreshold":"5","Interval":"10","Timeout":"5"},"Scheme":"internal","SecurityGroups":["sg-8e06a7eb"],"Subnets":["subnet-5f0b2319"]}},"ConsumerELB":{"Type":"AWS::ElasticLoadBalancing::LoadBalancer","Properties": {"ConnectionSettings":{"IdleTimeout":25},"Listeners":[{"LoadBalancerPort":"80","InstancePort":"8080","Protocol":"HTTP"}],"HealthCheck":{"Target":"HTTP:8080/ping","HealthyThreshold":"2","UnhealthyThreshold":"5","Interval":"10","Timeout":"5"},"Scheme":"internal","SecurityGroups":["sg- 8e06a7eb"],"Subnets":["subnet-5f0b2319"]}},"TelemetryPullELB":{"Type":"AWS::ElasticLoadBalancing::LoadBalancer","Properties":{"ConnectionSettings":{"IdleTimeout":25},"Listeners":[{"LoadBalancerPort":"80","InstancePort":"8080","Protocol":"HTTP"}],"HealthCheck": {"Target":"HTTP:8080/ping","HealthyThreshold":"2","UnhealthyThreshold":"5","Interval":"10","Timeout":"5"},"Scheme":"internal","SecurityGroups":["sg-8e06a7eb"],"Subnets":["subnet-5f0b2319"]}},"TelemetryPushELB":{"Type":"AWS::ElasticLoadBalancing::LoadBalancer","Properties": {"ConnectionSettings":{"IdleTimeout":25},"Listeners":[{"LoadBalancerPort":"80","InstancePort":"8080","Protocol":"HTTP"}],"HealthCheck":{"Target":"HTTP:8080/ping","HealthyThreshold":"2","UnhealthyThreshold":"5","Interval":"10","Timeout":"5"},"Scheme":"internal","SecurityGroups":["sg- 8e06a7eb"],"Subnets":["subnet-5f0b2319"]}},"UIELB":{"Type":"AWS::ElasticLoadBalancing::LoadBalancer","Properties":{"ConnectionSettings":{"IdleTimeout":25},"Listeners":[{"LoadBalancerPort":"80","InstancePort":"8080","Protocol":"HTTP"}],"HealthCheck": {"Target":"HTTP:8080/ping","HealthyThreshold":"2","UnhealthyThreshold":"5","Interval":"10","Timeout":"5"},"Scheme":"internal","SecurityGroups":["sg-8e06a7eb"],"Subnets":["subnet-5f0b2319"]}},"XMPPELB":{"Type":"AWS::ElasticLoadBalancing::LoadBalancer","Properties": {"ConnectionSettings":{"IdleTimeout":25},"Listeners":[{"LoadBalancerPort":"80","InstancePort":"8080","Protocol":"HTTP"}],"HealthCheck":{"Target":"HTTP:8080/ping","HealthyThreshold":"2","UnhealthyThreshold":"5","Interval":"10","Timeout":"5"},"Scheme":"internal","SecurityGroups":["sg- 8e06a7eb"],"Subnets":["subnet-5f0b2319"]}},"AdamsLaunchConfig":{"Type":"AWS::AutoScaling::LaunchConfiguration","Properties":{"ImageId":"ami-a94981c2","InstanceType":"t2.micro","SecurityGroups":[{"Ref":"SecurityGroup"}],"UserData": {"Fn::Base64":"adamsworker"},"KeyName":"RETeam"}},"CacheRefresherLaunchConfig":{"Type":"AWS::AutoScaling::LaunchConfiguration","Properties":{"ImageId":"ami-a94981c2","InstanceType":"t2.micro","SecurityGroups":[{"Ref":"SecurityGroup"}],"UserData":{"Fn::Base64":"stampede- cache-refresher"},"KeyName":"RETeam"}},"ConfigLaunchConfig":{"Type":"AWS::AutoScaling::LaunchConfiguration","Properties":{"ImageId":"ami-a94981c2","InstanceType":"t2.micro","SecurityGroups":[{"Ref":"SecurityGroup"}],"UserData":{"Fn::Base64":"stampede- config"},"KeyName":"RETeam"}},"ConsumerLaunchConfig":{"Type":"AWS::AutoScaling::LaunchConfiguration","Properties":{"ImageId":"ami-a94981c2","InstanceType":"t2.small","SecurityGroups":[{"Ref":"SecurityGroup"}],"UserData":{"Fn::Base64":"stampede- consumer"},"KeyName":"RETeam"}},"TelemetryPullLaunchConfig":{"Type":"AWS::AutoScaling::LaunchConfiguration","Properties":{"ImageId":"ami-a94981c2","InstanceType":"t2.micro","SecurityGroups":[{"Ref":"SecurityGroup"}],"UserData":{"Fn::Base64":"stampede-telemetry- pull"},"KeyName":"RETeam"}},"TelemetryPushLaunchConfig":{"Type":"AWS::AutoScaling::LaunchConfiguration","Properties":{"ImageId":"ami-a94981c2","InstanceType":"t2.micro","SecurityGroups":[{"Ref":"SecurityGroup"}],"UserData":{"Fn::Base64":"stampede-telemetry- push"},"KeyName":"RETeam"}},"UILaunchConfig":{"Type":"AWS::AutoScaling::LaunchConfiguration","Properties":{"ImageId":"ami-a94981c2","InstanceType":"t2.micro","SecurityGroups":[{"Ref":"SecurityGroup"}],"UserData": {"Fn::Base64":"ui"},"KeyName":"RETeam"}},"XMPPLaunchConfig":{"Type":"AWS::AutoScaling::LaunchConfiguration","Properties":{"ImageId":"ami-a94981c2","InstanceType":"t2.micro","SecurityGroups":[{"Ref":"SecurityGroup"}],"UserData":{"Fn::Base64":"stampede- xmpp"},"KeyName":"RETeam"}},"AdamsASG":{"Type":"AWS::AutoScaling::AutoScalingGroup","Properties":{"AvailabilityZones":["us-east-1d"],"HealthCheckGracePeriod":180,"LaunchConfigurationName": {"Ref":"AdamsLaunchConfig"},"MinSize":"1","MaxSize":"2","HealthCheckType":"EC2","Tags":[{"Key":"Name","Value":"stampede-dev-adams","PropagateAtLaunch":true}],"VPCZoneIdentifier":["subnet-5f0b2319"]},"UpdatePolicy":{"AutoScalingRollingUpdate": {"MaxBatchSize":1,"MinInstancesInService":0,"PauseTime":"PT0S","WaitOnResourceSignals":false}}},"CacheRefresherASG":{"Type":"AWS::AutoScaling::AutoScalingGroup","Properties":{"AvailabilityZones":["us-east-1d"],"HealthCheckGracePeriod":180,"LaunchConfigurationName": {"Ref":"CacheRefresherLaunchConfig"},"MinSize":"1","MaxSize":"2","HealthCheckType":"ELB","LoadBalancerNames":[{"Ref":"CacheRefresherELB"}],"Tags":[{"Key":"Name","Value":"stampede-dev-cacherefresher","PropagateAtLaunch":true}],"VPCZoneIdentifier":["subnet- 5f0b2319"]},"UpdatePolicy":{"AutoScalingRollingUpdate":{"MaxBatchSize":1,"MinInstancesInService":0,"PauseTime":"PT0S","WaitOnResourceSignals":false}}},"ConfigASG":{"Type":"AWS::AutoScaling::AutoScalingGroup","Properties":{"AvailabilityZones":["us-east- 1d"],"HealthCheckGracePeriod":180,"LaunchConfigurationName":{"Ref":"ConfigLaunchConfig"},"MinSize":"1","MaxSize":"2","HealthCheckType":"ELB","LoadBalancerNames":[{"Ref":"ConfigELB"}],"Tags":[{"Key":"Name","Value":"stampede-dev- config","PropagateAtLaunch":true}],"VPCZoneIdentifier":["subnet-5f0b2319"]},"UpdatePolicy":{"AutoScalingRollingUpdate":{"MaxBatchSize":1,"MinInstancesInService":0,"PauseTime":"PT0S","WaitOnResourceSignals":false}}},"ConsumerASG": {"Type":"AWS::AutoScaling::AutoScalingGroup","Properties":{"AvailabilityZones":["us-east-1d"],"HealthCheckGracePeriod":180,"LaunchConfigurationName":{"Ref":"ConsumerLaunchConfig"},"MinSize":"1","MaxSize":"2","HealthCheckType":"ELB","LoadBalancerNames": [{"Ref":"ConsumerELB"}],"Tags":[{"Key":"Name","Value":"stampede-dev-consumer","PropagateAtLaunch":true}],"VPCZoneIdentifier":["subnet-5f0b2319"]},"UpdatePolicy":{"AutoScalingRollingUpdate": {"MaxBatchSize":1,"MinInstancesInService":0,"PauseTime":"PT0S","WaitOnResourceSignals":false}}},"TelemetryPullASG":{"Type":"AWS::AutoScaling::AutoScalingGroup","Properties":{"AvailabilityZones":["us-east-1d"],"HealthCheckGracePeriod":180,"LaunchConfigurationName": {"Ref":"TelemetryPullLaunchConfig"},"MinSize":"1","MaxSize":"2","HealthCheckType":"ELB","LoadBalancerNames":[{"Ref":"TelemetryPullELB"}],"Tags":[{"Key":"Name","Value":"stampede-dev-telemetrypull","PropagateAtLaunch":true}],"VPCZoneIdentifier":["subnet- 5f0b2319"]},"UpdatePolicy":{"AutoScalingRollingUpdate":{"MaxBatchSize":1,"MinInstancesInService":0,"PauseTime":"PT0S","WaitOnResourceSignals":false}}},"TelemetryPushASG":{"Type":"AWS::AutoScaling::AutoScalingGroup","Properties":{"AvailabilityZones":["us-east- 1d"],"HealthCheckGracePeriod":180,"LaunchConfigurationName":{"Ref":"TelemetryPushLaunchConfig"},"MinSize":"1","MaxSize":"2","HealthCheckType":"ELB","LoadBalancerNames":[{"Ref":"TelemetryPushELB"}],"Tags":[{"Key":"Name","Value":"stampede-dev- telemetrypush","PropagateAtLaunch":true}],"VPCZoneIdentifier":["subnet-5f0b2319"]},"UpdatePolicy":{"AutoScalingRollingUpdate":{"MaxBatchSize":1,"MinInstancesInService":0,"PauseTime":"PT0S","WaitOnResourceSignals":false}}},"UIASG": {"Type":"AWS::AutoScaling::AutoScalingGroup","Properties":{"AvailabilityZones":["us-east-1d"],"HealthCheckGracePeriod":180,"LaunchConfigurationName":{"Ref":"UILaunchConfig"},"MinSize":"1","MaxSize":"2","HealthCheckType":"ELB","LoadBalancerNames":[{"Ref":"UIELB"}],"Tags": [{"Key":"Name","Value":"stampede-dev-ui","PropagateAtLaunch":true}],"VPCZoneIdentifier":["subnet-5f0b2319"]},"UpdatePolicy":{"AutoScalingRollingUpdate":{"MaxBatchSize":1,"MinInstancesInService":0,"PauseTime":"PT0S","WaitOnResourceSignals":false}}},"XMPPASG": {"Type":"AWS::AutoScaling::AutoScalingGroup","Properties":{"AvailabilityZones":["us-east-1d"],"HealthCheckGracePeriod":180,"LaunchConfigurationName":{"Ref":"XMPPLaunchConfig"},"MinSize":"1","MaxSize":"2","HealthCheckType":"ELB","LoadBalancerNames": [{"Ref":"XMPPELB"}],"Tags":[{"Key":"Name","Value":"stampede-dev-xmpp","PropagateAtLaunch":true}],"VPCZoneIdentifier":["subnet-5f0b2319"]},"UpdatePolicy":{"AutoScalingRollingUpdate": {"MaxBatchSize":1,"MinInstancesInService":0,"PauseTime":"PT0S","WaitOnResourceSignals":false}}},"CacheRefresherDNS":{"Type":"AWS::Route53::RecordSetGroup","Properties":{"HostedZoneName":"enernoc.net.","RecordSets": [{"Name":"cacherefresher.development.stampede.enernoc.net.","Type":"A","AliasTarget":{"HostedZoneId":{"Fn::GetAtt":["CacheRefresherELB","CanonicalHostedZoneNameID"]},"DNSName":{"Fn::GetAtt":["CacheRefresherELB","DNSName"]}}}]}},"ConfigDNS": {"Type":"AWS::Route53::RecordSetGroup","Properties":{"HostedZoneName":"enernoc.net.","RecordSets":[{"Name":"config.development.stampede.enernoc.net.","Type":"A","AliasTarget":{"HostedZoneId":{"Fn::GetAtt":["ConfigELB","CanonicalHostedZoneNameID"]},"DNSName": {"Fn::GetAtt":["ConfigELB","DNSName"]}}}]}},"ConsumerDNS":{"Type":"AWS::Route53::RecordSetGroup","Properties":{"HostedZoneName":"enernoc.net.","RecordSets":[{"Name":"consumer.development.stampede.enernoc.net.","Type":"A","AliasTarget":{"HostedZoneId":{"Fn::GetAtt": ["ConsumerELB","CanonicalHostedZoneNameID"]},"DNSName":{"Fn::GetAtt":["ConsumerELB","DNSName"]}}}]}},"RedisDNS":{"Type":"AWS::Route53::RecordSetGroup","Properties":{"HostedZoneName":"enernoc.net.","RecordSets": [{"Name":"redis.development.stampede.enernoc.net","Type":"CNAME","TTL":"300","ResourceRecords":["stampede-dev.ahiuaq.0001.use1.cache.amazonaws.com"]}]}},"TelemetryPullDNS":{"Type":"AWS::Route53::RecordSetGroup","Properties": {"HostedZoneName":"enernoc.net.","RecordSets":[{"Name":"telemetrypull.development.stampede.enernoc.net.","Type":"A","AliasTarget":{"HostedZoneId":{"Fn::GetAtt":["TelemetryPullELB","CanonicalHostedZoneNameID"]},"DNSName":{"Fn::GetAtt": ["TelemetryPullELB","DNSName"]}}}]}},"TelemetryPushDNS":{"Type":"AWS::Route53::RecordSetGroup","Properties":{"HostedZoneName":"enernoc.net.","RecordSets":[{"Name":"telemetrypush.development.stampede.enernoc.net.","Type":"A","AliasTarget":{"HostedZoneId":{"Fn::GetAtt": ["TelemetryPushELB","CanonicalHostedZoneNameID"]},"DNSName":{"Fn::GetAtt":["TelemetryPushELB","DNSName"]}}}]}},"UIDNS":{"Type":"AWS::Route53::RecordSetGroup","Properties":{"HostedZoneName":"enernoc.net.","RecordSets": [{"Name":"ui.development.stampede.enernoc.net.","Type":"A","AliasTarget":{"HostedZoneId":{"Fn::GetAtt":["UIELB","CanonicalHostedZoneNameID"]},"DNSName":{"Fn::GetAtt":["UIELB","DNSName"]}}}]}},"XMPPDNS":{"Type":"AWS::Route53::RecordSetGroup","Properties": {"HostedZoneName":"enernoc.net.","RecordSets":[{"Name":"xmpp.development.stampede.enernoc.net.","Type":"A","AliasTarget":{"HostedZoneId":{"Fn::GetAtt":["XMPPELB","CanonicalHostedZoneNameID"]},"DNSName":{"Fn::GetAtt":["XMPPELB","DNSName"]}}}]}}},"Outputs": {"WebsiteURL":{"Value":"http://ui.development.stampede.enernoc.net","Description":"Stampede UI"}}}
  8. Solution: use more of them { "Type" : "AWS::CloudFormation::Stack", "Properties"

    : { "Parameters": { "AMI": { "Ref": "AMI" }, "Branch": { "Ref": "Branch" }, "FriendlyName": "My Cloud Application", "ProjectBaseURL": "my.org", “ServiceName": “myapp" }, "TemplateURL": “https://s3.amazonaws.com/b/universal-container-1.json” } }
  9. Solution: use more of them { "Type" : "AWS::CloudFormation::Stack", "Properties"

    : { "Parameters": { "AMI": { "Ref": "AMI" }, "Branch": { "Ref": "Branch" }, "FriendlyName": "My Cloud Application", "HealthCheckEndpoint": "/api/v1/ping", "ProjectBaseURL": "my.org", “ServiceName": “myapp", }, "TemplateURL": “https://s3.amazonaws.com/b/universal-container-1.json" } }
  10. Solution: use more of them { "Type" : "AWS::CloudFormation::Stack", "Properties"

    : { "Parameters": { "AMI": { "Ref": "AMI" }, "Branch": { "Ref": "Branch" }, "FriendlyName": "My Cloud Application", "InstancePort": 8080, "HealthCheckEndpoint": "/api/v1/ping", "ProjectBaseURL": "my.org", “ServiceName": “myapp", }, "TemplateURL": “https://s3.amazonaws.com/b/universal-container-1.json" } }
  11. Solution: use more of them { "Type" : "AWS::CloudFormation::Stack", "Properties"

    : { "Parameters": { "AMI": { "Ref": "AMI" }, "Branch": { "Ref": "Branch" }, "FriendlyName": "My Cloud Application", "Instances": "4", "InstancePort": 8080, "InstanceType": "c4.large", "HealthCheckEndpoint": "/api/v1/ping", "MaxInstances": "8", "ProjectBaseURL": "my.org", “ServiceName": “myapp", }, "TemplateURL": “https://s3.amazonaws.com/b/universal-container-1.json" } }
  12. Helpful Links • Chef: https://docs.chef.io • CloudFormation: https://aws.amazon.com/documentation/cloudformation/ • Grunt:

    http://gruntjs.com • Maven: https://maven.apache.org/ • Packer: https://packer.io • Racker: https://github.com/aspring/racker • Rake: https://github.com/ruby/rake • Don L.: [email protected]