CodeMesh 2015 : Boot my (secure)->(portable) clouds!

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Boot my (secure) —> (portable) clouds! Nicki Watt
@techiewatt 04/11/2015 1

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2 About Me • Hands on Lead consultant
at OpenCredo  • Co-author Neo4j In Action  • Currently working with UK gov dept on cloud automation project  • Twitter: @techiewatt  

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3 Agenda • What is the problem  •
What are the options  • How: Principles, challenges, lessons, tools • Conclusion  

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4 What problem are we trying to address?

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 6 Act 1 : “Take advantage of cloud
computing”

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 7 Act 2 : “Efficiently Take advantage of
more cloud computing”

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 8

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 9

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10 How to create fast, repeatable, secure environments
capable of running in diﬀerent clouds!

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 11 • Public cloud workloads • Dev teams
anywhere in the world  • No human intervention Initial focus:

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 12 An example requirement: Team1 needs a development
CI/CD env

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 13 Input: Environment prefix: team1 Number Jenkins Slaves:
3 Environment domain suffix: t1tools.domain.io Initial SSH keys: AAAEFF user1@team1 Cloud: AWS

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 14

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 15 What are our options?

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 16 you need our cloud management platform !

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 17 vs

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 18

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 19 Principles for success

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 20 • Automate everything  • Separate config from
code • API driven clouds & tools • Prefer modular, open source tools  ASAP

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 25 • Self service functionality • Automated environment
creation   (under the hood) functionality

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 26 Challenges Lessons Tools

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ^ 27 • Automated IaaS Provisioning • Automated
Config Management • Securing stuff • Moving stuff Bootstrap

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 28 Challenge #1 Automated IaaS Provisioning

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 30 Availability Zone #1 Management subnet 10.0.0.32/24 Tooling
subnet 10.0.0.64/27 DNS VPN XXX Jenkins Master Jira Jenkins Slave-n Artifac tory Router Internet Gateway region DMZ subnet 10.0.0.0/27 • Networks • Firewall Rules • Routers • Compute   Resources • Public / Floating   IP Addresses

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 31 Lesson #1 There is NO single common
cloud API

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 32 Tool #1: Terraform Automated IaaS Provisioning

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 33 Creates, manages, and manipulates infrastructure resources.

subnet 10.0.0.64/27 DNS VPN XXX Jenkins Master Jira Jenkins Slave-n Artifac tory Router Internet Gateway region DMZ subnet 10.0.0.0/27 Multiple Cloud Providers

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 35 Declarative DSL ## OpenVPN Compute instance resource
"openstack_compute_instance_v2" "ovpn" { name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } ## OpenVPN Public IP resource "openstack_compute_floatingip_v2" "openvpn" { region = "" pool = "${var.public-ip-pool}" ... } terraform.tf

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 36 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn"
{ name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } ## OpenVPN Public IP resource "openstack_compute_floatingip_v2" "openvpn" { region = "" pool = "${var.public-ip-pool}" ... } terraform.tf Declarative DSL (OpenStack)

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 40 Declarative DSL (AWS) ## OpenVPN Compute instance
resource "aws_instance" "ovpn" { ami = "${var.ami_name}" instance_type = "${var.openvpn-instance-type}" vpc_security_group_ids = [ "${aws_security_group.ovpn.id}"] subnet_id = “${aws_subnet.dmz.id}" ... } ## DMZ network exposing Public IP resource "aws_subnet" "dmz" { vpc_id = "${aws_vpc.core.id}" cidr_block = "${var.dmz-net-cidr}" map_public_ip_on_launch = 1 ... } terraform.tf

subnet 10.0.0.64/27 DNS VPN XXX Jenkins Master Jira Jenkins Slave-n Artifac tory Router Internet Gateway region DMZ subnet 10.0.0.0/27 “Other” Infrastructure Providers

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 42 External DNS Availability Zone #1 Management subnet
10.0.0.32/24 Tooling subnet 10.0.0.64/27 DNS VPN XXX Jenkins Master Jira Jenkins Slave-n Artifac tory Router Internet Gateway region DMZ subnet 10.0.0.0/27 “Other” Infrastructure Providers

10.0.0.32/24 Tooling subnet 10.0.0.64/27 DNS VPN XXX Jenkins Master Jira Jenkins Slave-n Artifac tory Router Internet Gateway region DMZ subnet 10.0.0.0/27 “Other” Infrastructure Providers

10.0.0.32/24 Tooling subnet 10.0.0.64/27 DNS VPN XXX Jenkins Master Jira Jenkins Slave-n Artifac tory Router Internet Gateway region DMZ subnet 10.0.0.0/27 Simultaneous multi-cloud config

{...} ## OpenVPN Public IP resource "openstack_compute_floatingip_v2" "openvpn" {...} # External DNS (AWS Route 53) resource "aws_route53_record" "openvpn" { zone_id = "${var.route_53_domain_id}" name = "${var.env-prefix}-ovpn.ext.t1tools.domain.io" type = "A" ttl = "60" records = [“${openstack_compute_floatingip_v2. openvpn.address}"] } terraform.tf Declarative DSL

{ name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } variable "env-prefix" { default = "team1" } variable "image-name" { default = "centos-7-001" } variable "openvpn-flavour-name" { default = "x1.medium" } terraform.tf terraform.tfvars Vars & inter resource refs

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 47 resource "openstack_compute_instance_v2" "ovpn" { floating_ip = “${openstack_compute_floatingip_v2.
openvpn.address}" ... } resource "openstack_compute_floatingip_v2" "openvpn" { pool = "${var.public-ip-pool}" ... } # External DNS (AWS Route 53) resource "aws_route53_record" "openvpn" { name = "${var.env-prefix}-ovpn.ext.t1tools.domain.io" records = [“${openstack_compute_floatingip_v2. openvpn.address}"] } terraform.tf Vars & inter resource refs

openvpn.address}" ... } resource "openstack_compute_floatingip_v2" "openvpn" { pool = "${var.public-ip-pool}" ... } # External DNS (AWS Route 53) resource "aws_route53_record" "openvpn" { name = "${var.env-prefix}-ovpn.ext.t1tools.domain.io" records = [“${openstack_compute_floatingip_v2. openvpn.address}"] } 1 terraform.tf Vars & inter resource refs

openvpn.address}" ... } resource "openstack_compute_floatingip_v2" "openvpn" { pool = "${var.public-ip-pool}" ... } # External DNS (AWS Route 53) resource "aws_route53_record" "openvpn" { name = "${var.env-prefix}-ovpn.ext.t1tools.domain.io" records = [“${openstack_compute_floatingip_v2. openvpn.address}"] } 1 2a 2b terraform.tf Vars & inter resource refs

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 50 ## Jenkins Slave Compute instance resource "openstack_compute_instance_v2"
"js" { count = ${var.jenslave_count} region = "" name = "${var.env-prefix}-js${count.index}” image_name = "${var.image_name}" flavor_name = "${var.jenslave-flavour-name}" ... } terraform.tf Resource Scaling

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 51 Terraform plan (diff for infrastructure)

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 52 + module.mgt.openstack_compute_instance_v2.openvpn name: "" => "team1-ovpn" image_name:
"" => "centos-7-001" floating_ip: "" => "<computed>" flavour_name: "" => "x1.medium" user_data: "" => "b50c352a2e23ceba60e1b10fd3e2" . . . -/+ module.mgt.openstack_compute_instance_v2.ipa name: “team1-ipa" => “team1-ipa" image_name: "centos-7-001" => "centos-7-001" floating_ip: “" => "<computed>" flavour_name: "x1.medium" => “x1.large” (forces new resource) . . . Terraform plan

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 53 • Adhered to all our ASAP principles
• Handle multiple infrastructure providers  • Terraform still evolving, not perfect  • Other potential alternatives • Tosca/Cloudify IaaS provisioning Summary

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 54 Interlude … consistent cloud images ## OpenVPN
Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } terraform.tf

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 55 Interlude … consistent cloud images ## OpenVPN
Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } terraform.tf

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 56 Challenge #2 Automated Conﬁg Management

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 57 initial bootstrap vs. longer term maintenance

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- boot time cloud/VM instance customisation 58 conﬁguration management
tool Lesson #2 conscious de-coupling is your friend —>

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 59 —> Approach & Tools #2 Automated (Bootstrap)
Conﬁg Management

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider management subnet tooling subnet 10.20.251.36 public
IP Terraform complete … External DNS Provider t1team- ovpn.ext. t1tools. domain.io

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service management subnet
tooling subnet 10.20.251.36 Terraform complete … cloud-init takes over public IP

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 62 Boot time customisation of cloud instances (VMs)

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 63 Hooks into cloud provider’s metadata service cloud
provider metadata service

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 64 Accesses user supplied data for VM it
is running on cloud provider metadata service #cloud-config hostname: ${env-prefix}-jm fqdn: ${env-prefix}-jm.${domain} manage_etc_hosts: true puppet: conf: agent: server: "${env-prefix}-ipa.$ {domain}" runcmd:

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 65 Example (user-data) cloud config #cloud-config hostname: ${env-prefix}-jm
fqdn: ${env-prefix}-jm.${domain} manage_etc_hosts: true puppet: conf: agent: server: "${env-prefix}-ipa.${domain}" runcmd: - until curl -ksf https://${env-prefix}-ipa.${domain}:443/ca/ admin/ca/getStatus ; do sleep 30 ; done ; ipa-client-install — domain=${domain} ... --unattended --force-join - export COUNT=0 ; until puppet agent -t ; do echo "`date` - Attempting to run puppet agent for $COUNT time" ; if [[ $COUNT -eq 3 ]] ; then break ; fi ; sleep 30 ; ((COUNT++)) ; done

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 66 #cloud-config hostname: ${env-prefix}-jm fqdn: ${env-prefix}-jm.${domain} manage_etc_hosts: true
puppet: conf: agent: server: "${env-prefix}-ipa.${domain}" runcmd: - until curl -ksf https://${env-prefix}-ipa.${domain}:443/ca/ admin/ca/getStatus ; do sleep 30 ; done ; ipa-client-install — domain=${domain} ... --unattended --force-join - export COUNT=0 ; until puppet agent -t ; do echo "`date` - Attempting to run puppet agent for $COUNT time" ; if [[ $COUNT -eq 3 ]] ; then break ; fi ; sleep 30 ; ((COUNT++)) ; done Example (user-data) cloud config

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "jm" {
name = "${var.env-prefix}-jm" image_name = "${var.image_name}" flavor_name = "${var.jm-flavour-name}" user_data = "${template_file.clientconfig.rendered}" ... } ## UserData as input to cloud-init resource "template_file" "clientconfig" { filename = "${path.module}/clientconfig.template" vars { domain = "${var.domain}" env-prefix = "${var.env-prefix}" ... } } 70 Passing user-data via terraform

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 71 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "jm"
{ name = "${var.env-prefix}-jm" image_name = "${var.image_name}" flavor_name = "${var.jm-flavour-name}" user_data = "${template_file.clientconfig.rendered}" ... } ## UserData as input to cloud-init resource "template_file" "clientconfig" { filename = "${path.module}/clientconfig.template" vars { domain = "${var.domain}" env-prefix = "${var.env-prefix}" ... } } Passing user-data via terraform

tooling subnet 10.20.251.36 Terraform complete … public IP

tooling subnet 10.20.251.36 Terraform complete … cloud-init takes over public IP

tooling subnet 10.20.251.36 orchestration box: uses supplied token to connect to vault, download keys (github etc)

tooling subnet 10.20.251.36 orchestration box: downloads puppet source code

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service team1-ipa. t1tools.domain.io
management subnet tooling subnet orchestration box: initiates configuration on itself, installs and configures FreeIPA (DNS)

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service team1-ovpn. t1tools.domain.io
team1-ipa. t1tools.domain.io team1-jm. t1tools. domain.io team1-js1. t1tools. domain.io team1-js2. t1tools. domain.io team1-jira. t1tools. domain.io team1-elk. t1tools.domain.io management subnet tooling subnet all other boxes: waiting for DNS service to be available, then register & obtain DNS names

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service team1-ovpn. t1tools.domain.io
team1-ipa. t1tools.domain.io team1-jm. t1tools. domain.io team1-js1. t1tools. domain.io team1-js2. t1tools. domain.io team1-jira. t1tools. domain.io team1-elk. t1tools.domain.io management subnet tooling subnet all other boxes: Initiate bootstrap puppet run

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 78 Conﬁg Management Tools

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 80 Automated Config Mgt Summary • Adhered to
all our ASAP principles • Async bootstrap process   • Swapped Ansible and Puppet 

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 81 Challenge #3 Securing your stuff

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 82 •Perimeter security  •Infrastructure as code: security groups,
ﬁrewall rules etc  •OpenVPN

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 83 •User Access Control  •SSH keys only  •User
accounts -   managed by FreeIPA

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 84 • Secrets Management  • For IaaS provisioning 
• For Conﬁg management • For anything needing access to sensitive stuff …

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 85 Lesson #3 don’t roll your own!

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 86 Tool #3: Secure secrets management

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 87 A tool for managing secrets and other
sensitive content

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 88 • Uniﬁed API to access multiple backends
• ACL policies - who can access what • Audit Logs • And more …

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Anything Else

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unseal Create env speciﬁc mount, add secrets Acquire
Token Use token as input for initial bootstrap Init

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 92 $ vault init -key-shares=3 -key-threshold=2 Key 1:
8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Key 2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b Vault initialized with 3 keys and a key threshold of 2. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 2 of these keys to unseal it again. Vault does not store the master key. Without at least 2 keys, your Vault will remain permanently sealed. $ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 1 $ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Sealed: false Key Shares: 3 Key Threshold: 2 Unseal Progress: 0 Vault init & unseal

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- $ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Sealed: true Key Shares:
3 Key Threshold: 2 Unseal Progress: 1 $ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Sealed: false Key Shares: 3 Key Threshold: 2 Unseal Progress: 0 $ vault init -key-shares=3 -key-threshold=2 Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Key 2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b Vault initialized with 3 keys and a key threshold of 2. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 2 of these keys to unseal it again. Vault does not store the master key. Without at least 2 keys, your Vault will remain permanently sealed. 93 Vault init & unseal

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- $ vault init -key-shares=3 -key-threshold=2 Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba
Key 2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b Vault initialized with 3 keys and a key threshold of 2. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 2 of these keys to unseal it again. Vault does not store the master key. Without at least 2 keys, your Vault will remain permanently sealed. $ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 1 $ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Sealed: false Key Shares: 3 Key Threshold: 2 Unseal Progress: 0 94 Vault init & unseal

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unseal Create env speciﬁc mount, add secrets Use
token as input for initial bootstrap Init Acquire policy constrained token

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 96 $ vault mount -path=team1 generic Successfully mounted
'generic' at ‘team1'! $ vault mounts Path Type Default TTL Max TTL Description cubbyhole/ cubbyhole n/a n/a per-token private secr ... secret/ generic system system generic secret storage sys/ system n/a n/a system endpoints used f... team1/ generic system system Vault create new mount

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 97 Vault write, then read back secret $
vault write team1/cloud-provider-password value=ASDKJ234SF*2 Success! Data written to: team1/cloud-provider-password $ vault read team1/cloud-provider-password Key Value lease_duration 2592000 value ASDKJ234SF*2

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unseal Use token as input for initial bootstrap
Init Create env speciﬁc mount, add secrets Acquire policy constrained token

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 99 $ cat team1-bootstrap.policy path "team1/*" { policy
= "read" } path "auth/token/lookup-self" { policy = "read" } $ vault policy-write team1-puppet team1-bootstrap.policy Policy 'team1-bootstrap' written. Vault create custom policy

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 100 $ vault token-create -policy="team1-bootstrap" Key Value token
4a1cd43a-9206-9f48-caf8-b05e05457271 token_duration 2592000 token_renewable true token_policies [team1-bootstrap] $ vault auth 4a1cd43a-9206-9f48-caf8-b05e05457271 Successfully authenticated! token: 4a1cd43a-9206-9f48-caf8-b05e05457271 token_duration: 2592000 token_policies: [team1-bootstrap] $ vault read team1/cloud-provider-password Key Value lease_duration 2592000 value ASDKJ234SF*2 $ vault write team1/somekey value=somevalue Error writing data to team1/somekey: Error making API request. URL: PUT http://127.0.0.1:8200/v1/team1/somekey Code: 403. Errors: * permission denied Vault validate custom policy

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 102 • Scraped surface  • Stuck to ASAP
principles • Centralised Secrets management very handy • Try not to roll your own!  Security Summary

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 103 Challenge #4 Moving your stuff because state
complicates things …

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 104 Lesson #4 Baby steps … Initial approach
focuses on more DR (recreate) type scenarios, rather than realtime data migrations

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 105 Approach #4 Offsite backups … then restore

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 106 Conclusion #4 • This approach is limited
• Doesn’t handle realtime migrations • Still work in progress 

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 107 Conclusion

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 108 https://www.opencredo.com/2015/08/10/boot-my-secure-government-cloud

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 109 Be true to your principles, but flex
your approach (and tools) as required 

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 110 “The only thing constant in life is
change.” — François de La Rochefoucauld 

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 111 Don’t ask for an estimate, it’s still
work in progress — Me 

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 112 Thanks Questions? @techiewatt

CodeMesh 2015 : Boot my (secure)->(portable) clouds!

CodeMesh 2015 : Boot my (secure)->(portable) clouds!

Nicki Watt

More Decks by Nicki Watt

Other Decks in Technology

Featured

Transcript