Tweet
Tweet

Slide 1

Slide 1 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Boot my (secure) —> (portable) clouds! Nicki Watt @techiewatt 04/11/2015 1

Slide 2

Slide 2 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2 About Me • Hands on Lead consultant at OpenCredo
 • Co-author Neo4j In Action
 • Currently working with UK gov dept on cloud automation project
 • Twitter: @techiewatt 


Slide 3

Slide 3 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3 Agenda • What is the problem
 • What are the options
 • How: Principles, challenges, lessons, tools • Conclusion 


Slide 4

Slide 4 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4 What problem are we trying to address?

Slide 5

Slide 5 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 6 Act 1 : “Take advantage of cloud computing”

Slide 6

Slide 6 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 7 Act 2 : “Efficiently Take advantage of more cloud computing”

Slide 7

Slide 7 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 8

Slide 8

Slide 8 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 9

Slide 9

Slide 9 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10 How to create fast, repeatable, secure environments capable of running in different clouds!

Slide 10

Slide 10 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 11 • Public cloud workloads • Dev teams anywhere in the world
 • No human intervention Initial focus:

Slide 11

Slide 11 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 12 An example requirement: Team1 needs a development CI/CD env

Slide 12

Slide 12 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 13 Input: Environment prefix: team1 Number Jenkins Slaves: 3 Environment domain suffix: t1tools.domain.io Initial SSH keys: AAAEFF user1@team1 Cloud: AWS

Slide 13

Slide 13 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 14

Slide 14

Slide 14 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 15 What are our options?

Slide 15

Slide 15 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 16 you need our cloud management platform !

Slide 16

Slide 16 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 17 vs

Slide 17

Slide 17 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 18

Slide 18

Slide 18 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 19 Principles for success

Slide 19

Slide 19 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 20 • Automate everything
 • Separate config from code • API driven clouds & tools • Prefer modular, open source tools
 ASAP

Slide 20

Slide 20 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 25 • Self service functionality • Automated environment creation 
 (under the hood) functionality

Slide 21

Slide 21 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 26 Challenges Lessons Tools

Slide 22

Slide 22 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ^ 27 • Automated IaaS Provisioning • Automated Config Management • Securing stuff • Moving stuff Bootstrap

Slide 23

Slide 23 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 28 Challenge #1 Automated IaaS Provisioning

Slide 24

Slide 24 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 30 Availability Zone #1 Management subnet 10.0.0.32/24 Tooling subnet 10.0.0.64/27 DNS VPN XXX Jenkins Master Jira Jenkins Slave-n Artifac tory Router Internet Gateway region DMZ subnet 10.0.0.0/27 • Networks • Firewall Rules • Routers • Compute 
 Resources • Public / Floating 
 IP Addresses

Slide 25

Slide 25 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 31 Lesson #1 There is NO single common cloud API

Slide 26

Slide 26 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 32 Tool #1: Terraform Automated IaaS Provisioning

Slide 27

Slide 27 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 33 Creates, manages, and manipulates infrastructure resources.

Slide 28

Slide 28 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 34 Availability Zone #1 Management subnet 10.0.0.32/24 Tooling subnet 10.0.0.64/27 DNS VPN XXX Jenkins Master Jira Jenkins Slave-n Artifac tory Router Internet Gateway region DMZ subnet 10.0.0.0/27 Multiple Cloud Providers

Slide 29

Slide 29 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 35 Declarative DSL ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } ## OpenVPN Public IP resource "openstack_compute_floatingip_v2" "openvpn" { region = "" pool = "${var.public-ip-pool}" ... } terraform.tf

Slide 30

Slide 30 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 36 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } ## OpenVPN Public IP resource "openstack_compute_floatingip_v2" "openvpn" { region = "" pool = "${var.public-ip-pool}" ... } terraform.tf Declarative DSL (OpenStack)

Slide 31

Slide 31 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 37 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } ## OpenVPN Public IP resource "openstack_compute_floatingip_v2" "openvpn" { region = "" pool = "${var.public-ip-pool}" ... } terraform.tf Declarative DSL (OpenStack)

Slide 32

Slide 32 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 38 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } ## OpenVPN Public IP resource "openstack_compute_floatingip_v2" "openvpn" { region = "" pool = "${var.public-ip-pool}" ... } terraform.tf Declarative DSL (OpenStack)

Slide 33

Slide 33 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 39 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } ## OpenVPN Public IP resource "openstack_compute_floatingip_v2" "openvpn" { region = "" pool = "${var.public-ip-pool}" ... } terraform.tf Declarative DSL (OpenStack)

Slide 34

Slide 34 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 40 Declarative DSL (AWS) ## OpenVPN Compute instance resource "aws_instance" "ovpn" { ami = "${var.ami_name}" instance_type = "${var.openvpn-instance-type}" vpc_security_group_ids = [ "${aws_security_group.ovpn.id}"] subnet_id = “${aws_subnet.dmz.id}" ... } ## DMZ network exposing Public IP resource "aws_subnet" "dmz" { vpc_id = "${aws_vpc.core.id}" cidr_block = "${var.dmz-net-cidr}" map_public_ip_on_launch = 1 ... } terraform.tf

Slide 35

Slide 35 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 41 Availability Zone #1 Management subnet 10.0.0.32/24 Tooling subnet 10.0.0.64/27 DNS VPN XXX Jenkins Master Jira Jenkins Slave-n Artifac tory Router Internet Gateway region DMZ subnet 10.0.0.0/27 “Other” Infrastructure Providers

Slide 36

Slide 36 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 42 External DNS Availability Zone #1 Management subnet 10.0.0.32/24 Tooling subnet 10.0.0.64/27 DNS VPN XXX Jenkins Master Jira Jenkins Slave-n Artifac tory Router Internet Gateway region DMZ subnet 10.0.0.0/27 “Other” Infrastructure Providers

Slide 37

Slide 37 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 43 External DNS Availability Zone #1 Management subnet 10.0.0.32/24 Tooling subnet 10.0.0.64/27 DNS VPN XXX Jenkins Master Jira Jenkins Slave-n Artifac tory Router Internet Gateway region DMZ subnet 10.0.0.0/27 “Other” Infrastructure Providers

Slide 38

Slide 38 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 44 External DNS Availability Zone #1 Management subnet 10.0.0.32/24 Tooling subnet 10.0.0.64/27 DNS VPN XXX Jenkins Master Jira Jenkins Slave-n Artifac tory Router Internet Gateway region DMZ subnet 10.0.0.0/27 Simultaneous multi-cloud config

Slide 39

Slide 39 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 45 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn" {...} ## OpenVPN Public IP resource "openstack_compute_floatingip_v2" "openvpn" {...} # External DNS (AWS Route 53) resource "aws_route53_record" "openvpn" { zone_id = "${var.route_53_domain_id}" name = "${var.env-prefix}-ovpn.ext.t1tools.domain.io" type = "A" ttl = "60" records = [“${openstack_compute_floatingip_v2. openvpn.address}"] } terraform.tf Declarative DSL

Slide 40

Slide 40 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 46 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } variable "env-prefix" { default = "team1" } variable "image-name" { default = "centos-7-001" } variable "openvpn-flavour-name" { default = "x1.medium" } terraform.tf terraform.tfvars Vars & inter resource refs

Slide 41

Slide 41 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 47 resource "openstack_compute_instance_v2" "ovpn" { floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } resource "openstack_compute_floatingip_v2" "openvpn" { pool = "${var.public-ip-pool}" ... } # External DNS (AWS Route 53) resource "aws_route53_record" "openvpn" { name = "${var.env-prefix}-ovpn.ext.t1tools.domain.io" records = [“${openstack_compute_floatingip_v2. openvpn.address}"] } terraform.tf Vars & inter resource refs

Slide 42

Slide 42 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 48 resource "openstack_compute_instance_v2" "ovpn" { floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } resource "openstack_compute_floatingip_v2" "openvpn" { pool = "${var.public-ip-pool}" ... } # External DNS (AWS Route 53) resource "aws_route53_record" "openvpn" { name = "${var.env-prefix}-ovpn.ext.t1tools.domain.io" records = [“${openstack_compute_floatingip_v2. openvpn.address}"] } 1 terraform.tf Vars & inter resource refs

Slide 43

Slide 43 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 49 resource "openstack_compute_instance_v2" "ovpn" { floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } resource "openstack_compute_floatingip_v2" "openvpn" { pool = "${var.public-ip-pool}" ... } # External DNS (AWS Route 53) resource "aws_route53_record" "openvpn" { name = "${var.env-prefix}-ovpn.ext.t1tools.domain.io" records = [“${openstack_compute_floatingip_v2. openvpn.address}"] } 1 2a 2b terraform.tf Vars & inter resource refs

Slide 44

Slide 44 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 50 ## Jenkins Slave Compute instance resource "openstack_compute_instance_v2" "js" { count = ${var.jenslave_count} region = "" name = "${var.env-prefix}-js${count.index}” image_name = "${var.image_name}" flavor_name = "${var.jenslave-flavour-name}" ... } terraform.tf Resource Scaling

Slide 45

Slide 45 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 51 Terraform plan (diff for infrastructure)

Slide 46

Slide 46 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 52 + module.mgt.openstack_compute_instance_v2.openvpn name: "" => "team1-ovpn" image_name: "" => "centos-7-001" floating_ip: "" => "" flavour_name: "" => "x1.medium" user_data: "" => "b50c352a2e23ceba60e1b10fd3e2" . . . -/+ module.mgt.openstack_compute_instance_v2.ipa name: “team1-ipa" => “team1-ipa" image_name: "centos-7-001" => "centos-7-001" floating_ip: “" => "" flavour_name: "x1.medium" => “x1.large” (forces new resource) . . . Terraform plan

Slide 47

Slide 47 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 53 • Adhered to all our ASAP principles • Handle multiple infrastructure providers
 • Terraform still evolving, not perfect
 • Other potential alternatives • Tosca/Cloudify IaaS provisioning Summary

Slide 48

Slide 48 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 54 Interlude … consistent cloud images ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } terraform.tf

Slide 49

Slide 49 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 55 Interlude … consistent cloud images ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } terraform.tf

Slide 50

Slide 50 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 56 Challenge #2 Automated Config Management

Slide 51

Slide 51 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 57 initial bootstrap vs. longer term maintenance

Slide 52

Slide 52 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- boot time cloud/VM instance customisation 58 configuration management tool Lesson #2 conscious de-coupling is your friend —>

Slide 53

Slide 53 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 59 —> Approach & Tools #2 Automated (Bootstrap) Config Management

Slide 54

Slide 54 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider management subnet tooling subnet 10.20.251.36 public IP Terraform complete … External DNS Provider t1team- ovpn.ext. t1tools. domain.io

Slide 55

Slide 55 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service management subnet tooling subnet 10.20.251.36 Terraform complete … cloud-init takes over public IP

Slide 56

Slide 56 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 62 Boot time customisation of cloud instances (VMs)

Slide 57

Slide 57 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 63 Hooks into cloud provider’s metadata service cloud provider metadata service

Slide 58

Slide 58 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 64 Accesses user supplied data for VM it is running on cloud provider metadata service #cloud-config hostname: ${env-prefix}-jm fqdn: ${env-prefix}-jm.${domain} manage_etc_hosts: true puppet: conf: agent: server: "${env-prefix}-ipa.$ {domain}" runcmd:

Slide 59

Slide 59 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 65 Example (user-data) cloud config #cloud-config hostname: ${env-prefix}-jm fqdn: ${env-prefix}-jm.${domain} manage_etc_hosts: true puppet: conf: agent: server: "${env-prefix}-ipa.${domain}" runcmd: - until curl -ksf https://${env-prefix}-ipa.${domain}:443/ca/ admin/ca/getStatus ; do sleep 30 ; done ; ipa-client-install — domain=${domain} ... --unattended --force-join - export COUNT=0 ; until puppet agent -t ; do echo "`date` - Attempting to run puppet agent for $COUNT time" ; if [[ $COUNT -eq 3 ]] ; then break ; fi ; sleep 30 ; ((COUNT++)) ; done

Slide 60

Slide 60 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 66 #cloud-config hostname: ${env-prefix}-jm fqdn: ${env-prefix}-jm.${domain} manage_etc_hosts: true puppet: conf: agent: server: "${env-prefix}-ipa.${domain}" runcmd: - until curl -ksf https://${env-prefix}-ipa.${domain}:443/ca/ admin/ca/getStatus ; do sleep 30 ; done ; ipa-client-install — domain=${domain} ... --unattended --force-join - export COUNT=0 ; until puppet agent -t ; do echo "`date` - Attempting to run puppet agent for $COUNT time" ; if [[ $COUNT -eq 3 ]] ; then break ; fi ; sleep 30 ; ((COUNT++)) ; done Example (user-data) cloud config

Slide 61

Slide 61 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 67 #cloud-config hostname: ${env-prefix}-jm fqdn: ${env-prefix}-jm.${domain} manage_etc_hosts: true puppet: conf: agent: server: "${env-prefix}-ipa.${domain}" runcmd: - until curl -ksf https://${env-prefix}-ipa.${domain}:443/ca/ admin/ca/getStatus ; do sleep 30 ; done ; ipa-client-install — domain=${domain} ... --unattended --force-join - export COUNT=0 ; until puppet agent -t ; do echo "`date` - Attempting to run puppet agent for $COUNT time" ; if [[ $COUNT -eq 3 ]] ; then break ; fi ; sleep 30 ; ((COUNT++)) ; done Example (user-data) cloud config

Slide 62

Slide 62 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 68 #cloud-config hostname: ${env-prefix}-jm fqdn: ${env-prefix}-jm.${domain} manage_etc_hosts: true puppet: conf: agent: server: "${env-prefix}-ipa.${domain}" runcmd: - until curl -ksf https://${env-prefix}-ipa.${domain}:443/ca/ admin/ca/getStatus ; do sleep 30 ; done ; ipa-client-install — domain=${domain} ... --unattended --force-join - export COUNT=0 ; until puppet agent -t ; do echo "`date` - Attempting to run puppet agent for $COUNT time" ; if [[ $COUNT -eq 3 ]] ; then break ; fi ; sleep 30 ; ((COUNT++)) ; done Example (user-data) cloud config

Slide 63

Slide 63 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 69 #cloud-config hostname: ${env-prefix}-jm fqdn: ${env-prefix}-jm.${domain} manage_etc_hosts: true puppet: conf: agent: server: "${env-prefix}-ipa.${domain}" runcmd: - until curl -ksf https://${env-prefix}-ipa.${domain}:443/ca/ admin/ca/getStatus ; do sleep 30 ; done ; ipa-client-install — domain=${domain} ... --unattended --force-join - export COUNT=0 ; until puppet agent -t ; do echo "`date` - Attempting to run puppet agent for $COUNT time" ; if [[ $COUNT -eq 3 ]] ; then break ; fi ; sleep 30 ; ((COUNT++)) ; done Example (user-data) cloud config

Slide 64

Slide 64 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "jm" { name = "${var.env-prefix}-jm" image_name = "${var.image_name}" flavor_name = "${var.jm-flavour-name}" user_data = "${template_file.clientconfig.rendered}" ... } ## UserData as input to cloud-init resource "template_file" "clientconfig" { filename = "${path.module}/clientconfig.template" vars { domain = "${var.domain}" env-prefix = "${var.env-prefix}" ... } } 70 Passing user-data via terraform

Slide 65

Slide 65 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 71 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "jm" { name = "${var.env-prefix}-jm" image_name = "${var.image_name}" flavor_name = "${var.jm-flavour-name}" user_data = "${template_file.clientconfig.rendered}" ... } ## UserData as input to cloud-init resource "template_file" "clientconfig" { filename = "${path.module}/clientconfig.template" vars { domain = "${var.domain}" env-prefix = "${var.env-prefix}" ... } } Passing user-data via terraform

Slide 66

Slide 66 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service management subnet tooling subnet 10.20.251.36 Terraform complete … public IP

Slide 67

Slide 67 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service management subnet tooling subnet 10.20.251.36 Terraform complete … cloud-init takes over public IP

Slide 68

Slide 68 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service management subnet tooling subnet 10.20.251.36 orchestration box: uses supplied token to connect to vault, download keys (github etc)

Slide 69

Slide 69 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service management subnet tooling subnet 10.20.251.36 orchestration box: uses supplied token to connect to vault, download keys (github etc)

Slide 70

Slide 70 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service management subnet tooling subnet 10.20.251.36 orchestration box: downloads puppet source code

Slide 71

Slide 71 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service team1-ipa. t1tools.domain.io management subnet tooling subnet orchestration box: initiates configuration on itself, installs and configures FreeIPA (DNS)

Slide 72

Slide 72 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service team1-ovpn. t1tools.domain.io team1-ipa. t1tools.domain.io team1-jm. t1tools. domain.io team1-js1. t1tools. domain.io team1-js2. t1tools. domain.io team1-jira. t1tools. domain.io team1-elk. t1tools.domain.io management subnet tooling subnet all other boxes: waiting for DNS service to be available, then register & obtain DNS names

Slide 73

Slide 73 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service team1-ovpn. t1tools.domain.io team1-ipa. t1tools.domain.io team1-jm. t1tools. domain.io team1-js1. t1tools. domain.io team1-js2. t1tools. domain.io team1-jira. t1tools. domain.io team1-elk. t1tools.domain.io management subnet tooling subnet all other boxes: Initiate bootstrap puppet run

Slide 74

Slide 74 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 78 Config Management Tools

Slide 75

Slide 75 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 80 Automated Config Mgt Summary • Adhered to all our ASAP principles • Async bootstrap process 
 • Swapped Ansible and Puppet


Slide 76

Slide 76 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 81 Challenge #3 Securing your stuff

Slide 77

Slide 77 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 82 •Perimeter security
 •Infrastructure as code: security groups, firewall rules etc
 •OpenVPN

Slide 78

Slide 78 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 83 •User Access Control
 •SSH keys only
 •User accounts - 
 managed by FreeIPA

Slide 79

Slide 79 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 84 • Secrets Management
 • For IaaS provisioning
 • For Config management • For anything needing access to sensitive stuff …

Slide 80

Slide 80 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 85 Lesson #3 don’t roll your own!

Slide 81

Slide 81 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 86 Tool #3: Secure secrets management

Slide 82

Slide 82 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 87 A tool for managing secrets and other sensitive content

Slide 83

Slide 83 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 88 • Unified API to access multiple backends • ACL policies - who can access what • Audit Logs • And more …

Slide 84

Slide 84 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Anything Else

Slide 85

Slide 85 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unseal Create env specific mount, add secrets Acquire Token Use token as input for initial bootstrap Init

Slide 86

Slide 86 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 92 $ vault init -key-shares=3 -key-threshold=2 Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Key 2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b Vault initialized with 3 keys and a key threshold of 2. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 2 of these keys to unseal it again. Vault does not store the master key. Without at least 2 keys, your Vault will remain permanently sealed. $ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 1 $ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Sealed: false Key Shares: 3 Key Threshold: 2 Unseal Progress: 0 Vault init & unseal

Slide 87

Slide 87 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- $ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 1 $ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Sealed: false Key Shares: 3 Key Threshold: 2 Unseal Progress: 0 $ vault init -key-shares=3 -key-threshold=2 Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Key 2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b Vault initialized with 3 keys and a key threshold of 2. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 2 of these keys to unseal it again. Vault does not store the master key. Without at least 2 keys, your Vault will remain permanently sealed. 93 Vault init & unseal

Slide 88

Slide 88 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- $ vault init -key-shares=3 -key-threshold=2 Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Key 2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b Vault initialized with 3 keys and a key threshold of 2. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 2 of these keys to unseal it again. Vault does not store the master key. Without at least 2 keys, your Vault will remain permanently sealed. $ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 1 $ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Sealed: false Key Shares: 3 Key Threshold: 2 Unseal Progress: 0 94 Vault init & unseal

Slide 89

Slide 89 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unseal Create env specific mount, add secrets Use token as input for initial bootstrap Init Acquire policy constrained token

Slide 90

Slide 90 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 96 $ vault mount -path=team1 generic Successfully mounted 'generic' at ‘team1'! $ vault mounts Path Type Default TTL Max TTL Description cubbyhole/ cubbyhole n/a n/a per-token private secr ... secret/ generic system system generic secret storage sys/ system n/a n/a system endpoints used f... team1/ generic system system Vault create new mount

Slide 91

Slide 91 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 97 Vault write, then read back secret $ vault write team1/cloud-provider-password value=ASDKJ234SF*2 Success! Data written to: team1/cloud-provider-password $ vault read team1/cloud-provider-password Key Value lease_duration 2592000 value ASDKJ234SF*2

Slide 92

Slide 92 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unseal Use token as input for initial bootstrap Init Create env specific mount, add secrets Acquire policy constrained token

Slide 93

Slide 93 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 99 $ cat team1-bootstrap.policy path "team1/*" { policy = "read" } path "auth/token/lookup-self" { policy = "read" } $ vault policy-write team1-puppet team1-bootstrap.policy Policy 'team1-bootstrap' written. Vault create custom policy

Slide 94

Slide 94 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 100 $ vault token-create -policy="team1-bootstrap" Key Value token 4a1cd43a-9206-9f48-caf8-b05e05457271 token_duration 2592000 token_renewable true token_policies [team1-bootstrap] $ vault auth 4a1cd43a-9206-9f48-caf8-b05e05457271 Successfully authenticated! token: 4a1cd43a-9206-9f48-caf8-b05e05457271 token_duration: 2592000 token_policies: [team1-bootstrap] $ vault read team1/cloud-provider-password Key Value lease_duration 2592000 value ASDKJ234SF*2 $ vault write team1/somekey value=somevalue Error writing data to team1/somekey: Error making API request. URL: PUT http://127.0.0.1:8200/v1/team1/somekey Code: 403. Errors: * permission denied Vault validate custom policy

Slide 95

Slide 95 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 102 • Scraped surface
 • Stuck to ASAP principles • Centralised Secrets management very handy • Try not to roll your own!
 Security Summary

Slide 96

Slide 96 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 103 Challenge #4 Moving your stuff because state complicates things …

Slide 97

Slide 97 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 104 Lesson #4 Baby steps … Initial approach focuses on more DR (recreate) type scenarios, rather than realtime data migrations

Slide 98

Slide 98 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 105 Approach #4 Offsite backups … then restore

Slide 99

Slide 99 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 106 Conclusion #4 • This approach is limited • Doesn’t handle realtime migrations • Still work in progress


Slide 100

Slide 100 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 107 Conclusion

Slide 101

Slide 101 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 108 https://www.opencredo.com/2015/08/10/boot-my-secure-government-cloud

Slide 102

Slide 102 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 109 Be true to your principles, but flex your approach (and tools) as required


Slide 103

Slide 103 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 110 “The only thing constant in life is change.” — François de La Rochefoucauld


Slide 104

Slide 104 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 111 Don’t ask for an estimate, it’s still work in progress — Me


Slide 105

Slide 105 text

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 112 Thanks Questions? @techiewatt