AnsibleFest NYC 2014

Covers Google Cloud Platform walk through, deeper dive on Compute Engine, and live demo of Ansible's gce* modules plus a new gc_dns module.

Transcript

1. Ansible and Google Cloud Platform Eric Johnson Technical Program Manager AnsibleFest NYC 2014
2. Agenda: May 20th, 2014 Whirlwind Tour of Google Cloud Platform Google Compute Engine Using Ansible and Compute Engine (demo) 1 2 3
3. Whirlwind Tour of Google Cloud Platform Google Compute Engine Using Ansible and Compute Engine (demo) 1 2 3
4. Storage Cloud Storage Cloud SQL Cloud Datastore Compute Compute Engine App Engine App Services BigQuery Cloud Endpoints Google Cloud Platform Cloud DNS
5. Compute Compute Engine App Engine Storage Cloud Storage Cloud SQL Cloud Datastore Power Computation App Services BigQuery Cloud Endpoints Cloud DNS
6. Storage Cloud Storage Cloud SQL Cloud Datastore Storing What You Want, However You Want Compute Compute Engine App Engine App Services BigQuery Cloud Endpoints Cloud DNS
7. App Services BigQuery Cloud Endpoints Building Robust, Intelligent Systems Storage Cloud Storage Cloud SQL Cloud Datastore Compute Compute Engine App Engine Cloud DNS
8. For the past 15 years, Google has been building out the world’s fastest, most powerful, highest quality cloud infrastructure on the planet. Images by Connie Zhou Why Google Cloud Platform?
9. A Network that Spans the Globe
10. Innovating Software & Driving Technology Forward Spanner Dremel MapReduce Big Table Colossus 2012 2013 2002 2004 2006 2008 2010 GFS Compute Engine
11. Building Products that Scale Google Maps Gmail Google Drive YouTube
12. “[Google's] ability to build, organize, and operate a huge network of servers and fiber- optic cables with an efficiency and speed that rocks physics on its heels. This is what makes Google Google: its physical network, its thousands of fiber miles, and those many thousands of servers that, in aggregate, add up to the mother of all clouds.” - Wired Images by Connie Zhou
13. Whirlwind Tour of Google Cloud Platform Google Compute Engine Using Ansible and Compute Engine (demo) 1 2 3
14. • IaaS: VMs, Network, Storage • Google DNA (speed, scale, reliable, secure) • Fast Provisioning, Consistent Performance • Enterprise Ready • 24x7 Support • 99.95% monthly SLA • ISO 27001, SSAE-16 SOC 1,2,3 • Accessible Through • Web @ https://cloud.google.com/console • gcutil command-line utility • REST API • Partners (Commercial and FOSS) Google Compute Engine
15. Launching 50 virtual machines
16. Block Storage Persistent Disks
17. Persistent Disk 10 TB • Data and root partitions • Billed only on provisioned capacity • Performance caps scale linearly with size • Volume striping is automatic • Differential snapshots • Create new PDs based on snapshots
18. PD Use-cases Root Stateful root volume User managed data volume Root RW Data RO Data Instant distribution of static content
19. Global Snapshot and Restore GCS vol1 vol2 vol3 vol1. t2 vol2. t2 vol3. t3 t1 t1 t1 t1 t1 t1 t1 t1 t1 t1 t1 t1 t1 t2 t2 t2 t2 t3 t3 t3 • Point in time snapshot to Google Cloud Storage (GCS) • Differential snapshots • GCS global replication! • Restore from snapshot anywhere in the world us-central1-a europe-west1-a
20. Virtual Machines
21. standard • For workloads with balanced CPU and memory highmem • For workloads with higher memory requirements highcpu • For workloads with higher CPU requirements Shared Core Instances • For inexpensive prototyping and staging workloads (g1-small, f1-micro) Machine Types Machine Type Virtual Cores Memory (GB) n1-standard-1 1 3.75 n1-standard-2 2 7.50 n1-standard-4 4 15.00 n1-standard-8 8 30.00 n1-standard-16 16 60.00 n1-highmem-2 2 13.00 n1-highmem-4 4 26.00 n1-highmem-8 8 52.00 n1-highmem-16 16 104.00 n1-highcpu-2 2 1.80 n1-highcpu-4 4 3.60 n1-highcpu-8 8 7.20 n1-highcpu-16 16 14.40
22. Google confidential │ Do not distribute go/internal-gce101 Images + Custom Image/Kernel Support (debian backports image works great for docker) https://developers.google.com/compute/docs/operating-systems
23. Sub-hour Billing + Sustained Use Discount 1 minute granularity, 10 minute minimum $ (30%) Full billing cycle Sustained Use Discount • >25% use, cost decreases • Up to 30% reduction • No commitment / contract
24. Enhanced Reliability us-central1-a scheduled maintenance event Live Migration • No downtime during scheduled datacenter maintenance events Automatic Restart • Instances automatically restarted if subjected to system events such as hardware failure
25. Networking
26. Images by Connie Zhou Networking • Google's Network • Projects are isolated private networks • TCP, UDP, ICMP only • Multiple private network groups and firewalls • Tags and address ranges • Addresses • public: static or ephemeral • private: ephemeral with DNS • Routes, gateways, VPNs, and IP Forwarding
27. Region: us-central1 Target Pool (tp-a) Load-balancer • Region based • Session affinity, hashing tuple options, • Protocol, src ip:port, dst ip:port • Consists of: • Target pool: collection of instances • Forwarding rules: protocol:port, pool • HTTP health check: optional us-central1-b us-central1-a www0 www1 www2 www3 Forwarding Rules tcp:443 ➔ tp-a tcp:80 ➔ tp-a Internet http://googlecloudplatform.blogspot.com/2013/11/compute-engine-load-balancing-hits-1-million-requests-per-second.html
28. ...and
29. Noteworthy • Regions • Central United States, Europe, Asia (new!) • Metadata • Startup scripts • SSH Keys • Tags (instance/network) • OAuth2 and Scopes • Access other Google Cloud Platform services • Ecosystem is growing • Partners: RightScale, Scalr, New Relic, MongoLab, MapR, and many more... • Open Source: Ansible, Chef, Puppet, Salt, Vagrant, jclouds, fog, libcloud
30. Whirlwind Tour of Google Cloud Platform Google Compute Engine Using Ansible and Compute Engine (demo) 1 2 3
31. Demo Ansible playbook that defines, 1. 4 Compute Engine instances (2 per zone) 2. Create a Firewall Rule and Load-Balancer a. Set up a DNS record for the LB IP 3. Deploy software and custom web page to each instance Region: us-central1 Target Pool (lb-tp) us-central1-a myinstance1 myinstance3 us-central1-b myinstance2 myinstance4 Forwarding Rules tcp:80 ➔ lb-tp GCE / DNS APIs $ ansible-playbook gce-demo.yml
32. Live Test (before) http://www.erjohn.so $ dig www.erjohn.so +noall +answer
33. My domain But first... 1. Verify domain ownership 2. Use Cloud SDK and dns sub-command to create a managed-zone 3. Update registrar with correct NS records
34. The "gce-demo" playbook gce-demo.yml 1 - name: Create Compute Engine instances 2 hosts: local 3 gather_facts: False 4 vars: 5 pid: google.com:erjohnso 6 email: 1234567890-xxxxxxxxxxxxx@developer.gserviceaccount.com 7 pem: /home/erjohnso/pkey.pem 8 machine_type: n1-standard-1 9 image: debian-7 10 names_zonea: myinstance1,myinstance3 11 zonea: us-central1-a 12 names_zoneb: myinstance2,myinstance4 13 zoneb: us-central1-b
35. The "gce-demo" playbook gce-demo.yml 1 - name: Create Compute Engine instances 2 hosts: local 3 gather_facts: False 4 vars: 5 pid: google.com:erjohnso 6 email: 1234567890-xxxxxxxxxxxxx@developer.gserviceaccount.com 7 pem: /home/erjohnso/pkey.pem 8 machine_type: n1-standard-1 9 image: debian-7 10 names_zonea: myinstance1,myinstance3 11 zonea: us-central1-a 12 names_zoneb: myinstance2,myinstance4 13 zoneb: us-central1-b
36. The "gce-demo" playbook gce-demo.yml 14 tasks: 15 - name: Bring up the instances in the first zone 16 gce: 17 instance_names: "{{ names_zonea }}" 18 machine_type: "{{ machine_type }}" 19 image: "{{ image }}" 20 zone: "{{ zonea }}" 21 project_id: "{{ pid }}" 22 pem_file: "{{ pem }}" 23 service_account_email: "{{ email }}" 24 register: gcea 25 - name: Bring up the instances in the second zone 26 gce: 27 instance_names: "{{ names_zoneb }}" 28 machine_type: "{{ machine_type }}" 29 image: "{{ image }}" 30 zone: "{{ zoneb }}" 31 project_id: "{{ pid }}" 32 pem_file: "{{ pem }}" 33 service_account_email: "{{ email }}" 34 register: gceb
37. The "gce-demo" playbook gce-demo.yml 35 post_tasks: 36 - name: Wait for SSH for instances in first zone 37 wait_for: delay=1 host={{ item.public_ip }} port=22 state=started timeout=30 38 with_items: gcea.instance_data 39 - name: Wait for SSH for instances in second zone 40 wait_for: delay=1 host={{ item.public_ip }} port=22 state=started timeout=30 41 with_items: gceb.instance_data 42
38. The "gce-demo" playbook gce-demo.yml 43 - name: Set up networking and DNS 44 hosts: local 45 gather_facts: False 46 vars: 47 pid: google.com:erjohnso 48 email: 1234567890-xxxxxxxxxxxxx@developer.gserviceaccount.com 49 pem: /home/erjohnso/pkey.pem 50 region: us-central1 51 members: us-central1-a/myinstance1,us-central1-a/myinstance3,us-central1- b/myinstanc... 52 tasks: 53 - name: Allow HTTP traffic 54 gce_net: 55 fwname: all-http 56 name: default 57 allowed: tcp:80 58 project_id: "{{ pid }}" 59 pem_file: "{{ pem }}" 60 service_account_email: "{{ email }}"
39. The "gce-demo" playbook gce-demo.yml 61 - name: Create the load-balancer and add members 62 gce_lb: 63 name: lb 64 region: "{{ region }}" 65 members: "{{ members }}" 66 project_id: "{{ pid }}" 67 pem_file: "{{ pem }}" 68 service_account_email: "{{ email }}" 69 register: gcelb 70 - name: Create an A record for www.erjohn.so 71 gc_dns: 72 command: create 73 name: erjohnso 74 record: www.erjohn.so. 75 ttl: 600 76 record_type: A 77 value: "{{ gcelb.external_ip }}" 78 project_id: "{{ pid }}" 79 pem_file: "{{ pem }}" 80 service_account_email: "{{ email }}" 81
40. The "gce-demo" playbook gce-demo.yml 82 - name: Deploy apache, mod_headers, and custom web page 83 hosts: gce_instances 84 sudo: yes 85 tasks: 86 - name: Install python-apt 87 command: apt-get install python-apt -y 88 - name: Install apache on instances 89 apt: pkg=apache2 state=present 90 - name: Create custom index.html 91 template: src=index.html.j2 dest=/var/www/index.html 92 - name: Set file stats on index.html 93 file: path=/var/www/index.html owner=root group=root mode=0644 94 - name: Deploy modified apache conf 95 copy: src=apache2.conf dest=/etc/apache2/apache2.conf 96 owner=root group=root mode=0644 97 - name: Enable mod_headers 98 file: path=/etc/apache2/mods-enabled/headers.load 99 src=/etc/apache2/mods-available/headers.load state=link 100 - name: Re-load apache 101 service: name=apache2 state=reloaded
41. Live Test (after) http://www.erjohn.so $ dig www.erjohn.so +noall +answer
42. Available now and more coming! • Existing modules: gce, gce_net, gce_pd, gce_lb, gc_storage, inventory plugin • Create, destroy instances + dynamic inventory • Create, destroy networks, firewall rules • Create, destroy Persistent Disks • Create, destroy load-balancer, healthchecks • Create, destroy buckets, objects Available Now!! Coming Soon!! • The gc_dns module • Create, destroy managed-zones • Create, update DNS records
43. Google confidential │ Do not distribute $500 in Cloud Platform credit to launch your idea! Build. Store. Analyze. On the same infrastructure that powers Google Start building Go to g.co/cloudstarterpack Click ‘Apply Now’ and complete the application with promo code: ansnyc-2014 Starter Pack Offer Description 1 2 3 ;-)
44. cloud.google.com - Thank you! Read more at • Compute Engine: https://cloud.google.com/products/compute-engine • Ansible + Compute Engine: http://docs.ansible.com/guide_gce.html • ... and look for the new gc_dns module soon! Get Started on Google Cloud Platform • $500 credit, http://g.co/cloudstarterpack (use promo code: ansnyc-2014) Questions? {'Freenode': 'erjohnso', 'Github': 'erjohnso', 'Twitter': 'no!'} ;-)