ChefConf2014: Google+Chef

Covers Google Cloud Platform walk through, deeper dive on Compute Engine, and shows knife-google and GCE LWRP demos.

video: https://www.youtube.com/watch?v=D2OICR18zIo

Transcript

1. Chef and Google Cloud Platform Eric Johnson Technical Program Manager
2. Agenda: April 16th, 2014 Whirlwind Tour of Google Cloud Platform Google Compute Engine Using Chef and Compute Engine (demo) 1 2 3
3. Whirlwind Tour of Google Cloud Platform Google Compute Engine Using SaltStack and Compute Engine (demo) 1 2 3
4. Storage Cloud Storage Cloud SQL Cloud Datastore Compute Compute Engine App Engine App Services BigQuery Cloud Endpoints Google Cloud Platform Cloud DNS
5. Compute Compute Engine App Engine Storage Cloud Storage Cloud SQL Cloud Datastore Power Computation App Services BigQuery Cloud Endpoints Cloud DNS
6. Storage Cloud Storage Cloud SQL Cloud Datastore Storing What You Want, However You Want Compute Compute Engine App Engine App Services BigQuery Cloud Endpoints Cloud DNS
7. App Services BigQuery Cloud Endpoints Building Robust, Intelligent Systems Storage Cloud Storage Cloud SQL Cloud Datastore Compute Compute Engine App Engine Cloud DNS
8. For the past 15 years, Google has been building out the world’s fastest, most powerful, highest quality cloud infrastructure on the planet. Images by Connie Zhou Why Google Cloud Platform?
9. A Network that Spans the Globe
10. Innovating Software & Driving Technology Forward Spanner Dremel MapReduce Big Table Colossus 2012 2013 2002 2004 2006 2008 2010 GFS Compute Engine
11. Building Products that Scale Google Maps Gmail Google Drive YouTube
12. “[Google's] ability to build, organize, and operate a huge network of servers and fiber- optic cables with an efficiency and speed that rocks physics on its heels. This is what makes Google Google: its physical network, its thousands of fiber miles, and those many thousands of servers that, in aggregate, add up to the mother of all clouds.” - Wired Images by Connie Zhou
13. Whirlwind Tour of Google Cloud Platform Google Compute Engine Using SaltStack and Compute Engine (demo) 1 2 3
14. • IaaS: VMs, Network, Storage • Google DNA (speed, scale, reliable, secure) • Fast Provisioning, Consistent Performance • Enterprise Ready • 24x7 Support • 99.95% monthly SLA • ISO 27001, SSAE-16 SOC 1,2,3 • Accessible Through • Web @ https://cloud.google.com/console • gcutil command-line utility • REST API • Partners (Commercial and FOSS) Google Compute Engine
15. Launching 50 virtual machines
16. Block Storage Persistent Disks
17. Persistent Disk 10 TB • Data and root partitions • Billed only by capacity (GB/month) • Performance caps scale linearly with size • Volume striping is automatic • Differential snapshots • Create new PDs based on snapshots
18. PD Use-cases Root Stateful root volume User managed data volume Root RW Data RO Data Instant distribution of static content
19. Global Snapshot and Restore GCS vol1 vol2 vol3 vol1. t2 vol2. t2 vol3. t3 t1 t1 t1 t1 t1 t1 t1 t1 t1 t1 t1 t1 t1 t2 t2 t2 t2 t3 t3 t3 • Point in time snapshot to Google Cloud Storage (GCS) • Differential snapshots • GCS global replication! • Restore from snapshot anywhere in the world us-central1-a europe-west1-a
20. Virtual Machines
21. standard • For workloads with balanced CPU and memory highmem • For workloads with higher memory requirements highcpu • For workloads with higher CPU requirements Shared Core Instances • For inexpensive prototyping and staging workloads (g1-small, f1-micro) Machine Types Machine Type Virtual Cores Memory (GB) n1-standard-1 1 3.75 n1-standard-2 2 7.50 n1-standard-4 4 15.00 n1-standard-8 8 30.00 n1-standard-16 16 60.00 n1-highmem-2 2 13.00 n1-highmem-4 4 26.00 n1-highmem-8 8 52.00 n1-highmem-16 16 104.00 n1-highcpu-2 2 1.80 n1-highcpu-4 4 3.60 n1-highcpu-8 8 7.20 n1-highcpu-16 16 14.40
22. Images + Custom Kernel Support (debian backports works great for docker)
23. Sub-hour Billing + Sustained Use Discount 1 minute granularity, 10 minute minimum $ (30%) Full billing cycle Sustained Use Discount • >25% use, price reduced • Up to 30% reduction • No commitment / contract
24. Enhanced Reliability us-central1-a scheduled maintenance event Live Migration • No downtime during scheduled datacenter maintenance events Automatic Restart • Instances automatically restarted if subjected to system events such as hardware failure
25. Networking
26. Images by Connie Zhou Networking • Google's Network • Projects are isolated private networks • TCP, UDP, ICMP only • Multiple private network groups and firewalls • Tags and address ranges • Addresses • public: static or ephemeral • private: ephemeral with DNS • Routes, gateways, VPNs, and IP Forwarding
27. Region: us-central1 Target Pool (tp-a) Load-balancer • Region based • Session affinity, hashing options on tuples, • Protocol, src ip:port, dst ip:port • Consists of: • Target pool: collection of instances • Forwarding rules: protocol:port, pool • HTTP health check: optional us-central1-b us-central1-a www0 www1 www2 www3 Forwarding Rules tcp:443 ➔ tp-a tcp:80 ➔ tp-a Internet http://googlecloudplatform.blogspot.com/2013/11/compute-engine-load-balancing-hits-1-million-requests-per-second.html
28. ...and
29. Noteworthy • Regions • Central United States, Europe, Asia (new!) • Metadata • Startup scripts • SSH Keys • Tags (instance/network) • OAuth2 and Scopes • Access other Google Cloud Platform services • Ecosystem is growing • Partners: RightScale, Scalr, New Relic, MongoLab, MapR, and many more... • Open Source: Chef, Puppet, Salt, Ansible, Vagrant, Docker, CoreOS, fog, libcloud
30. Whirlwind Tour of Google Cloud Platform Google Compute Engine Using Chef and Compute Engine (demo) 1 2 3
31. Chef Environment • Create a Chef Server in Google Compute Engine • I used CentOS 6 image + Chef's installation instructions • Create a Chef Workstation in Google Compute Engine • I used Debian 7 and Omnibus installer script • Create a Node in Google Compute Engine • I used knife-google
32. knife google # Getting set up knife google setup # Requires Google Client ID / Secret # The bootstrap command you know intimately (with a touch of Compute Engine) knife google server create from-knife -Z us-central1-a -m n1-standard-1 \ -I debian-7-wheezy-v20140408 -x $USER \ -i $HOME/.ssh/google_compute_engine \ -r 'apache2' # Other goodies like Persistent Disks, Zones, Regions - but not much else knife google --help
33. GCE LWRP 1. [DONE] Use the Developers Console to create the Chef Server and Workstation 2. [TODO] Use the NEW google-compute-engine LWRP, recipe, and chef-zero to: a. Create 4 instances (2 per zone) b. Create a firewall rule and Load-Balancer 3. [TODO] Generate some HTTP requests to LB IP Region: us-central1 Target Pool (lb-tp) us-central1-a myinstance1 myinstance3 us-central1-b myinstance2 myinstance4 Forwarding Rules tcp:80 ➔ lb-tp GCE API chef-workstation chef-server chef-client -z -o 'gce::demo-create'
34. The "demo-create" recipe demo-create 1 puts "Create two instances, split odd/even into different zones" 2 (1..4).each do |i| 3 gce_instance "#{name_prefix}#{i}" do 4 machine_type "n1-standard-1" 5 if i % 2 == 0 6 zone_name "#{zone_a}" 7 else 8 zone_name "#{zone_b}" 9 end 10 boot_disk_image "debian-7-wheezy-v20140408" 11 service_account_scopes ["compute", "userinfo.email", "devstorage.full_control"] 12 tags ["chefconf2014"] 13 metadata "demo"=>"chefconf2014", "foo"=>"bar" 14 auto_restart true 15 on_host_maintenance "MIGRATE" 16 # enable turbo mode! 17 wait_for false 18 # bootstrap attributes 19 first_boot_json "/home/erjohnso/first_boot_json" 20 client_rb "/home/erjohnso/client_rb" 21 validation_pem "/home/erjohnso/validation_pem" 22 action :create 23 end 24 servers << "#{name_prefix}#{i}" 25 end
35. The "demo-create" recipe demo-create 1 puts "Create two instances, split odd/even into different zones" 2 (1..4).each do |i| 3 gce_instance "#{name_prefix}#{i}" do 4 machine_type "n1-standard-1" 5 if i % 2 == 0 6 zone_name "#{zone_a}" 7 else 8 zone_name "#{zone_b}" 9 end 10 boot_disk_image "debian-7-wheezy-v20140408" 11 service_account_scopes ["compute", "userinfo.email", "devstorage.full_control"] 12 tags ["chefconf2014"] 13 metadata "demo"=>"chefconf2014", "foo"=>"bar" 14 auto_restart true 15 on_host_maintenance "MIGRATE" 16 # enable turbo mode! 17 wait_for false 18 # bootstrap attributes 19 first_boot_json "/home/erjohnso/first_boot_json" 20 client_rb "/home/erjohnso/client_rb" 21 validation_pem "/home/erjohnso/validation_pem" 22 action :create 23 end 24 servers << "#{name_prefix}#{i}" 25 end
36. The "demo-create" recipe demo-create 1 puts "Create two instances, split odd/even into different zones" 2 (1..4).each do |i| 3 gce_instance "#{name_prefix}#{i}" do 4 machine_type "n1-standard-1" 5 if i % 2 == 0 6 zone_name "#{zone_a}" 7 else 8 zone_name "#{zone_b}" 9 end 10 boot_disk_image "debian-7-wheezy-v20140408" 11 service_account_scopes ["compute", "userinfo.email", "devstorage.full_control"] 12 tags ["chefconf2014"] 13 metadata "demo"=>"chefconf2014", "foo"=>"bar" 14 auto_restart true 15 on_host_maintenance "MIGRATE" 16 # enable turbo mode! 17 wait_for false 18 # bootstrap attributes 19 first_boot_json "/home/erjohnso/first_boot_json" 20 client_rb "/home/erjohnso/client_rb" 21 validation_pem "/home/erjohnso/validation_pem" 22 action :create 23 end 24 servers << "#{name_prefix}#{i}" 25 end
37. The "demo-create" recipe demo-create 1 puts "Create two instances, split odd/even into different zones" 2 (1..4).each do |i| 3 gce_instance "#{name_prefix}#{i}" do 4 machine_type "n1-standard-1" 5 if i % 2 == 0 6 zone_name "#{zone_a}" 7 else 8 zone_name "#{zone_b}" 9 end 10 boot_disk_image "debian-7-wheezy-v20140408" 11 service_account_scopes ["compute", "userinfo.email", "devstorage.full_control"] 12 tags ["chefconf2014"] 13 metadata "demo"=>"chefconf2014", "foo"=>"bar" 14 auto_restart true 15 on_host_maintenance "MIGRATE" 16 # enable turbo mode! 17 wait_for false 18 # bootstrap attributes 19 first_boot_json "/home/erjohnso/first_boot_json" 20 client_rb "/home/erjohnso/client_rb" 21 validation_pem "/home/erjohnso/validation_pem" 22 action :create 23 end 24 servers << "#{name_prefix}#{i}" 25 end
38. The "demo-create" recipe demo-create 1 puts "Create two instances, split odd/even into different zones" 2 (1..4).each do |i| 3 gce_instance "#{name_prefix}#{i}" do 4 machine_type "n1-standard-1" 5 if i % 2 == 0 6 zone_name "#{zone_a}" 7 else 8 zone_name "#{zone_b}" 9 end 10 boot_disk_image "debian-7-wheezy-v20140408" 11 service_account_scopes ["compute", "userinfo.email", "devstorage.full_control"] 12 tags ["chefconf2014"] 13 metadata "demo"=>"chefconf2014", "foo"=>"bar" 14 auto_restart true 15 on_host_maintenance "MIGRATE" 16 # enable turbo mode! 17 wait_for false 18 # bootstrap attributes 19 first_boot_json "/home/erjohnso/first_boot_json" 20 client_rb "/home/erjohnso/client_rb" 21 validation_pem "/home/erjohnso/validation_pem" 22 action :create 23 end 24 servers << "#{name_prefix}#{i}" 25 end
39. The "demo-create" recipe demo-create 1 puts "Create two instances, split odd/even into different zones" 2 (1..4).each do |i| 3 gce_instance "#{name_prefix}#{i}" do 4 machine_type "n1-standard-1" 5 if i % 2 == 0 6 zone_name "#{zone_a}" 7 else 8 zone_name "#{zone_b}" 9 end 10 boot_disk_image "debian-7-wheezy-v20140408" 11 service_account_scopes ["compute", "userinfo.email", "devstorage.full_control"] 12 tags ["chefconf2014"] 13 metadata "demo"=>"chefconf2014", "foo"=>"bar" 14 auto_restart true 15 on_host_maintenance "MIGRATE" 16 # enable turbo mode! 17 wait_for false 18 # bootstrap attributes 19 first_boot_json "/home/erjohnso/first_boot_json" 20 client_rb "/home/erjohnso/client_rb" 21 validation_pem "/home/erjohnso/validation_pem" 22 action :create 23 end 24 servers << "#{name_prefix}#{i}" 25 end
40. The "demo-create" recipe (continued) demo-create 1 gce_firewall "cc-allow-http" do 2 network "default" 3 allowed_ports [80] 4 end 5 6 gce_lb_healthcheck "cc-hc" do 7 request_path "/" 8 port 80 9 action :create 10 end 11 12 gce_lb_targetpool "cc-tp" do 13 region "#{my_region}" 14 instances servers 15 health_checks ["cc-hc"] 16 action :create 17 end 18 19 gce_lb_forwardingrule "cc-fr" do 20 region "#{my_region}" 21 ip_protocol "TCP" 22 port_range "80-8080" 23 target_pool "cc-tp" 24 action :create 25 end
41. The "demo-create" recipe (continued) demo-create 1 gce_firewall "cc-allow-http" do 2 network "default" 3 allowed_ports [80] 4 end 5 6 gce_lb_healthcheck "cc-hc" do 7 request_path "/" 8 port 80 9 action :create 10 end 11 12 gce_lb_targetpool "cc-tp" do 13 region "#{my_region}" 14 instances servers 15 health_checks ["cc-hc"] 16 action :create 17 end 18 19 gce_lb_forwardingrule "cc-fr" do 20 region "#{my_region}" 21 ip_protocol "TCP" 22 port_range "80-8080" 23 target_pool "cc-tp" 24 action :create 25 end
42. The "demo-create" recipe (continued) demo-create 1 gce_firewall "cc-allow-http" do 2 network "default" 3 allowed_ports [80] 4 end 5 6 gce_lb_healthcheck "cc-hc" do 7 request_path "/" 8 port 80 9 action :create 10 end 11 12 gce_lb_targetpool "cc-tp" do 13 region "#{my_region}" 14 instances servers 15 health_checks ["cc-hc"] 16 action :create 17 end 18 19 gce_lb_forwardingrule "cc-fr" do 20 region "#{my_region}" 21 ip_protocol "TCP" 22 port_range "80-8080" 23 target_pool "cc-tp" 24 action :create 25 end
43. The "demo-create" recipe (continued) demo-create 1 gce_firewall "cc-allow-http" do 2 network "default" 3 allowed_ports [80] 4 end 5 6 gce_lb_healthcheck "cc-hc" do 7 request_path "/" 8 port 80 9 action :create 10 end 11 12 gce_lb_targetpool "cc-tp" do 13 region "#{my_region}" 14 instances servers 15 health_checks ["cc-hc"] 16 action :create 17 end 18 19 gce_lb_forwardingrule "cc-fr" do 20 region "#{my_region}" 21 ip_protocol "TCP" 22 port_range "80-8080" 23 target_pool "cc-tp" 24 action :create 25 end
44. The "demo-create" recipe (continued) demo-create 1 gce_firewall "cc-allow-http" do 2 network "default" 3 allowed_ports [80] 4 end 5 6 gce_lb_healthcheck "cc-hc" do 7 request_path "/" 8 port 80 9 action :create 10 end 11 12 gce_lb_targetpool "cc-tp" do 13 region "#{my_region}" 14 instances servers 15 health_checks ["cc-hc"] 16 action :create 17 end 18 19 gce_lb_forwardingrule "cc-fr" do 20 region "#{my_region}" 21 ip_protocol "TCP" 22 port_range "80-8080" 23 target_pool "cc-tp" 24 action :create 25 end
45. Available now and more coming! • Bootstrap with knife-google • Create, destroy instances • Create, destroy Persistent Disks Available Now!! Coming Soon!! • The google-compute-engine LWRP • create, destroy, and bootstrap instances • create, destroy, and snapshot persistent disks • Plus... Set instance tags and metadata • Plus... Networks and Firewall Rules • Plus... Load-balancer and HTTP Health Checks
46. Want More? Come see another working demo at the Google booth! A Developer's and Chef Administrator's best friend... • A self-service provisioning portal using Compute Engine • Developers can specify Cookbooks for bootstrap • Administrative management of templates / cookbooks
47. cloud.google.com Try it for yourself: $500 credit http://cloud.google.com/starterpack - promo code chef14-con Read more at • Compute Engine: https://cloud.google.com/products/compute-engine • Chef + Compute Engine: http://www.getchef.com/solutions/google-cloud-platform/ Use, collaborate, and improve • knife-google: http://docs.opscode.com/plugin_knife_google.html • GCE LWRP: https://github.com/chef-partners/google-compute-engine (coming soon!) • Test Kitchen GCE: http://andyleonard.com/tag/kitchen-gce/ (thanks Andrew Leonard!) Questions? Find me at Speaker Office Hours Today, 4:15-4:35 Or on github, @erjohnso ^_^