Share
Embed
ChefConf2014: Google+Chef
Covers Google Cloud Platform walk through, deeper dive on Compute Engine, and shows knife-google and GCE LWRP demos.
video: https://www.youtube.com/watch?v=D2OICR18zIo
Eric Johnson
April 16, 2014
Transcript
47 slides
-
Agenda: April 16th, 2014 Whirlwind Tour of Google Cloud Platform
Google Compute Engine Using Chef and Compute Engine (demo) 1 2 3 -
Whirlwind Tour of Google Cloud Platform Google Compute Engine Using
SaltStack and Compute Engine (demo) 1 2 3 -
Storage Cloud Storage Cloud SQL Cloud Datastore Compute Compute Engine
App Engine App Services BigQuery Cloud Endpoints Google Cloud Platform Cloud DNS -
Compute Compute Engine App Engine Storage Cloud Storage Cloud SQL
Cloud Datastore Power Computation App Services BigQuery Cloud Endpoints Cloud DNS -
Storage Cloud Storage Cloud SQL Cloud Datastore Storing What You
Want, However You Want Compute Compute Engine App Engine App Services BigQuery Cloud Endpoints Cloud DNS -
App Services BigQuery Cloud Endpoints Building Robust, Intelligent Systems Storage
Cloud Storage Cloud SQL Cloud Datastore Compute Compute Engine App Engine Cloud DNS -
For the past 15 years, Google has been building out
the world’s fastest, most powerful, highest quality cloud infrastructure on the planet. Images by Connie Zhou Why Google Cloud Platform? -
Innovating Software & Driving Technology Forward Spanner Dremel MapReduce Big
Table Colossus 2012 2013 2002 2004 2006 2008 2010 GFS Compute Engine -
“[Google's] ability to build, organize, and operate a huge network
of servers and fiber- optic cables with an efficiency and speed that rocks physics on its heels. This is what makes Google Google: its physical network, its thousands of fiber miles, and those many thousands of servers that, in aggregate, add up to the mother of all clouds.” - Wired Images by Connie Zhou -
Whirlwind Tour of Google Cloud Platform Google Compute Engine Using
SaltStack and Compute Engine (demo) 1 2 3 -
• IaaS: VMs, Network, Storage • Google DNA (speed, scale,
reliable, secure) • Fast Provisioning, Consistent Performance • Enterprise Ready • 24x7 Support • 99.95% monthly SLA • ISO 27001, SSAE-16 SOC 1,2,3 • Accessible Through • Web @ https://cloud.google.com/console • gcutil command-line utility • REST API • Partners (Commercial and FOSS) Google Compute Engine -
Persistent Disk 10 TB • Data and root partitions •
Billed only by capacity (GB/month) • Performance caps scale linearly with size • Volume striping is automatic • Differential snapshots • Create new PDs based on snapshots -
PD Use-cases Root Stateful root volume User managed data volume
Root RW Data RO Data Instant distribution of static content -
Global Snapshot and Restore GCS vol1 vol2 vol3 vol1. t2
vol2. t2 vol3. t3 t1 t1 t1 t1 t1 t1 t1 t1 t1 t1 t1 t1 t1 t2 t2 t2 t2 t3 t3 t3 • Point in time snapshot to Google Cloud Storage (GCS) • Differential snapshots • GCS global replication! • Restore from snapshot anywhere in the world us-central1-a europe-west1-a -
standard • For workloads with balanced CPU and memory highmem
• For workloads with higher memory requirements highcpu • For workloads with higher CPU requirements Shared Core Instances • For inexpensive prototyping and staging workloads (g1-small, f1-micro) Machine Types Machine Type Virtual Cores Memory (GB) n1-standard-1 1 3.75 n1-standard-2 2 7.50 n1-standard-4 4 15.00 n1-standard-8 8 30.00 n1-standard-16 16 60.00 n1-highmem-2 2 13.00 n1-highmem-4 4 26.00 n1-highmem-8 8 52.00 n1-highmem-16 16 104.00 n1-highcpu-2 2 1.80 n1-highcpu-4 4 3.60 n1-highcpu-8 8 7.20 n1-highcpu-16 16 14.40 -
Sub-hour Billing + Sustained Use Discount 1 minute granularity, 10
minute minimum $ (30%) Full billing cycle Sustained Use Discount • >25% use, price reduced • Up to 30% reduction • No commitment / contract -
Enhanced Reliability us-central1-a scheduled maintenance event Live Migration • No
downtime during scheduled datacenter maintenance events Automatic Restart • Instances automatically restarted if subjected to system events such as hardware failure -
Images by Connie Zhou Networking • Google's Network • Projects
are isolated private networks • TCP, UDP, ICMP only • Multiple private network groups and firewalls • Tags and address ranges • Addresses • public: static or ephemeral • private: ephemeral with DNS • Routes, gateways, VPNs, and IP Forwarding -
Region: us-central1 Target Pool (tp-a) Load-balancer • Region based •
Session affinity, hashing options on tuples, • Protocol, src ip:port, dst ip:port • Consists of: • Target pool: collection of instances • Forwarding rules: protocol:port, pool • HTTP health check: optional us-central1-b us-central1-a www0 www1 www2 www3 Forwarding Rules tcp:443 ➔ tp-a tcp:80 ➔ tp-a Internet http://googlecloudplatform.blogspot.com/2013/11/compute-engine-load-balancing-hits-1-million-requests-per-second.html -
Noteworthy • Regions • Central United States, Europe, Asia (new!)
• Metadata • Startup scripts • SSH Keys • Tags (instance/network) • OAuth2 and Scopes • Access other Google Cloud Platform services • Ecosystem is growing • Partners: RightScale, Scalr, New Relic, MongoLab, MapR, and many more... • Open Source: Chef, Puppet, Salt, Ansible, Vagrant, Docker, CoreOS, fog, libcloud -
Whirlwind Tour of Google Cloud Platform Google Compute Engine Using
Chef and Compute Engine (demo) 1 2 3 -
Chef Environment • Create a Chef Server in Google Compute
Engine • I used CentOS 6 image + Chef's installation instructions • Create a Chef Workstation in Google Compute Engine • I used Debian 7 and Omnibus installer script • Create a Node in Google Compute Engine • I used knife-google -
knife google # Getting set up knife google setup #
Requires Google Client ID / Secret # The bootstrap command you know intimately (with a touch of Compute Engine) knife google server create from-knife -Z us-central1-a -m n1-standard-1 \ -I debian-7-wheezy-v20140408 -x $USER \ -i $HOME/.ssh/google_compute_engine \ -r 'apache2' # Other goodies like Persistent Disks, Zones, Regions - but not much else knife google --help -
GCE LWRP 1. [DONE] Use the Developers Console to create
the Chef Server and Workstation 2. [TODO] Use the NEW google-compute-engine LWRP, recipe, and chef-zero to: a. Create 4 instances (2 per zone) b. Create a firewall rule and Load-Balancer 3. [TODO] Generate some HTTP requests to LB IP Region: us-central1 Target Pool (lb-tp) us-central1-a myinstance1 myinstance3 us-central1-b myinstance2 myinstance4 Forwarding Rules tcp:80 ➔ lb-tp GCE API chef-workstation chef-server chef-client -z -o 'gce::demo-create' -
The "demo-create" recipe demo-create 1 puts "Create two instances, split
odd/even into different zones" 2 (1..4).each do |i| 3 gce_instance "#{name_prefix}#{i}" do 4 machine_type "n1-standard-1" 5 if i % 2 == 0 6 zone_name "#{zone_a}" 7 else 8 zone_name "#{zone_b}" 9 end 10 boot_disk_image "debian-7-wheezy-v20140408" 11 service_account_scopes ["compute", "userinfo.email", "devstorage.full_control"] 12 tags ["chefconf2014"] 13 metadata "demo"=>"chefconf2014", "foo"=>"bar" 14 auto_restart true 15 on_host_maintenance "MIGRATE" 16 # enable turbo mode! 17 wait_for false 18 # bootstrap attributes 19 first_boot_json "/home/erjohnso/first_boot_json" 20 client_rb "/home/erjohnso/client_rb" 21 validation_pem "/home/erjohnso/validation_pem" 22 action :create 23 end 24 servers << "#{name_prefix}#{i}" 25 end -
The "demo-create" recipe demo-create 1 puts "Create two instances, split
odd/even into different zones" 2 (1..4).each do |i| 3 gce_instance "#{name_prefix}#{i}" do 4 machine_type "n1-standard-1" 5 if i % 2 == 0 6 zone_name "#{zone_a}" 7 else 8 zone_name "#{zone_b}" 9 end 10 boot_disk_image "debian-7-wheezy-v20140408" 11 service_account_scopes ["compute", "userinfo.email", "devstorage.full_control"] 12 tags ["chefconf2014"] 13 metadata "demo"=>"chefconf2014", "foo"=>"bar" 14 auto_restart true 15 on_host_maintenance "MIGRATE" 16 # enable turbo mode! 17 wait_for false 18 # bootstrap attributes 19 first_boot_json "/home/erjohnso/first_boot_json" 20 client_rb "/home/erjohnso/client_rb" 21 validation_pem "/home/erjohnso/validation_pem" 22 action :create 23 end 24 servers << "#{name_prefix}#{i}" 25 end -
The "demo-create" recipe demo-create 1 puts "Create two instances, split
odd/even into different zones" 2 (1..4).each do |i| 3 gce_instance "#{name_prefix}#{i}" do 4 machine_type "n1-standard-1" 5 if i % 2 == 0 6 zone_name "#{zone_a}" 7 else 8 zone_name "#{zone_b}" 9 end 10 boot_disk_image "debian-7-wheezy-v20140408" 11 service_account_scopes ["compute", "userinfo.email", "devstorage.full_control"] 12 tags ["chefconf2014"] 13 metadata "demo"=>"chefconf2014", "foo"=>"bar" 14 auto_restart true 15 on_host_maintenance "MIGRATE" 16 # enable turbo mode! 17 wait_for false 18 # bootstrap attributes 19 first_boot_json "/home/erjohnso/first_boot_json" 20 client_rb "/home/erjohnso/client_rb" 21 validation_pem "/home/erjohnso/validation_pem" 22 action :create 23 end 24 servers << "#{name_prefix}#{i}" 25 end -
The "demo-create" recipe demo-create 1 puts "Create two instances, split
odd/even into different zones" 2 (1..4).each do |i| 3 gce_instance "#{name_prefix}#{i}" do 4 machine_type "n1-standard-1" 5 if i % 2 == 0 6 zone_name "#{zone_a}" 7 else 8 zone_name "#{zone_b}" 9 end 10 boot_disk_image "debian-7-wheezy-v20140408" 11 service_account_scopes ["compute", "userinfo.email", "devstorage.full_control"] 12 tags ["chefconf2014"] 13 metadata "demo"=>"chefconf2014", "foo"=>"bar" 14 auto_restart true 15 on_host_maintenance "MIGRATE" 16 # enable turbo mode! 17 wait_for false 18 # bootstrap attributes 19 first_boot_json "/home/erjohnso/first_boot_json" 20 client_rb "/home/erjohnso/client_rb" 21 validation_pem "/home/erjohnso/validation_pem" 22 action :create 23 end 24 servers << "#{name_prefix}#{i}" 25 end -
The "demo-create" recipe demo-create 1 puts "Create two instances, split
odd/even into different zones" 2 (1..4).each do |i| 3 gce_instance "#{name_prefix}#{i}" do 4 machine_type "n1-standard-1" 5 if i % 2 == 0 6 zone_name "#{zone_a}" 7 else 8 zone_name "#{zone_b}" 9 end 10 boot_disk_image "debian-7-wheezy-v20140408" 11 service_account_scopes ["compute", "userinfo.email", "devstorage.full_control"] 12 tags ["chefconf2014"] 13 metadata "demo"=>"chefconf2014", "foo"=>"bar" 14 auto_restart true 15 on_host_maintenance "MIGRATE" 16 # enable turbo mode! 17 wait_for false 18 # bootstrap attributes 19 first_boot_json "/home/erjohnso/first_boot_json" 20 client_rb "/home/erjohnso/client_rb" 21 validation_pem "/home/erjohnso/validation_pem" 22 action :create 23 end 24 servers << "#{name_prefix}#{i}" 25 end -
The "demo-create" recipe demo-create 1 puts "Create two instances, split
odd/even into different zones" 2 (1..4).each do |i| 3 gce_instance "#{name_prefix}#{i}" do 4 machine_type "n1-standard-1" 5 if i % 2 == 0 6 zone_name "#{zone_a}" 7 else 8 zone_name "#{zone_b}" 9 end 10 boot_disk_image "debian-7-wheezy-v20140408" 11 service_account_scopes ["compute", "userinfo.email", "devstorage.full_control"] 12 tags ["chefconf2014"] 13 metadata "demo"=>"chefconf2014", "foo"=>"bar" 14 auto_restart true 15 on_host_maintenance "MIGRATE" 16 # enable turbo mode! 17 wait_for false 18 # bootstrap attributes 19 first_boot_json "/home/erjohnso/first_boot_json" 20 client_rb "/home/erjohnso/client_rb" 21 validation_pem "/home/erjohnso/validation_pem" 22 action :create 23 end 24 servers << "#{name_prefix}#{i}" 25 end -
The "demo-create" recipe (continued) demo-create 1 gce_firewall "cc-allow-http" do 2
network "default" 3 allowed_ports [80] 4 end 5 6 gce_lb_healthcheck "cc-hc" do 7 request_path "/" 8 port 80 9 action :create 10 end 11 12 gce_lb_targetpool "cc-tp" do 13 region "#{my_region}" 14 instances servers 15 health_checks ["cc-hc"] 16 action :create 17 end 18 19 gce_lb_forwardingrule "cc-fr" do 20 region "#{my_region}" 21 ip_protocol "TCP" 22 port_range "80-8080" 23 target_pool "cc-tp" 24 action :create 25 end -
The "demo-create" recipe (continued) demo-create 1 gce_firewall "cc-allow-http" do 2
network "default" 3 allowed_ports [80] 4 end 5 6 gce_lb_healthcheck "cc-hc" do 7 request_path "/" 8 port 80 9 action :create 10 end 11 12 gce_lb_targetpool "cc-tp" do 13 region "#{my_region}" 14 instances servers 15 health_checks ["cc-hc"] 16 action :create 17 end 18 19 gce_lb_forwardingrule "cc-fr" do 20 region "#{my_region}" 21 ip_protocol "TCP" 22 port_range "80-8080" 23 target_pool "cc-tp" 24 action :create 25 end -
The "demo-create" recipe (continued) demo-create 1 gce_firewall "cc-allow-http" do 2
network "default" 3 allowed_ports [80] 4 end 5 6 gce_lb_healthcheck "cc-hc" do 7 request_path "/" 8 port 80 9 action :create 10 end 11 12 gce_lb_targetpool "cc-tp" do 13 region "#{my_region}" 14 instances servers 15 health_checks ["cc-hc"] 16 action :create 17 end 18 19 gce_lb_forwardingrule "cc-fr" do 20 region "#{my_region}" 21 ip_protocol "TCP" 22 port_range "80-8080" 23 target_pool "cc-tp" 24 action :create 25 end -
The "demo-create" recipe (continued) demo-create 1 gce_firewall "cc-allow-http" do 2
network "default" 3 allowed_ports [80] 4 end 5 6 gce_lb_healthcheck "cc-hc" do 7 request_path "/" 8 port 80 9 action :create 10 end 11 12 gce_lb_targetpool "cc-tp" do 13 region "#{my_region}" 14 instances servers 15 health_checks ["cc-hc"] 16 action :create 17 end 18 19 gce_lb_forwardingrule "cc-fr" do 20 region "#{my_region}" 21 ip_protocol "TCP" 22 port_range "80-8080" 23 target_pool "cc-tp" 24 action :create 25 end -
The "demo-create" recipe (continued) demo-create 1 gce_firewall "cc-allow-http" do 2
network "default" 3 allowed_ports [80] 4 end 5 6 gce_lb_healthcheck "cc-hc" do 7 request_path "/" 8 port 80 9 action :create 10 end 11 12 gce_lb_targetpool "cc-tp" do 13 region "#{my_region}" 14 instances servers 15 health_checks ["cc-hc"] 16 action :create 17 end 18 19 gce_lb_forwardingrule "cc-fr" do 20 region "#{my_region}" 21 ip_protocol "TCP" 22 port_range "80-8080" 23 target_pool "cc-tp" 24 action :create 25 end -
Available now and more coming! • Bootstrap with knife-google •
Create, destroy instances • Create, destroy Persistent Disks Available Now!! Coming Soon!! • The google-compute-engine LWRP • create, destroy, and bootstrap instances • create, destroy, and snapshot persistent disks • Plus... Set instance tags and metadata • Plus... Networks and Firewall Rules • Plus... Load-balancer and HTTP Health Checks -
Want More? Come see another working demo at the Google
booth! A Developer's and Chef Administrator's best friend... • A self-service provisioning portal using Compute Engine • Developers can specify Cookbooks for bootstrap • Administrative management of templates / cookbooks -
cloud.google.com Try it for yourself: $500 credit http://cloud.google.com/starterpack - promo
code chef14-con Read more at • Compute Engine: https://cloud.google.com/products/compute-engine • Chef + Compute Engine: http://www.getchef.com/solutions/google-cloud-platform/ Use, collaborate, and improve • knife-google: http://docs.opscode.com/plugin_knife_google.html • GCE LWRP: https://github.com/chef-partners/google-compute-engine (coming soon!) • Test Kitchen GCE: http://andyleonard.com/tag/kitchen-gce/ (thanks Andrew Leonard!) Questions? Find me at Speaker Office Hours Today, 4:15-4:35 Or on github, @erjohnso ^_^