Share
Embed
Config / Managment Tools and Google Compute Engine
Slides for a talk given to the Seattle DevOps meetup on Oct 29th 2013.
Eric Johnson
October 29, 2013
Transcript
25 slides
-
Config / Mgmnt Tools and Google Cloud Platform Eric Johnson
Program Manager, Google Compute Engine -
Cloud Platform Agenda • Why Google? • Whirlwind tour of
Google Cloud Platform • Google Compute Engine Instance with: • Vagrant • Chef • Puppet • Ansible -
For the past 15 years, Google has been building out
a massively fast, powerful, and reliable cloud infrastructure across the planet. Images by Connie Zhou -
Cloud Platform Developing our infrastructure while respecting our ecosystem •
Pioneering data center efficiency • Financed over 250 Megawatts of new wind power • First data centers to receive IOS 14001 certification • 100% carbon neutral -
2002 2004 2006 2008 2010 2012 Colossus Dremel MapReduce Spanner
Big Table GFS Driving Technology Forward Cloud Platform -
Google Cloud Platform Storage Cloud Storage Cloud SQL Cloud Datastore
Persistent Disk App Services BigQuery Cloud Endpoints Caching Queues Cloud Platform Compute App Engine Compute Engine -
Cloud Platform • BLOB Storage, Immutable Objects • Strong read-after-write
consistency • API and Web UI Accessible • Versioning • Static Sites, ACLs • Resumable Transfers • Object Change Notifications • Object lifecycle management Google Cloud Storage -
• Fully managed, MySQL(like) • Ease of Use and Development
• Highly Reliable • Flexible Charging • Security, Availability, Durability • EU and US Data Centers • Easy Migration & Data Portability • Control Cloud Platform Cloud SQL -
Cloud Platform • Schemaless, Non-relational NoSQL Access • Auto-scale •
Authentication That Just Works • Fast and Easy Provisioning • RESTful Endpoints • ACID Transactions • Query Language (akin to SQL) • Local Development Tools • Built-in Redundancy Cloud Datastore -
Cloud Platform • Fully Managed Big Data Analytics Service •
Fast • Scalable • Flexible and Familiar • Security and Reliability BigQuery -
Cloud Platform • Fully Managed Platform • Easy Development &
Deployment • Focus On Your Code Not Your Server • Automatic Scaling • Popular Programming Language Support • Services (Cron, Queue, Memcache, etc) • Datastore • Versioning and Traffic Splitting • Local Developer Tools • Third-party Frameworks and Extensions App Engine -
Cloud Platform • Sub-hour Billing • Up to 10TB Persistent
Disk • Over 64 Instance Types • Standard Linux Distributions • Advanced Networking • Instance Metadata and Startup Scripts • Load Balancing • Persistent Disks, snapshots • Fast and Easy Provisioning • Consistent Performance Google Compute Engine -
Cloud Platform Demo Time Times Four! • Not an exhaustive
dive into each tool, but we’ll watch a new GCE instance created in: ◦ Vagrant ◦ Chef ◦ Puppet ◦ Ansible • We’ll skip all the gory setup details, but they’re included in the slide deck if you’re curious -
Cloud Platform Vagrant (vagrant-google) $ cat -n Vagrantfile 1 Vagrant.configure("2")
do |config| 2 config.vm.box = "gce" 3 config.vm.provision :shell, :privileged => false, :inline => $PROVISION_NODE 3 4 config.vm.provider :google do |google, override| 5 google.google_project_id = $GOOGLE_PROJECT_ID 6 google.google_client_email = $GOOGLE_CLIENT_EMAIL 7 google.google_key_location = $GOOGLE_KEY_LOCATION 8 9 google.name = "via-vagrant" 10 google.zone = "us-central1-b" 11 google.image = "debian-7-wheezy-v20130926" 12 google.machine_type = "n1-standard-1" 13 14 override.ssh.username = $LOCAL_USER 15 override.ssh.private_key_path = $LOCAL_SSH_KEY 16 end 17 end $ vagrant up --provider=google $ vagrant destroy -f -
Cloud Platform Chef (knife-google) 1 erjohnso@chef:~$ knife google help 2
Available google subcommands: (for details, knife SUB-COMMAND --help) 3 4 ** GOOGLE COMMANDS ** 5 knife google disk create NAME --google-disk-sizeGb N --google-compute-zone ZONE (options) 6 knife google disk delete NAME --google-compute-zone ZONE 7 knife google disk list --google-compute-zone ZONE (options) 8 knife google server create NAME -m MACHINE_TYPE -I IMAGE -Z ZONE (options) 9 knife google server delete SERVER [SERVER] --google-compute-zone ZONE (options) 10 knife google server list --google-compute-zone ZONE (options) 11 knife google setup 12 knife google zone list (options) $ knife google server create via-chef -m n1-standard-1 \ > -I debian-7-wheezy-v20130926 -Z us-central1-b \ > -x erjohnso -i ~/.ssh/google_compute_engine \ > -s https://chef-server...com.internal:443 \ > -r “recipe[apt],recipe[apache2]” $ knife google server delete via-chef -Z us-central1-b \ > --purge -
Cloud Platform Puppet (gce_compute) erjohnso@puppet$ cat -n .puppet/device.conf create.pp 1
[gce_puppet] 2 type gce 3 url [/dev/null]:google.com:erjohnso 4 5 gce_instance { "via-puppet": ensure => present, machine_type => 'n1-standard-1', 6 zone => 'us-central1-b', network => 'default', 7 image => 'projects/debian-cloud/global/images/debian-7-wheezy-v20130926', 8 manifest => 'class apache ($version = "latest") { 9 package {"apache2": ensure => $version, } 10 file {"/var/www/index.html": ensure => present, require => Package["apache2"], 11 content => "<html><body><pre>Hi, this is $gce_external_ip.</pre></body></html>", 12 } 13 service {"apache2": ensure => running, enable => true, 14 require => File["/var/www/index.html"], 15 } 16 } 17 include apache' 18 } $ puppet apply --certname gce_puppet ./create.pp $ puppet apply --certname gce_puppet ./destroy.pp -
Cloud Platform Ansible (gce* modules) erjohnso@ansible$ cat -n hosts.ini gce-up.yml
1 [localhost] 2 127.0.0.1 3 [gce_instance] 4 via-ansible 5 6 - name: Bring up a GCE instance with ansible 7 hosts: localhost 8 tasks: 9 - name: Launch new instance 10 local_action: gce name=via-ansible machine_type=n1-standard-1 11 image=centos-6 zone=us-central1-b 12 - name: Update instance 13 hosts: gce_instance 14 tasks: 15 - name: Install apache 16 yum: pkg=httpd state=present $ ansible localhost -m gce \ > -a “name=via-ansible zone=us-central1-b” $ ansible localhost -m gce \ > -a “name=via-ansible zone=us-central1-b state=absent” -
Cloud Platform References • Chef (knife-google): https://github.com/opscode/knife-google • Puppet (gce_compute):
https://github.com/puppetlabs/puppetlabs-gce_compute • Also, but not covered (node_gce): https://github.com/puppetlabs/puppetlabs-node_gce • Vagrant (vagrant-google): https://github.com/GoogleCloudPlatform/vagrant-google • Ansible (in core): https://github.com/ansible/ansible • Salt... Coming soon! -
Cloud Platform And finally... DevOps + GCE = Awesome! •
Use the platform and send us feedback ◦ https://cloud.google.com/ ◦ $2,000 Credit - Use “ansf-con” promo code at https: //cloud.google.com/starterpack • Help improve Google module support Thank you! -
Cloud Platform Vagrant Setup Workstation: 1. // Create GCE Debian-7
instance, name it ‘chef-workstation’ 2. $ sudo apt-get update && sudo apt-get upgrade -y 3. $ sudo apt-get install git vim build-essential zlib1g-dev libssl-dev build-essential zlib1g-dev libssl-dev ruby1.9.1-dev -y 4. $ sudo gem1.9.1 install --no-ri --no-rdoc bundler 5. $ wget http://files.vagrantup.com/packages/.../vagrant_x.y.z_x86_64.deb 6. $ sudo dpkg -i vagrant_x.y.z_x86_64.deb 7. $ git clone https://github.com/GoogleCloudPlatform/vagrant-google.git 8. $ cd vagrant-google 9. $ bundle 10. $ gem build vagrant-google.gemspec 11. $ vagrant plugin install vagrant-google-0.1.1.gem 12. $ vagrant box add gce google.box 13. // Create the Vagrantfile 14. $ vagrant up --provider=google 15. $ vagrant destroy -f -
Cloud Platform Chef Setup Server: 1. // Create GCE CentOS-6
instance, name it ‘chef-server’ 2. $ wget https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-server-11.0.8-1.el6.x86_64.rpm 3. $ sudo rpm -i chef-server-11.0.8.el6.x86_64.rpm 4. $ sudo chef-server-ctl reconfigure && sleep 30 && sudo chef-server-ctl test Workstation: 1. // Create GCE Debian-7 instance, name it ‘chef-workstation’ 2. $ sudo apt-get update && sudo apt-get upgrade -y 3. $ curl -L https://www.opscode.com/chef/install.sh | sudo bash 4. // cp chef-server:/etc/chef-server/admin.pem chef-workstation:/etc/chef-server/admin.pem 5. // cp chef-server:/etc/chef-server/chef-validator.pem chef-workstation:/etc/chef-server/chef-validator.pem 6. $ git clone git://github.com/opscode/chef-repo.git 7. $ knife configure -i # server https://chef-server.c.erjohnso.google.com.internal:443, cookbook_path = ~/chef-repo/cookbooks 8. // Verify that it’s working on the workstation by running, 9. $ knife client list 10. $ knife user list 11. $ knife cookbook site install apt # repeat for ‘apache2’ 12. $ knife cookbook upload apt apache2 13. // Install knife-google on the workstation by, 14. $ sudo /opt/chef/embedded/bin/gem install knife-google 15. // Register App on cloud console, ‘Web Application’ -> ‘OAuth2.0 Client ID’ to generate Client ID and Client Secret 16. $ knife google setup # use your Project ID, Client ID, and Client Secret 17. $ knife google server list -Z us-central1-b 18. $ gcutil ssh `hostname -s` 19. $ knife google server create node1 -m n1-standard-1 -I debian-7-wheezy-v20130926 -Z us-central1-b -x erjohnso -i ~/.ssh/google_compute_engine -
Cloud Platform Puppet Setup Workstation: 1. // Create GCE Debian-7
instance 2. $ sudo apt-get update && sudo apt-get upgrade -y 3. $ sudo apt-get install puppet 4. $ mkdir -p ~/.puppet/modules 5. $ puppet module install puppetlabs-gce_compute 6. $ gctuil version # 1.8.4 which is fine since current gce_compute module relies on 1.8.3 7. $ gcutil ssh `hostname -s` # register my ssh key with metadata server 8. $ cat <<EOF > ~/.puppet/device.conf 9. [gce_puppet] 10. type gce 11. url [/dev/null]:google.com:erjohnso 12. $ cat <<EOF > ~/create.pp 13. gce_instance { “via-puppet”: ensure => present, machine_type => ‘n1-standard-1’, zone => ‘us-central1-b’, network => ‘default’, 14. image => ‘projects/debian-cloud/global/images/debian-7-wheezy-v20130926’, tags => [‘web’], 15. manifest => 'class apache ($version = "latest") { 16. package {"apache2": ensure => $version, } 17. file {"/var/www/index.html": ensure => present, require => Package["apache2"], 18. content => "<html><body><pre>Hi, this is $gce_external_ip.</pre></body></html>", 19. } 20. service {"apache2": ensure => running, enable => true, require => File["/var/www/index.html"], } 21. } 22. include apache' 23. } 24. $ puppet apply --certname gce_puppet ./create.pp -
Cloud Platform Ansible Setup Workstation: 1. // Create GCE Debian-7
instance 2. $ sudo apt-get update && sudo apt-get upgrade -y 3. $ sudo apt-get install python-paramiko python-yaml python-jinja2 python-pycryptopp git -y 4. $ git clone https://github.com/apache/libcloud # currently uses dev branch of libcloud until 0.14+ is released, then a pip install could be used 5. $ cd libcloud; sudo python setup.py install; cd ~ 6. $ cp libcloud/demos/secrets-dist.py ~/secrets.py 7. // edit ~/secrets.py and update GCE_PARAMS, GCE_KEYWORD_PARMS, use Service Account email_address and private_key location, and project_id 8. // make sure secrets.py is in your PYTHONPATH 9. // convert private key from PKCS12 to RSA PEM 10. $ openssl -in pkey.p12 -passin pass:notasecret -nodes -nocerts | openssl rsa -out pkey.pem 11. $ git clone https://github.com/ansible/ansible 12. $ cd ansible 13. $ cat <<EOF > ~/hosts.ini 14. [localhost] 15. 127.0.0.1 16. $ export ANSIBLE_HOST_KEY_CHECKING=False 17. $ export ANSIBLE_HOSTS=~/hosts.ini 18. $ source hacking/env-setup 19. $ eval `ssh-agent` 20. $ ssh-add ~/.ssh/google_compute_engine 21. $ ansible all -m ping 22. 127.0.0.1 | success >> { 23. “changed”: false, 24. “ping”: “pong” 25. } 26. $ ansible all -m gce -a “name=via-ansible zone=us-central1-b” 27. $ ansible all -m gce -a “name=via-ansible zone=us-central1-b state=absent”