Config / Managment Tools and Google Compute Engine

Slides for a talk given to the Seattle DevOps meetup on Oct 29th 2013.

Transcript

1. Config / Mgmnt Tools and Google Cloud Platform Eric Johnson Program Manager, Google Compute Engine
2. Cloud Platform Agenda • Why Google? • Whirlwind tour of Google Cloud Platform • Google Compute Engine Instance with: • Vagrant • Chef • Puppet • Ansible
3. For the past 15 years, Google has been building out a massively fast, powerful, and reliable cloud infrastructure across the planet. Images by Connie Zhou
4. Cloud Platform Developing our infrastructure while respecting our ecosystem • Pioneering data center efficiency • Financed over 250 Megawatts of new wind power • First data centers to receive IOS 14001 certification • 100% carbon neutral
5. 2002 2004 2006 2008 2010 2012 Colossus Dremel MapReduce Spanner Big Table GFS Driving Technology Forward Cloud Platform
6. Google Cloud Platform Storage Cloud Storage Cloud SQL Cloud Datastore Persistent Disk App Services BigQuery Cloud Endpoints Caching Queues Cloud Platform Compute App Engine Compute Engine
7. Cloud Platform • BLOB Storage, Immutable Objects • Strong read-after-write consistency • API and Web UI Accessible • Versioning • Static Sites, ACLs • Resumable Transfers • Object Change Notifications • Object lifecycle management Google Cloud Storage
8. • Fully managed, MySQL(like) • Ease of Use and Development • Highly Reliable • Flexible Charging • Security, Availability, Durability • EU and US Data Centers • Easy Migration & Data Portability • Control Cloud Platform Cloud SQL
9. Cloud Platform • Schemaless, Non-relational NoSQL Access • Auto-scale • Authentication That Just Works • Fast and Easy Provisioning • RESTful Endpoints • ACID Transactions • Query Language (akin to SQL) • Local Development Tools • Built-in Redundancy Cloud Datastore
10. Cloud Platform • Fully Managed Big Data Analytics Service • Fast • Scalable • Flexible and Familiar • Security and Reliability BigQuery
11. Cloud Platform • Fully Managed Platform • Easy Development & Deployment • Focus On Your Code Not Your Server • Automatic Scaling • Popular Programming Language Support • Services (Cron, Queue, Memcache, etc) • Datastore • Versioning and Traffic Splitting • Local Developer Tools • Third-party Frameworks and Extensions App Engine
12. Cloud Platform • Sub-hour Billing • Up to 10TB Persistent Disk • Over 64 Instance Types • Standard Linux Distributions • Advanced Networking • Instance Metadata and Startup Scripts • Load Balancing • Persistent Disks, snapshots • Fast and Easy Provisioning • Consistent Performance Google Compute Engine
13. Your instance here Images by Connie Zhou Google Compute Engine
14. Launching 100 VMs screencast in case of emergency Cloud Platform
15. Cloud Platform Demo Time Times Four! • Not an exhaustive dive into each tool, but we’ll watch a new GCE instance created in: ◦ Vagrant ◦ Chef ◦ Puppet ◦ Ansible • We’ll skip all the gory setup details, but they’re included in the slide deck if you’re curious
16. Cloud Platform Vagrant (vagrant-google) $ cat -n Vagrantfile 1 Vagrant.configure("2") do |config| 2 config.vm.box = "gce" 3 config.vm.provision :shell, :privileged => false, :inline => $PROVISION_NODE 3 4 config.vm.provider :google do |google, override| 5 google.google_project_id = $GOOGLE_PROJECT_ID 6 google.google_client_email = $GOOGLE_CLIENT_EMAIL 7 google.google_key_location = $GOOGLE_KEY_LOCATION 8 9 google.name = "via-vagrant" 10 google.zone = "us-central1-b" 11 google.image = "debian-7-wheezy-v20130926" 12 google.machine_type = "n1-standard-1" 13 14 override.ssh.username = $LOCAL_USER 15 override.ssh.private_key_path = $LOCAL_SSH_KEY 16 end 17 end $ vagrant up --provider=google $ vagrant destroy -f
17. Cloud Platform Chef (knife-google) 1 erjohnso@chef:~$ knife google help 2 Available google subcommands: (for details, knife SUB-COMMAND --help) 3 4 ** GOOGLE COMMANDS ** 5 knife google disk create NAME --google-disk-sizeGb N --google-compute-zone ZONE (options) 6 knife google disk delete NAME --google-compute-zone ZONE 7 knife google disk list --google-compute-zone ZONE (options) 8 knife google server create NAME -m MACHINE_TYPE -I IMAGE -Z ZONE (options) 9 knife google server delete SERVER [SERVER] --google-compute-zone ZONE (options) 10 knife google server list --google-compute-zone ZONE (options) 11 knife google setup 12 knife google zone list (options) $ knife google server create via-chef -m n1-standard-1 \ > -I debian-7-wheezy-v20130926 -Z us-central1-b \ > -x erjohnso -i ~/.ssh/google_compute_engine \ > -s https://chef-server...com.internal:443 \ > -r “recipe[apt],recipe[apache2]” $ knife google server delete via-chef -Z us-central1-b \ > --purge
18. Cloud Platform Puppet (gce_compute) erjohnso@puppet$ cat -n .puppet/device.conf create.pp 1 [gce_puppet] 2 type gce 3 url [/dev/null]:google.com:erjohnso 4 5 gce_instance { "via-puppet": ensure => present, machine_type => 'n1-standard-1', 6 zone => 'us-central1-b', network => 'default', 7 image => 'projects/debian-cloud/global/images/debian-7-wheezy-v20130926', 8 manifest => 'class apache ($version = "latest") { 9 package {"apache2": ensure => $version, } 10 file {"/var/www/index.html": ensure => present, require => Package["apache2"], 11 content => "<html><body><pre>Hi, this is $gce_external_ip.</pre></body></html>", 12 } 13 service {"apache2": ensure => running, enable => true, 14 require => File["/var/www/index.html"], 15 } 16 } 17 include apache' 18 } $ puppet apply --certname gce_puppet ./create.pp $ puppet apply --certname gce_puppet ./destroy.pp
19. Cloud Platform Ansible (gce* modules) erjohnso@ansible$ cat -n hosts.ini gce-up.yml 1 [localhost] 2 127.0.0.1 3 [gce_instance] 4 via-ansible 5 6 - name: Bring up a GCE instance with ansible 7 hosts: localhost 8 tasks: 9 - name: Launch new instance 10 local_action: gce name=via-ansible machine_type=n1-standard-1 11 image=centos-6 zone=us-central1-b 12 - name: Update instance 13 hosts: gce_instance 14 tasks: 15 - name: Install apache 16 yum: pkg=httpd state=present $ ansible localhost -m gce \ > -a “name=via-ansible zone=us-central1-b” $ ansible localhost -m gce \ > -a “name=via-ansible zone=us-central1-b state=absent”
20. Cloud Platform References • Chef (knife-google): https://github.com/opscode/knife-google • Puppet (gce_compute): https://github.com/puppetlabs/puppetlabs-gce_compute • Also, but not covered (node_gce): https://github.com/puppetlabs/puppetlabs-node_gce • Vagrant (vagrant-google): https://github.com/GoogleCloudPlatform/vagrant-google • Ansible (in core): https://github.com/ansible/ansible • Salt... Coming soon!
21. Cloud Platform And finally... DevOps + GCE = Awesome! • Use the platform and send us feedback ◦ https://cloud.google.com/ ◦ $2,000 Credit - Use “ansf-con” promo code at https: //cloud.google.com/starterpack • Help improve Google module support Thank you!
22. Cloud Platform Vagrant Setup Workstation: 1. // Create GCE Debian-7 instance, name it ‘chef-workstation’ 2. $ sudo apt-get update && sudo apt-get upgrade -y 3. $ sudo apt-get install git vim build-essential zlib1g-dev libssl-dev build-essential zlib1g-dev libssl-dev ruby1.9.1-dev -y 4. $ sudo gem1.9.1 install --no-ri --no-rdoc bundler 5. $ wget http://files.vagrantup.com/packages/.../vagrant_x.y.z_x86_64.deb 6. $ sudo dpkg -i vagrant_x.y.z_x86_64.deb 7. $ git clone https://github.com/GoogleCloudPlatform/vagrant-google.git 8. $ cd vagrant-google 9. $ bundle 10. $ gem build vagrant-google.gemspec 11. $ vagrant plugin install vagrant-google-0.1.1.gem 12. $ vagrant box add gce google.box 13. // Create the Vagrantfile 14. $ vagrant up --provider=google 15. $ vagrant destroy -f
23. Cloud Platform Chef Setup Server: 1. // Create GCE CentOS-6 instance, name it ‘chef-server’ 2. $ wget https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-server-11.0.8-1.el6.x86_64.rpm 3. $ sudo rpm -i chef-server-11.0.8.el6.x86_64.rpm 4. $ sudo chef-server-ctl reconfigure && sleep 30 && sudo chef-server-ctl test Workstation: 1. // Create GCE Debian-7 instance, name it ‘chef-workstation’ 2. $ sudo apt-get update && sudo apt-get upgrade -y 3. $ curl -L https://www.opscode.com/chef/install.sh | sudo bash 4. // cp chef-server:/etc/chef-server/admin.pem chef-workstation:/etc/chef-server/admin.pem 5. // cp chef-server:/etc/chef-server/chef-validator.pem chef-workstation:/etc/chef-server/chef-validator.pem 6. $ git clone git://github.com/opscode/chef-repo.git 7. $ knife configure -i # server https://chef-server.c.erjohnso.google.com.internal:443, cookbook_path = ~/chef-repo/cookbooks 8. // Verify that it’s working on the workstation by running, 9. $ knife client list 10. $ knife user list 11. $ knife cookbook site install apt # repeat for ‘apache2’ 12. $ knife cookbook upload apt apache2 13. // Install knife-google on the workstation by, 14. $ sudo /opt/chef/embedded/bin/gem install knife-google 15. // Register App on cloud console, ‘Web Application’ -> ‘OAuth2.0 Client ID’ to generate Client ID and Client Secret 16. $ knife google setup # use your Project ID, Client ID, and Client Secret 17. $ knife google server list -Z us-central1-b 18. $ gcutil ssh `hostname -s` 19. $ knife google server create node1 -m n1-standard-1 -I debian-7-wheezy-v20130926 -Z us-central1-b -x erjohnso -i ~/.ssh/google_compute_engine
24. Cloud Platform Puppet Setup Workstation: 1. // Create GCE Debian-7 instance 2. $ sudo apt-get update && sudo apt-get upgrade -y 3. $ sudo apt-get install puppet 4. $ mkdir -p ~/.puppet/modules 5. $ puppet module install puppetlabs-gce_compute 6. $ gctuil version # 1.8.4 which is fine since current gce_compute module relies on 1.8.3 7. $ gcutil ssh `hostname -s` # register my ssh key with metadata server 8. $ cat <<EOF > ~/.puppet/device.conf 9. [gce_puppet] 10. type gce 11. url [/dev/null]:google.com:erjohnso 12. $ cat <<EOF > ~/create.pp 13. gce_instance { “via-puppet”: ensure => present, machine_type => ‘n1-standard-1’, zone => ‘us-central1-b’, network => ‘default’, 14. image => ‘projects/debian-cloud/global/images/debian-7-wheezy-v20130926’, tags => [‘web’], 15. manifest => 'class apache ($version = "latest") { 16. package {"apache2": ensure => $version, } 17. file {"/var/www/index.html": ensure => present, require => Package["apache2"], 18. content => "<html><body><pre>Hi, this is $gce_external_ip.</pre></body></html>", 19. } 20. service {"apache2": ensure => running, enable => true, require => File["/var/www/index.html"], } 21. } 22. include apache' 23. } 24. $ puppet apply --certname gce_puppet ./create.pp
25. Cloud Platform Ansible Setup Workstation: 1. // Create GCE Debian-7 instance 2. $ sudo apt-get update && sudo apt-get upgrade -y 3. $ sudo apt-get install python-paramiko python-yaml python-jinja2 python-pycryptopp git -y 4. $ git clone https://github.com/apache/libcloud # currently uses dev branch of libcloud until 0.14+ is released, then a pip install could be used 5. $ cd libcloud; sudo python setup.py install; cd ~ 6. $ cp libcloud/demos/secrets-dist.py ~/secrets.py 7. // edit ~/secrets.py and update GCE_PARAMS, GCE_KEYWORD_PARMS, use Service Account email_address and private_key location, and project_id 8. // make sure secrets.py is in your PYTHONPATH 9. // convert private key from PKCS12 to RSA PEM 10. $ openssl -in pkey.p12 -passin pass:notasecret -nodes -nocerts | openssl rsa -out pkey.pem 11. $ git clone https://github.com/ansible/ansible 12. $ cd ansible 13. $ cat <<EOF > ~/hosts.ini 14. [localhost] 15. 127.0.0.1 16. $ export ANSIBLE_HOST_KEY_CHECKING=False 17. $ export ANSIBLE_HOSTS=~/hosts.ini 18. $ source hacking/env-setup 19. $ eval `ssh-agent` 20. $ ssh-add ~/.ssh/google_compute_engine 21. $ ansible all -m ping 22. 127.0.0.1 | success >> { 23. “changed”: false, 24. “ping”: “pong” 25. } 26. $ ansible all -m gce -a “name=via-ansible zone=us-central1-b” 27. $ ansible all -m gce -a “name=via-ansible zone=us-central1-b state=absent”