Tweet
Tweet

Slide 1

Slide 1 text

TechAdemy.nl

Slide 2

Slide 2 text

Joshua Thijssen Freelance consultant and trainer @ NoxLogic & TechAdemy Founder of the Dutch Web Alliance. Development in PHP, Python, Perl, C, Java. Lead developer of Saffire. Blog: http://adayinthelifeof.nl Email: [email protected] Twitter: @jaytaph

Slide 3

Slide 3 text

• TechAdemy workshop days • Monthly workshops with 2 half-day sessions. • Training & Consultancy • Techademy.nl

Slide 4

Slide 4 text

http://www.phparch.com/books/mastering-the-spl-library/ PDF, ePub, Mobi, or hardcopy

Slide 5

Slide 5 text

• Vagrant • Puppet • Hiera • Creating modules & manifests Today’s webinar

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

• Don’t confuse vagrant with puppet (or any other provider) • “My vagrant does not work” equals “My spoon needs new batteries” or “This printer won’t can’t the dishes”. • 2 step process (provide & provision)

Slide 8

Slide 8 text

• Provide your environment • downloads basebox • creates vm from basebox • fiddles with settings and starts the machine

Slide 9

Slide 9 text

• Provision your machine • puppet (remote puppet master) • shell script • chef (solo or master) • ansible

Slide 10

Slide 10 text

providing

Slide 11

Slide 11 text

• Vagrant is originally virtualbox only • Now: vmware fusion as well (paid) • Custom providers - like EC2

Slide 12

Slide 12 text

• This means: configuration changes • Different Vagrantfile syntax

Slide 13

Slide 13 text

Vagrant::Config.run do |config| .... end Vagrant.configure("2") do |config| .... end Old configuration file New configuration file

Slide 14

Slide 14 text

Vagrant.configure("1") do |config| end Vagrant.configure("2") do |config| end Combine old and new (but... don’t)

Slide 15

Slide 15 text

Vagrant::configure("2") do |config| # Use a standard box config.vm.box = 'precise64' config.vm.box_url = 'http://files.vagrantup.com/precise64.box' # Define our virtual machine settings config.vm.define :symfony2 do |symfony2| symfony2.vm.hostname = "symfony2-project.dev" symfony2.vm.network :private_network, ip: "192.168.33.10" symfony2.vm.synced_folder ".", "/vagrant", :nfs => true # Here we customize our virtualbox provider. If there are others, add them accordingly below symfony2.vm.provider :virtualbox do |vbox| vbox.gui = true vbox.customize [ 'modifyvm', :id, '--chipset', 'ich9', # solves kernel panic issue on some host machines ] end # Provision through puppet symfony2.vm.provision :puppet do |puppet| puppet.manifests_path = "support/puppet/manifests" puppet.module_path = "support/puppet/modules" puppet.manifest_file = "symfony2.pp" puppet.options = [ '--verbose' ] # Add --debug for more info end end end

Slide 16

Slide 16 text

• Shared directories • vboxfs can be slow • Try NFS.. Still not the best, but ok’ish • But does not work under windows

Slide 17

Slide 17 text

• Shared directories • vboxfs can be slow • Try NFS.. Still not the best, but ok’ish • But does not work under windows • :nfs => (RUBY_PLATFORM =~ /linux/ or RUBY_PLATFORM =~ /darwin/)

Slide 18

Slide 18 text

• Vagrantfiles are just RUBY includes! • (could well have been: Vagrantfile.php) • Thank god it’s not...

Slide 19

Slide 19 text

Vagrant.configure("2") do |config| # define basebox # define machine # define settings # define provider end

Slide 20

Slide 20 text

• Vagrant can boot up multiple machines at once. • Separate web - db - solr - whatevers • Have a vagrantfile.single and vagrantfile.multi • Let users symlink correct vagrantfile (or use script)

Slide 21

Slide 21 text

• Vagrant issues: • Issue: No “REAL” way to push arguments • Issue: boxes are created sequential. This can take a LOOONG time :(

Slide 22

Slide 22 text

• Two different setups: CentOS and Debian baseboxes (please don’t, but you can)

Slide 23

Slide 23 text

#!/bin/sh if [ -z $1 ] ; then echo "Usage $0 [centos|ubuntu]" exit 1; fi echo $1 > .vagrant_os vagrant up

Slide 24

Slide 24 text

Vagrant::configure("2") do |config| os = File.read '.vagrant_os' case os.strip when "ubuntu" config.vm.box = 'UbuntuServer12.04amd64-nox' config.vm.box_url = 'https://..../UbuntuServer12.04amd64-nox.box?dl=1' when "centos" config.vm.box = "centos6.4-64bit" config.vm.box_url = "http://.../CentOS-6.4-x86_64-v20130427.box" else abort("incorrect info in .vagrant_os. Use ./vagrant.sh script to start.") end .... end Again: Vagrantfile is ruby!

Slide 25

Slide 25 text

• There are other ways to send over arguments. But don’t use it. Vagrant does not really support arguments. :(

Slide 26

Slide 26 text

• Vagrant does not support parallel setup of boxes, but we can fake it:

Slide 27

Slide 27 text

#!/bin/sh MAX_PROCS=3 if [ -z $2 ] ; then echo "Usage $0 [single|multi|enterprise] [centos|ubuntu]" exit 1; fi parallel_provision() { while read box; do echo "Provisioning '$box'. Output will be in: $box.out.txt" 1>&2 echo $box done | xargs -P $MAX_PROCS -I"BOXNAME" \ sh -c 'vagrant provision BOXNAME >BOXNAME.out.txt 2>&1 || echo "Error Occurred: BOXNAME"' } ln -sf support/Vagrantfile.$1 Vagrantfile echo $2 > .vagrant_os vagrant up --no-provision vagrant status | grep '^\w*.*(\w*)$' | grep -o '^\w*' | parallel_provision

Slide 28

Slide 28 text

• We setup the boxes (without any provisioning) • We “find” the number of boxes that are available. • We “run” the provisioners in the background. • Does not work with windows obviously.

Slide 29

Slide 29 text

provisioning

Slide 30

Slide 30 text

• Multiple provisioners: • Shell - simple, easy, fast • Puppet apply • Puppet master

Slide 31

Slide 31 text

Vagrant::configure("2") do |config| # Define our virtual machine settings config.vm.define :symfony2 do |symfony2| # Provision through puppet symfony2.vm.provision :puppet do |puppet| puppet.manifests_path = "support/puppet/manifests" puppet.module_path = "support/puppet/modules" puppet.manifest_file = "symfony2.pp" puppet.options = [ '--verbose' ] # Add --debug for more info end end end

Slide 32

Slide 32 text

Vagrant.configure("2") do |config| config.vm.provision :puppet do |puppet| puppet.facter = { "vagrant" => "1" } end end Supply facts directly from vagrantfile if ($vagrant) { ... }

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

Puppet versions

Slide 35

Slide 35 text

• Puppet • 3.0 or up • No really.. 3.0 or up • Ok.. 2.7 is still acceptable for our purpose • But don’t. Use 3.x

Slide 36

Slide 36 text

• What’s new in Puppet 3.0?

Slide 37

Slide 37 text

• Speeeeeeed • Hiera is included • No more dynamic scoping (LSB’ish) • No more puppetd, puppetca, etc etc

Slide 38

Slide 38 text

• Puppet - client/server environment • Puppet - single (puppet apply)

Slide 39

Slide 39 text

• We use puppet apply for our development machines. • We can use puppet client/server for our production environments (or use puppet apply as well).

Slide 40

Slide 40 text

• Don’t run a puppet master without DECENT knowledge of infrastructure and administration (aka: system admin) • By default: developers are NOT system administrators. • Puppet does not magically turns you into a sysadmin.

Slide 41

Slide 41 text

The trifecta Package/file/service: Learn it, live it, love it. If you can only do this, you can still do a lot.

Slide 42

Slide 42 text

• Package (installation) • File (configuration) • Service (startup)

Slide 43

Slide 43 text

package { 'openssh-server': ensure => installed, } file { '/etc/ssh/sshd_config': source => 'puppet:///modules/sshd/sshd_config', owner => 'root', group => 'root', mode => '640', notify => Service['sshd'], require => Package['openssh-server'], } service { 'sshd': ensure => running, enable => true, hasstatus => true, hasrestart => true, }

Slide 44

Slide 44 text

defining vs declaring

Slide 45

Slide 45 text

class jaystuff { package { “mc” : ensure => installed, } } include jaystuff Defining (does not evaluate) Declare (execute it)

Slide 46

Slide 46 text

include jaystuff Include vs Class class { “jaystuff” : }

Slide 47

Slide 47 text

• include evaluates once, like: php’s include_once() • class declaration will fail when declared twice, BUT you can use class parameters. class { “jaystuff” : some_var => true, all_of_it => “yes please”, }

Slide 48

Slide 48 text

• class can inherit other classes • But don’t do this (too often) • Normally, only for parameters class techademy::webserver inherits techademy::webserver::params { # useful stuff here }

Slide 49

Slide 49 text

Don’t require everything

Slide 50

Slide 50 text

package { '...': } -> file { '...': } -> file { '...': } -> service { '...': } -> ..... Don’t require everything

Slide 51

Slide 51 text

• Don’t require everything. • Only require what you need. • Changing my.cnf settings doesn’t need mysql-server, it needs my.cnf!

Slide 52

Slide 52 text

• require, before, notify, subscribe • -> <- ~> <~ • LTR: preferred: -> ~>

Slide 53

Slide 53 text

symfony2

Slide 54

Slide 54 text

• symfony2 requires (lots) of manual handling. • Does not work well with virtual machines (yet). • For instance: app_dev.php, config.php etc • Wants / needs decent user/web permissions

Slide 55

Slide 55 text

• chmod +a • setfacl • umask(000) • run webserver as “vagrant”

Slide 56

Slide 56 text

• app/cache • app/logs

Slide 57

Slide 57 text

• Best option: move them to the directories where they belong (/var/cache, and /var/log) • Sometimes: readonly filesystems for webroots

Slide 58

Slide 58 text

# Not 100% sure if this works! parameters.yml: kernel.logs_dir: /var/log/webapp kernel.cache_dir: /var/cache/webapp # app/AppKernel.php public function getCacheDir() { return ‘/var/cache/webapp/' . $this->environment; } public function getLogDir() { return '/var/log/webapp'; }

Slide 59

Slide 59 text

Custom facts

Slide 60

Slide 60 text

• Add your custom facts to create better (and more robust) modules • ZendServer fact • “Environment” fact • Add to your manifests and autoload!

Slide 61

Slide 61 text

Facter.add(:zendserver) do confine :kernel => :linux setcode do if FileTest.exists?("/usr/local/zend/bin") "true" else "false" end end end

Slide 62

Slide 62 text

• class dependencies: • tomcat depends on java • activemq depends on java • don’t hardcode java into tomcat or activemq • can’t run both otherwise! (or, use include)

Slide 63

Slide 63 text

class tomcat { package { “openjdk-7” : ensure => installed, } } class activemq { package { “openjdk-7” : ensure => installed, } } node default { class { “activemq” : } class { “tomcat” : } # Nope! }

Slide 64

Slide 64 text

class tomcat { Class[‘java’] -> Class[‘activemq’] } class java { package { “openjdk-7” : ensure => installed, } } node default { class { “java” : } class { “activemq” : } class { “tomcat” : } # yes! } class activemq { Class[‘java’] -> Class[‘activemq’] }

Slide 65

Slide 65 text

• apt-get update before packages • Do we need to run this ALL the time? • Run, if not already ran in the last hour.

Slide 66

Slide 66 text

class techademy::repo::debian { Exec["apt-update"] -> Package <| |> # This will only execute apt-get update every day once. exec { "apt-update": command => "/usr/bin/apt-get update touch /tmp/.aptgetupdate ", onlyif => "sh -c 'filemtime=`stat -c %Y /tmp/.aptgetupdate` currtime=`date +%s` diff=$(( currtime - filemtime )) test \$diff -gt 86400'" } } /modules/techademy/manifests/repo/debian.pp

Slide 67

Slide 67 text

class techademy::repo { class { “techademy::repo::${operatingsystem}” : } } /modules/techademy/manifests/repo.pp

Slide 68

Slide 68 text

class techademy::repo { if defined( "techademy::repo::${operatingsystem}") { class { “techademy::repo::${operatingsystem}” : } } } /modules/techademy/manifests/repo.pp

Slide 69

Slide 69 text

• Puppet stages • Not fond of them. But can be used. • Correct usage of dependencies will work better

Slide 70

Slide 70 text

stage { ‘pre’ : before => Stage[‘main’], } stage { ‘post’ : } Stage[‘main’] -> Stage[‘last’]; class { “important-stuff-to-do-first” : stage => pre, } class { “cleanup-or-something” : stage => post, }

Slide 71

Slide 71 text

• augeas can do lots of neat stuff. • but not for everything out of the box, sadly • Don’t copy complete configuration files • Let augeas update what you need

Slide 72

Slide 72 text

augeas { 'set-xdebug-ini-values': context => '/files/etc/php.d/xdebug.ini', changes => [ 'set Xdebug/xdebug.remote_enable On', 'set Xdebug/xdebug.remote_connect_back On', 'set Xdebug/xdebug.remote_port 9000', 'set Xdebug/xdebug.remote_handler dbgp', 'set Xdebug/xdebug.remote_autostart On', 'set Xdebug/xdebug.remote_log /vagrant/xdebug.log', ], require => Package['php'], notify => Service['apache'], }

Slide 73

Slide 73 text

augeas { 'set-php-ini-values': context => '/files/etc/php.ini', changes => [ 'set PHP/error_reporting "E_ALL | E_STRICT"', 'set PHP/display_errors On', 'set PHP/display_startup_errors On', 'set PHP/html_errors On', 'set Date/date.timezone Europe/Amsterdam', ], require => Package['php'], notify => Service['apache'], }

Slide 74

Slide 74 text

Writing modules

Slide 75

Slide 75 text

• puppet module • puppet help module

Slide 76

Slide 76 text

$ puppet module generate techademy-webserver Notice: Generating module at /home/jthijssen/techademy-webserver techademy-webserver techademy-webserver/spec techademy-webserver/spec/spec_helper.rb techademy-webserver/Modulefile techademy-webserver/README techademy-webserver/manifests techademy-webserver/manifests/init.pp techademy-webserver/tests techademy-webserver/tests/init.pp $

Slide 77

Slide 77 text

Hiera

Slide 78

Slide 78 text

• Hiera makes sense. • Separate data from code. • Separate configuration data from configuration code (hierarchical)

Slide 79

Slide 79 text

# hiera.yaml --- :backends: - yaml - json :yaml: :datadir: /etc/puppet/hieradata :json: :datadir: /etc/puppet/hieradata :hierarchy: - “%{::clientcert}” - “%{::environment}” - common

Slide 80

Slide 80 text

• Defaults: json and yaml • Write your own backend (or use existing ones) • http://docs.puppetlabs.com/hiera/1/ custom_backends.html

Slide 81

Slide 81 text

• Multiple backends can be confusing

Slide 82

Slide 82 text

# hiera.yml --- :backends: - yaml - json :hierarchy: - one - two - three Search order: one.yaml, two.yaml, three.yaml, one.json, two.json, three.json

Slide 83

Slide 83 text

http://docs.puppetlabs.com/hiera/1/hierarchy.html

Slide 84

Slide 84 text

# hiera.yaml --- :backends: - yaml :yaml: :datadir: /etc/puppet/hieradata :hierarchy: - “%{::clientcert}” - “%{::environment}” - common

Slide 85

Slide 85 text

web01.techademy.dev.yaml db01.techademy.dev.yaml production.dev.yaml common.yaml

Slide 86

Slide 86 text

--- # web01.techademy.dev.yaml: webserver: nginx memcached: memory: 128M pooled: no services: - webserver - memcached db01.techademy.dev.yaml production.dev.yaml common.yaml --- # common.yaml: services: - ssh --- # db01.techademy.dev.yaml: mysql: port: 3306 services: - mysql --- # production.yaml: selinux: type: enforcing services: - selinux

Slide 87

Slide 87 text

web01.techademy.dev: webserver: nginx # web01 memcached: # web01 memory: 128M # web01 pooled: no # web01 services: - webserver # web01 - memcached # web01 - selinux # production - ssh # common selinux: type: enforcing # production

Slide 88

Slide 88 text

class foo::bar($ip = “0.0.0.0”, $somethingelse = “default”) { ... } Use them as parameters in your parameterized classes Looks for “::foo::bar::ip” and “::foo::bar::somethingelse”

Slide 89

Slide 89 text

• Only uses “priority”, ie: hiera() • Cannot do hiera_array() or hiera_hash() • Need to do manually inside class

Slide 90

Slide 90 text

• hiera(key, default, override) • key = key to search • default = value when key is not found • override = hierarchy file to seek first

Slide 91

Slide 91 text

• hiera(“foo”, “bar”, “importantdata”) • Search for “foo”, if not found, return “bar” • Seek: importantdata.yaml -> then the rest of hierarchy.

Slide 92

Slide 92 text

config.vm.provision :puppet do |puppet| puppet.manifests_path = "manifests" puppet.manifest_file = "base.pp" puppet.module_path = "modules" puppet.options = "--hiera_config hiera.yaml" end Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults

Slide 93

Slide 93 text

• Separate your hieradata from your manifests! • hiera_array() / hiera_hash() :(

Slide 94

Slide 94 text

DONE! https://joind.in/event/view/1483