Advanced Puppet For PHP Devs - Webinar

Transcript

1. TechAdemy.nl
2. Joshua Thijssen Freelance consultant and trainer @ NoxLogic & TechAdemy Founder of the Dutch Web Alliance. Development in PHP, Python, Perl, C, Java. Lead developer of Saffire. Blog: http://adayinthelifeof.nl Email: jthijssen@noxlogic.nl Twitter: @jaytaph
3. • TechAdemy workshop days • Monthly workshops with 2 half-day sessions. • Training & Consultancy • Techademy.nl
4. http://www.phparch.com/books/mastering-the-spl-library/ PDF, ePub, Mobi, or hardcopy
5. • Vagrant • Puppet • Hiera • Creating modules & manifests Today’s webinar
7. • Don’t confuse vagrant with puppet (or any other provider) • “My vagrant does not work” equals “My spoon needs new batteries” or “This printer won’t can’t the dishes”. • 2 step process (provide & provision)
8. • Provide your environment • downloads basebox • creates vm from basebox • fiddles with settings and starts the machine
9. • Provision your machine • puppet (remote puppet master) • shell script • chef (solo or master) • ansible
10. providing
11. • Vagrant is originally virtualbox only • Now: vmware fusion as well (paid) • Custom providers - like EC2
12. • This means: configuration changes • Different Vagrantfile syntax
13. Vagrant::Config.run do |config| .... end Vagrant.configure("2") do |config| .... end Old configuration file New configuration file
14. Vagrant.configure("1") do |config| end Vagrant.configure("2") do |config| end Combine old and new (but... don’t)
15. Vagrant::configure("2") do |config| # Use a standard box config.vm.box = 'precise64' config.vm.box_url = 'http://files.vagrantup.com/precise64.box' # Define our virtual machine settings config.vm.define :symfony2 do |symfony2| symfony2.vm.hostname = "symfony2-project.dev" symfony2.vm.network :private_network, ip: "192.168.33.10" symfony2.vm.synced_folder ".", "/vagrant", :nfs => true # Here we customize our virtualbox provider. If there are others, add them accordingly below symfony2.vm.provider :virtualbox do |vbox| vbox.gui = true vbox.customize [ 'modifyvm', :id, '--chipset', 'ich9', # solves kernel panic issue on some host machines ] end # Provision through puppet symfony2.vm.provision :puppet do |puppet| puppet.manifests_path = "support/puppet/manifests" puppet.module_path = "support/puppet/modules" puppet.manifest_file = "symfony2.pp" puppet.options = [ '--verbose' ] # Add --debug for more info end end end
16. • Shared directories • vboxfs can be slow • Try NFS.. Still not the best, but ok’ish • But does not work under windows
17. • Shared directories • vboxfs can be slow • Try NFS.. Still not the best, but ok’ish • But does not work under windows • :nfs => (RUBY_PLATFORM =~ /linux/ or RUBY_PLATFORM =~ /darwin/)
18. • Vagrantfiles are just RUBY includes! • (could well have been: Vagrantfile.php) • Thank god it’s not...
19. Vagrant.configure("2") do |config| # define basebox # define machine # define settings # define provider end
20. • Vagrant can boot up multiple machines at once. • Separate web - db - solr - whatevers • Have a vagrantfile.single and vagrantfile.multi • Let users symlink correct vagrantfile (or use script)
21. • Vagrant issues: • Issue: No “REAL” way to push arguments • Issue: boxes are created sequential. This can take a LOOONG time :(
22. • Two different setups: CentOS and Debian baseboxes (please don’t, but you can)
23. #!/bin/sh if [ -z $1 ] ; then echo "Usage $0 [centos|ubuntu]" exit 1; fi echo $1 > .vagrant_os vagrant up
24. Vagrant::configure("2") do |config| os = File.read '.vagrant_os' case os.strip when "ubuntu" config.vm.box = 'UbuntuServer12.04amd64-nox' config.vm.box_url = 'https://..../UbuntuServer12.04amd64-nox.box?dl=1' when "centos" config.vm.box = "centos6.4-64bit" config.vm.box_url = "http://.../CentOS-6.4-x86_64-v20130427.box" else abort("incorrect info in .vagrant_os. Use ./vagrant.sh script to start.") end .... end Again: Vagrantfile is ruby!
25. • There are other ways to send over arguments. But don’t use it. Vagrant does not really support arguments. :(
26. • Vagrant does not support parallel setup of boxes, but we can fake it:
27. #!/bin/sh MAX_PROCS=3 if [ -z $2 ] ; then echo "Usage $0 [single|multi|enterprise] [centos|ubuntu]" exit 1; fi parallel_provision() { while read box; do echo "Provisioning '$box'. Output will be in: $box.out.txt" 1>&2 echo $box done | xargs -P $MAX_PROCS -I"BOXNAME" \ sh -c 'vagrant provision BOXNAME >BOXNAME.out.txt 2>&1 || echo "Error Occurred: BOXNAME"' } ln -sf support/Vagrantfile.$1 Vagrantfile echo $2 > .vagrant_os vagrant up --no-provision vagrant status | grep '^\w*.*(\w*)$' | grep -o '^\w*' | parallel_provision
28. • We setup the boxes (without any provisioning) • We “find” the number of boxes that are available. • We “run” the provisioners in the background. • Does not work with windows obviously.
29. provisioning
30. • Multiple provisioners: • Shell - simple, easy, fast • Puppet apply • Puppet master
31. Vagrant::configure("2") do |config| # Define our virtual machine settings config.vm.define :symfony2 do |symfony2| # Provision through puppet symfony2.vm.provision :puppet do |puppet| puppet.manifests_path = "support/puppet/manifests" puppet.module_path = "support/puppet/modules" puppet.manifest_file = "symfony2.pp" puppet.options = [ '--verbose' ] # Add --debug for more info end end end
32. Vagrant.configure("2") do |config| config.vm.provision :puppet do |puppet| puppet.facter = { "vagrant" => "1" } end end Supply facts directly from vagrantfile if ($vagrant) { ... }
34. Puppet versions
35. • Puppet • 3.0 or up • No really.. 3.0 or up • Ok.. 2.7 is still acceptable for our purpose • But don’t. Use 3.x
36. • What’s new in Puppet 3.0?
37. • Speeeeeeed • Hiera is included • No more dynamic scoping (LSB’ish) • No more puppetd, puppetca, etc etc
38. • Puppet - client/server environment • Puppet - single (puppet apply)
39. • We use puppet apply for our development machines. • We can use puppet client/server for our production environments (or use puppet apply as well).
40. • Don’t run a puppet master without DECENT knowledge of infrastructure and administration (aka: system admin) • By default: developers are NOT system administrators. • Puppet does not magically turns you into a sysadmin.
41. The trifecta Package/file/service: Learn it, live it, love it. If you can only do this, you can still do a lot.
42. • Package (installation) • File (configuration) • Service (startup)
43. package { 'openssh-server': ensure => installed, } file { '/etc/ssh/sshd_config': source => 'puppet:///modules/sshd/sshd_config', owner => 'root', group => 'root', mode => '640', notify => Service['sshd'], require => Package['openssh-server'], } service { 'sshd': ensure => running, enable => true, hasstatus => true, hasrestart => true, }
44. defining vs declaring
45. class jaystuff { package { “mc” : ensure => installed, } } include jaystuff Defining (does not evaluate) Declare (execute it)
46. include jaystuff Include vs Class class { “jaystuff” : }
47. • include evaluates once, like: php’s include_once() • class declaration will fail when declared twice, BUT you can use class parameters. class { “jaystuff” : some_var => true, all_of_it => “yes please”, }
48. • class can inherit other classes • But don’t do this (too often) • Normally, only for parameters class techademy::webserver inherits techademy::webserver::params { # useful stuff here }
49. Don’t require everything
50. package { '...': } -> file { '...': } -> file { '...': } -> service { '...': } -> ..... Don’t require everything
51. • Don’t require everything. • Only require what you need. • Changing my.cnf settings doesn’t need mysql-server, it needs my.cnf!
52. • require, before, notify, subscribe • -> <- ~> <~ • LTR: preferred: -> ~>
53. symfony2
54. • symfony2 requires (lots) of manual handling. • Does not work well with virtual machines (yet). • For instance: app_dev.php, config.php etc • Wants / needs decent user/web permissions
55. • chmod +a • setfacl • umask(000) • run webserver as “vagrant”
56. • app/cache • app/logs
57. • Best option: move them to the directories where they belong (/var/cache, and /var/log) • Sometimes: readonly filesystems for webroots
58. # Not 100% sure if this works! parameters.yml: kernel.logs_dir: /var/log/webapp kernel.cache_dir: /var/cache/webapp # app/AppKernel.php public function getCacheDir() { return ‘/var/cache/webapp/' . $this->environment; } public function getLogDir() { return '/var/log/webapp'; }
59. Custom facts
60. • Add your custom facts to create better (and more robust) modules • ZendServer fact • “Environment” fact • Add to your manifests and autoload!
61. Facter.add(:zendserver) do confine :kernel => :linux setcode do if FileTest.exists?("/usr/local/zend/bin") "true" else "false" end end end
62. • class dependencies: • tomcat depends on java • activemq depends on java • don’t hardcode java into tomcat or activemq • can’t run both otherwise! (or, use include)
63. class tomcat { package { “openjdk-7” : ensure => installed, } } class activemq { package { “openjdk-7” : ensure => installed, } } node default { class { “activemq” : } class { “tomcat” : } # Nope! }
64. class tomcat { Class[‘java’] -> Class[‘activemq’] } class java { package { “openjdk-7” : ensure => installed, } } node default { class { “java” : } class { “activemq” : } class { “tomcat” : } # yes! } class activemq { Class[‘java’] -> Class[‘activemq’] }
65. • apt-get update before packages • Do we need to run this ALL the time? • Run, if not already ran in the last hour.
66. class techademy::repo::debian { Exec["apt-update"] -> Package <| |> # This will only execute apt-get update every day once. exec { "apt-update": command => "/usr/bin/apt-get update touch /tmp/.aptgetupdate ", onlyif => "sh -c 'filemtime=`stat -c %Y /tmp/.aptgetupdate` currtime=`date +%s` diff=$(( currtime - filemtime )) test \$diff -gt 86400'" } } /modules/techademy/manifests/repo/debian.pp
67. class techademy::repo { class { “techademy::repo::${operatingsystem}” : } } /modules/techademy/manifests/repo.pp
68. class techademy::repo { if defined( "techademy::repo::${operatingsystem}") { class { “techademy::repo::${operatingsystem}” : } } } /modules/techademy/manifests/repo.pp
69. • Puppet stages • Not fond of them. But can be used. • Correct usage of dependencies will work better
70. stage { ‘pre’ : before => Stage[‘main’], } stage { ‘post’ : } Stage[‘main’] -> Stage[‘last’]; class { “important-stuff-to-do-first” : stage => pre, } class { “cleanup-or-something” : stage => post, }
71. • augeas can do lots of neat stuff. • but not for everything out of the box, sadly • Don’t copy complete configuration files • Let augeas update what you need
72. augeas { 'set-xdebug-ini-values': context => '/files/etc/php.d/xdebug.ini', changes => [ 'set Xdebug/xdebug.remote_enable On', 'set Xdebug/xdebug.remote_connect_back On', 'set Xdebug/xdebug.remote_port 9000', 'set Xdebug/xdebug.remote_handler dbgp', 'set Xdebug/xdebug.remote_autostart On', 'set Xdebug/xdebug.remote_log /vagrant/xdebug.log', ], require => Package['php'], notify => Service['apache'], }
73. augeas { 'set-php-ini-values': context => '/files/etc/php.ini', changes => [ 'set PHP/error_reporting "E_ALL | E_STRICT"', 'set PHP/display_errors On', 'set PHP/display_startup_errors On', 'set PHP/html_errors On', 'set Date/date.timezone Europe/Amsterdam', ], require => Package['php'], notify => Service['apache'], }
74. Writing modules
75. • puppet module • puppet help module
76. $ puppet module generate techademy-webserver Notice: Generating module at /home/jthijssen/techademy-webserver techademy-webserver techademy-webserver/spec techademy-webserver/spec/spec_helper.rb techademy-webserver/Modulefile techademy-webserver/README techademy-webserver/manifests techademy-webserver/manifests/init.pp techademy-webserver/tests techademy-webserver/tests/init.pp $
77. Hiera
78. • Hiera makes sense. • Separate data from code. • Separate configuration data from configuration code (hierarchical)
79. # hiera.yaml --- :backends: - yaml - json :yaml: :datadir: /etc/puppet/hieradata :json: :datadir: /etc/puppet/hieradata :hierarchy: - “%{::clientcert}” - “%{::environment}” - common
80. • Defaults: json and yaml • Write your own backend (or use existing ones) • http://docs.puppetlabs.com/hiera/1/ custom_backends.html
81. • Multiple backends can be confusing
82. # hiera.yml --- :backends: - yaml - json :hierarchy: - one - two - three Search order: one.yaml, two.yaml, three.yaml, one.json, two.json, three.json
83. http://docs.puppetlabs.com/hiera/1/hierarchy.html
84. # hiera.yaml --- :backends: - yaml :yaml: :datadir: /etc/puppet/hieradata :hierarchy: - “%{::clientcert}” - “%{::environment}” - common
85. web01.techademy.dev.yaml db01.techademy.dev.yaml production.dev.yaml common.yaml
86. --- # web01.techademy.dev.yaml: webserver: nginx memcached: memory: 128M pooled: no services: - webserver - memcached db01.techademy.dev.yaml production.dev.yaml common.yaml --- # common.yaml: services: - ssh --- # db01.techademy.dev.yaml: mysql: port: 3306 services: - mysql --- # production.yaml: selinux: type: enforcing services: - selinux
87. web01.techademy.dev: webserver: nginx # web01 memcached: # web01 memory: 128M # web01 pooled: no # web01 services: - webserver # web01 - memcached # web01 - selinux # production - ssh # common selinux: type: enforcing # production
88. class foo::bar($ip = “0.0.0.0”, $somethingelse = “default”) { ... } Use them as parameters in your parameterized classes Looks for “::foo::bar::ip” and “::foo::bar::somethingelse”
89. • Only uses “priority”, ie: hiera() • Cannot do hiera_array() or hiera_hash() • Need to do manually inside class
90. • hiera(key, default, override) • key = key to search • default = value when key is not found • override = hierarchy file to seek first
91. • hiera(“foo”, “bar”, “importantdata”) • Search for “foo”, if not found, return “bar” • Seek: importantdata.yaml -> then the rest of hierarchy.
92. config.vm.provision :puppet do |puppet| puppet.manifests_path = "manifests" puppet.manifest_file = "base.pp" puppet.module_path = "modules" puppet.options = "--hiera_config hiera.yaml" end Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
93. • Separate your hieradata from your manifests! • hiera_array() / hiera_hash() :(
94. DONE! https://joind.in/event/view/1483