mcollective

A quick talk given internally to the staff of Infoxchange about mcollective

Transcript

1. MCOLLECTIVE (aka. stop borking puppet in prod)
2. Glossary Mcollective consists of: Middleware, a messaging queue server or pool (ActiveMQ, Redis or RabbitMQ) Client, let you interact with the nodes using “mco” Server, running on each configured node Plugins that extend functionality (data gathered, actions possible, etc)
3. Middleware Architecture Client Nodes puppet plugin service plugin package plugin puppet plugin service plugin package plugin
4. Setup Certificates need to be generated before hand for secure setup Everything is done in puppet (with some help from the build-in “package”) Using the mcollective class provided by puppetlabs (and required dependancies classes Erlang and RabbitMQ), customised to store private keys in hiera-eyaml
5. Securing mcollective Using puppetmaster as a CA:
 > puppet cert generate janesmith
 > puppet cert generate johndoe
 > puppet cert generate mcollective-servers Copy the resulting certificates in a secure location
 > $libdir/ssl/certs/johndoe.pem
 (this will later be used by the client) 
 > $libdir/ssl/private_keys/johndoe.pem
 (this will later be used by the client)
 > $libdir/ssl/public_keys/johndoe.pem
 (this will later be used by the nodes) 
 
 > $libdir/ssl/*/mcollective-servers
 (this will be used to secure the connection with the middleware)
6. Middleware First, some prerequisites: ! node 'rabbitmq-serv' {
 include ixacommon::debian::wheezy" include ixanagios::client" include ixapuppet::debian::puppetlabsapt" include erlang" ! #Need at least Erlang-R16B for puppet SSL certs" package { 'erlang-base':" ensure => 'latest'," }" [...]
7. Middleware Some basic configuration to start things off : ! class { '::mcollective':" connector => rabbitmq," middleware => true," server => false," client => false," }
8. Middleware Defaults is psk, let's make it a bit more secure: ! [...]" middleware_ssl => true," securityprovider => 'ssl'," ssl_ca_cert => '/var/lib/puppet/ssl/certs/ca.pem'," ssl_server_public => 'puppet:///modules/ixamcollective/certs/mcollective- servers.pem'," #Storing private keys in hiera-eyaml instead" ssl_server_private => hiera('mcollective-servers')," [...]"
9. Middleware Finish off with some useful options: ! [...]" middleware_user => mcollective," middleware_password => hiera('mcollective_pw')," middleware_admin_user => admin," middleware_admin_password => hiera('mcollective_admin_pw')," delete_guest_user => true," [...]"
10. Mcollective server “server” is misleading More accurately an agent that can gather data/run commands/etc Can be expanded with plugins to handle packages, service, puppet status, processes, gather data. Possible to also write your own agents to expand on functionality (most of them are a collection of ruby scripts)
11. Mcollective Server First, configuring role and where to find the messaging queue:
 
 class { '::mcollective':
 connector => rabbitmq,
 manage_packages => true,
 server => true,
 middleware_hosts => [ 'rabbitmq.serv' ],
 [...]
 

12. Mcollective Server Then tell it how to connect (securely!):
 
 middleware_ssl => true,
 securityprovider => 'ssl',
 middleware_user =>'mcollective',
 middleware_password => hiera('mcollective_pw'),
 # These are the certs for clients on client nodes
 # ie. either individual users certs or machine certs 
 ssl_client_certs => 'puppet:///modules/ixamcollective/client_certs',
 ssl_ca_cert => '/var/lib/puppet/ssl/certs/ca.pem',
 # This pair allow secure communication between MQ and mcollective
 ssl_server_public => 'puppet:///modules/ixamcollective/certs/mcollective-servers.pem',
 ssl_server_private => hiera('mcollective-servers'), 
 

13. Mcollective Server Basic connectivity check:
 from /var/log/mcollective.log : 
 
 INFO -- : rabbitmq.rb:35:in `on_ssl_connecting' Establishing SSL session with stomp+ssl:// mcollective@rabbitmq.serv:61614
 INFO -- : rabbitmq.rb:10:in `on_connecting' TCP Connection attempt 0 to stomp+ssl:// mcollective@rabbitmq.serv:61614
 INFO -- : rabbitmq.rb:40:in `on_ssl_connected' SSL session established with stomp+ssl:// mcollective@rabbitmq.serv:61614
 INFO -- : rabbitmq.rb:15:in `on_connected' Connected to stomp+ssl://mcollective@rabbitmq.serv:61614 Common issues preventing connectivity: Mismatched SSL certificates outdated Erlang Version/ruby_stomp gem
 
 

14. Mcollective Server Now to add some plugins to extend functionality:
 Plugins are available as native package on puppetlabs repos (yum and apt). You need to restart the mcollective service to pick up new plugins
 
 package {['mcollective-puppet-common', 'mcollective-puppet-agent',]:
 ensure => '1.7.0-1puppetlabs1',
 notify => Service["mcollective"],
 }
 package {['mcollective-service-agent', 'mcollective-service-common']:
 ensure => present,
 notify => Service["mcollective"],
 }
 
 Alternatively packaged plugins can be set using the mcollective module:
 
 mcollective::plugin { 'puppet':
 package => true,
 }
 and custom plugins like so:
 
 mcollective::plugin { 'myplugin':
 source => 'puppet:///modules/site_mcollective/plugins',
 }
15. Mcollective client The mcollective module require per-user access configuration using key pairs (if SSL is enabled). This can also be used for ACLs. To access the extra functionality, you need to install the plugins matching the plugins on your servers. You do not have to use puppet and the mcollective module to set up users….but it helps a lot.
16. Mcollective client The initial setup is straight forward:
 class { '::mcollective':
 client => true,
 manage_packages => true,
 connector => rabbitmq,
 middleware_hosts => [ 'rabbitmq.serv' ],
 middleware_password => hiera('mcollective_pw'),
 middleware_user => 'mcollective',
 [...]
 }

17. Mcollective client We are using SSL, so let's configure that:
 
 class { '::mcollective':
 [...]
 middleware_ssl => true,
 securityprovider => 'ssl',
 ssl_ca_cert => '/var/lib/puppet/ssl/certs/ca.pem',
 #We don't need the private key since we'll use user certificates
 ssl_server_public => 'puppet:///modules/ixamcollective/certs/mcollective- servers.pem',
 [...]
 } 

18. Mcollective client All of that is a bit useless without users:
 
 mcollective::user{"janesmith":
 certificate => 'puppet:///modules/ixamcollective/certs/janesmith.pem',
 private_key => hiera('janesmith'),
 }
 
 mcollective::user{"johndoe":
 certificate => 'puppet:///modules/ixamcollective/certs/johndoe.pem',
 private_key => hiera('johndoe'),
 } Nota: the module has been modified to use hiera lookups for the private key for users as well
19. Mcollective client Let's not forget the plugins:
 
 package {['mcollective-puppet-common', 'mcollective- puppet-client',]:
 ensure => present,
 }
 package {['mcollective-service-agent', 'mcollective- service-client',]:
 ensure => present,
 }

20. Next steps ACL/permissions: In the server/node configuration block:
 mcollective::actionpolicy { 'puppet':
 default => 'deny',
 }
 
 mcollective::actionpolicy::rule { 'Only Jane can be trusted to use puppet':
 agent => 'puppet',
 callerid => 'cert=janesmith',
 }
21. Mcollective in action