Deep-Dive: Cloud-Native Masterless Puppet, with Bolt and PuppetDB

Building, managing and scaling a Puppet master in a largely ephemeral cloud environment can be quite a daunting prospect, with certificate management, node lifecycles and the usual high-availability points to consider.

In this session, we will deep-dive into how ForgeRock’s IT team are using masterless Puppet together with Bolt and PuppetDB to create a truly elastic cloud-native Puppet environment.

This approach has enabled us to scale out our infrastructure in the cloud from a standing start with minimal ongoing engineering overhead, and it can help you too!

Transcript

1. Deep-Dive: Cloud-Native Masterless Puppet, with Bolt and PuppetDB Craig Watson - Senior Systems Engineer, ForgeRock Puppetize PDX 2019
2. » Senior Systems Engineer, ForgeRock IT - Bristol, UK » Puppet user since 2011, community member since 2012 » Background: Systems Engineering, Public Cloud consultancy » Dad, musical theatre/heavy metal and Liverpool FC fan » AWS Certified SysOps Associate & DevOps Professional » Google Certified Cloud Architect » Puppet Certified Professional (2016/2017) Who Am I?
3. ForgeRock IT - Core Services » Apps - JIRA, Confluence, BitBucket, Artifactory » Release Engineering Build Servers - Jenkins » Supporting Services - network, RADIUS, backups, DNS, VPNs, backups, etc. » Environments - staging and production
4. ForgeRock IT Cloud Migration Project Planning
5. Master of Puppets - Somewhere Back in Time » Servers are pets/flowers, rather than cattle/crops » Puppet Master - monolith, with certificate signing and node definitions by regex or logic-based ENC » Masters become “core infrastructure” or SPoFs » Without the Puppet Master: » No new servers can be created » No existing servers can be updated
9. Don’t future-proof. Make the future possible.
10. Key Design Features Scalable Cloud-Agnostic Cloud-Native Testable
11. Why Cloud-Native? [Cloud native] techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high- impact changes frequently and predictably with minimal toil. Cloud-Native Computing Foundation - https://github.com/cncf/toc/blob/master/DEFINITION.md
12. Puppet Design and Tools
13. Roles and Profiles » Structured, modular, standardised way of organising Puppet code » Encourages generic code (cattle/crops) over custom (pets/flowers) » Role - single purpose for a server (e.g. confluence-app) » Profile - software or platform (e.g. nginx, php-fpm, jenkins, java)
14. Instance Metadata - Cloud ENC » Traditionally, we only have the system hostname/domain/FQDN » Cloud systems have access to key-value metadata #!/bin/bash > /etc/facter/facts.d/metadata.txt for DATA in $(curl .. http://169.254.169.254/computeMetadata/v1/instance/attributes/); do KEY=$(echo "${DATA}" | sed 's/-/_/g') VALUE=$(curl .. "http://169.254.169.254/computeMetadata/v1/instance/attributes/${DATA}") echo "${KEY}=${VALUE}" >> /etc/facter/facts.d/metadata.txt done » Deployed by Terraform, translated into static facts by startup/userdata scripts
15. Using Hiera with Metadata » Used to assign classes/profiles to roles, and pass module parameters » Can be made cloud-agnostic by using the virtual fact --- version: 5 defaults: datadir: data hierarchy: - name: Data paths: - "%{facts.virtual}/role_%{facts.role}.yaml" - "%{facts.virtual}/default.yaml" - "role/%{facts.role}.yaml" - "env/%{facts.envname}/%{facts.role}.yaml" - "env/%{facts.envname}/default.yaml" - default.yaml
16. Module Management with librarian-puppet » Recursive module dependency management » Uses Puppetfile to declare modules, similar format to R10K » Bounded SemVer dependencies: >= 3.7.1 < 4.0.0 » Can refer to “local” modules, as well as Git and Puppet Forge
17. forge "https://forgeapi.puppetlabs.com" mod 'forgerock-profiles', :path => './local/profiles' mod 'elastic-elasticsearch', '>= 6.3.3 < 7.0.0' mod 'elastic-kibana', '>= 6.3.1 < 7.0.0' mod 'elastic-logstash', '>= 6.1.5 < 7.0.0' mod 'jgazeley-freeradius', '>= 3.8.1 < 4.0.0' mod 'pcfens-filebeat', '>= 3.4.0 < 4.0.0' mod 'puppet-jenkins', '>= 2.0.0 < 3.0.0' mod 'puppet-nginx', '0.15.0' mod 'puppet-puppetboard', '>= 5.0.0 < 6.0.0' mod 'puppet-selinux', '>= 1.6.0 < 2.0.0' mod 'puppetlabs-docker', '>= 3.5.0 < 4.0.0' mod 'puppetlabs-lvm', '>= 1.2.0 < 2.0.0' mod 'puppetlabs-ntp', '>= 7.4.0 < 8.0.0' mod 'puppetlabs-puppetdb', '>= 7.1.0 < 8.0.0' mod 'saz-dnsmasq', '>= 1.4.0 < 2.0.0' mod 'saz-ssh', '>= 5.0.0 < 6.0.0' mod 'saz-timezone', '>= 5.1.0 < 6.0.0'
18. Masterless/Agentless Puppet » Puppet code distributed to each node - /etc/puppetlabs/code » Decentralised - no outside dependencies (that we manage) » Horizontally scalable - no single point of failure (that we manage) » Runs using puppet apply rather than puppet agent node default { lookup('classes', Array[String], 'unique').include } /etc/puppetlabs/code/manifests/site.pp
19. Secrets - EYAML and Cloud KMS » Hiera EYAML plugins available for major cloud providers » AWS - https://github.com/adenot/hiera-eyaml-kms » GCP - https://github.com/craigwatson/hiera-eyaml-gkms » Keys can be locked down to decrypt-only via IAM policy » KMS does not store data, it just provides and holds keys --- profiles::confluence::db_password: ENC[GKMS,CiQAPPX7KHnvqMjmxXUsaIJZil55rm1oBbs=] /etc/puppetlabs/code/data/env/nonprod/confluence.yaml
20. --- version: 5 defaults: datadir: data lookup_key: eyaml_lookup_key hierarchy: - name: Data paths: - "%{facts.virtual}/%{facts.envname}/%{facts.role}.yaml" - "%{facts.virtual}/role_%{facts.role}.yaml" - "%{facts.virtual}/default.yaml" - "role/%{facts.role}.yaml" - "env/%{facts.envname}.yaml" - default.yaml options: gkms_project: "%{facts.gce.project.projectId}" gkms_location: europe-west2 gkms_keyring: mykeyring gkms_crypto_key: my_eyaml_key gkms_auth_type: machineaccount
21. Secrets - EYAML and Cloud KMS » We wrote a helper script to generate encrypted base64 ciphertext ./eyaml.sh -e nonprod -a encrypt -v 'correcthorsebatterystaple' ENC[GKMS,CiQAPPX7KHnvqMjmxXUsaIJZil55rm1oBbs=] ./eyaml.sh -e nonprod -a decrypt -v 'ENC[GKMS,CiQAPPX7KHnvqMjmxXUsaIJZil55rm1oBbs=]' correcthorsebatterystaple » Can also decrypt data (Hiera does this on-the-fly via EYAML backend)
22. Orchestration - Deep-Dive
23. » Instance bootstrap/startup/userdata script » Install Puppet repository, binaries and codebase » Stores metadata as static facts » Configure and run Puppet Orchestration Tasks yum -y install puppet-code » Updating Puppet code: puppet apply -—verbose /etc/puppetlabs/code/manifests/site.pp » Running Puppet:
24. Full Orchestration Pipeline » Build Package Install Modules 
 librarian-puppet Build RPM
 fpm Download Repo
 gsutil rsync Add Package
 createrepo Upload Repo
 gsutil rsync Run PQL Query bolt-wrapper.sh Return Nodes
 PuppetDB SSH to each node
 Bolt Run Command Bolt » Install Puppet Code to target instance / Run Puppet
25. RPM Packages and Yum Repositories » Uses Jenkins job/build number to version packages » Runs syntax, lint and spec tests before building RPM » Entire codebase packaged into RPM by fpm » Added to Yum repository in Google Cloud Storage or Amazon S3 » All nodes are set up with Yum plugin for GCS or S3
26. Bolt - Task Runner » Turn-key solution for task execution via SSH » Can use PuppetDB for dynamic inventory » Takes advantage of logic and data typing » Also allows arbitrary CLI commands to be executed - bolt command run » Our usage is extremely simple, and may not be best practice!
27. Bolt - Wrapper Script ./bolt-wrapper.sh -e staging -r puppetdb -c 'yum -y install puppet-code' bolt command run 'yum -y install puppet-code' \ --query "inventory[certname] { facts.role = 'puppetdb' and facts.environment = 'staging' }" » Custom Bash script used to wrap Bolt » Allows us to abstract PQL queries
28. Caveats - Masterless Puppet and PuppetDB » Configuration has been tested with Puppet 6.4.2 & 6.9.0 » Actual configuration may vary! » Bug affecting standalone PuppetDB existed in puppetdb-termini 6.5.0 » Due to deprecation of internal Puppet SSL functions, which assumed a master was used » Fixed in puppetdb-termini 6.6.0 » Backwards-compatible with Puppet <= 6.4.0 » Added configuration options for masterless/standalone operation
29. PuppetDB » Deployed standalone, with PostgreSQL backend » Nodes check into PuppetDB every 30 minutes via --noop cron and fqdn_rand() » Receives three sets of data each run - facts, catalog, report » Nodes can be purged if not seen for a certain amount of time
30. Creating a Standalone PuppetDB Cluster » Two PuppetDB servers, behind an SSL-terminating load balancer » We use Google CloudSQL to provide a SaaS Postgres database » PuppetDB Puppet module » https://forge.puppet.com/puppetlabs/puppetdb » Puppetboard provides a UI layer » https://forge.puppet.com/puppet/puppetboard » https://github.com/voxpupuli/puppetboard
31. Sending Node Data to PuppetDB (1) » Install puppetdb-termini package » Configure Puppet routes --- apply: catalog: terminus: compiler cache: puppetdb resource: terminus: ral cache: puppetdb facts: terminus: facter cache: puppetdb_apply /etc/puppetlabs/puppet/routes.yaml
32. Sending Node Data to PuppetDB (2) » Finally, configure Puppet [main] server_urls = https://puppetdb.example.com:443 soft_write_failure = true verify_client_certificate = false /etc/puppetlabs/puppet/puppetdb.conf [main] storeconfigs = false report = true reports = puppetdb localcacert = /etc/pki/tls/certs/ca-bundle.crt certificate_revocation = false /etc/puppetlabs/puppet/puppet.conf
33. Connecting Bolt to PuppetDB » We use Jenkins as a Bolt control node --- modulepath: '/etc/puppetlabs/code/modules' ssh: host-key-check: false run-as: root user: bolt puppetdb: server_urls: ["https://puppetdb.example.com:443"] cacert: /etc/pki/tls/certs/ca-bundle.crt /var/lib/jenkins/.puppetlabs/bolt/bolt.yaml
34. Full Orchestration Pipeline » Build Package Install Modules 
 librarian-puppet Build RPM
 fpm Download Repo
 gsutil rsync Add Package
 createrepo Upload Repo
 gsutil rsync Run PQL Query bolt-wrapper.sh Return Nodes
 PuppetDB SSH to each node
 Bolt Run Command Bolt » Install Puppet Code to target instance / Run Puppet
35. Final Thoughts
36. Project Outcomes » Allowed us to accelerate cloud adoption » Flexible enough to allow direction changes without rework » Increase team throughput, and (eventually) decrease toil » Near-horizontal scalability » Both cloud-native and cloud-agnostic
37. Room for Improvement - Work in Progress » PuppetDB can be a single point of failure » Documentation is sparse on masterless Puppet » Rewrite Bolt wrapper script into plan » KMS keys can be split further to per-role-per-environment » Bolt requires SSH users and key to be provisioned, managed & secured
38. Thank You! craigwatson1987 craigwatson