Scalable Cloud-Native Masterless Puppet, with PuppetDB and Bolt

This presentation was run at the virtual PuppetCamp EU, on 5th May 2020.

Transcript

1. Copyright © 2020 ForgeRock. All rights reserved Craig Watson Senior Systems Engineer - ForgeRock IT Virtual Puppet Camp Germany - 5th May, 2020 Scalable Cloud-Native Masterless Puppet, with PuppetDB and Bolt
2. Copyright © 2020 ForgeRock. All rights reserved Who Am I? Senior Systems Engineer, ForgeRock IT - Bristol, UK Puppet user since 2011, community member since 2012 AWS: 2013, Google Cloud: 2017 Background: Systems Engineering, Public Cloud consultancy and systems design Dad, heavy metal, Liverpool FC and Doctor Who fan AWS Certified SysOps Associate & DevOps Professional Puppet Certified Professional (2016 & 2017) Presenter at Puppetize PDX 2019 2
3. Copyright © 2020 ForgeRock. All rights reserved Copyright © 2020 ForgeRock. All rights reserved A Little History 3
4. Copyright © 2020 ForgeRock. All rights reserved Copyright © 2020 ForgeRock. All rights reserved Master of Puppets - Somewhere Back in Time 4 Over time, Puppet Masters become monoliths Servers are “long-lived cattle” Lift-and-shift cloud migrations become problematic Hybrid infrastructure? Use on-premise masters for cloud? Solutions exist (auto-signing, compile-masters) Most of the time, results in a compromise! Scalability and manageability most often sacrificed
7. Copyright © 2020 ForgeRock. All rights reserved Copyright © 2020 ForgeRock. All rights reserved Cloud-Native, Scalable Puppet 7
8. Copyright © 2020 ForgeRock. All rights reserved Copyright © 2020 ForgeRock. All rights reserved Masterless/Agentless Puppet - Summary 8 Puppet runs locally via puppet apply Puppet codebase distributed to every node Exact mechanism can vary (RPM/DEB, tar-ball, Git …) Decentralised - no/few outside dependencies Packages can be downloaded from object storage (S3/GCS) Scalable - no single point of failure for new nodes Bootstrap/user-data scripts take care of all provisioning Testable - Allows easy development via Vagrant Everything is local! First step to immutable infrastructure As Puppet runs locally, images can be taken post-run
9. Copyright © 2020 ForgeRock. All rights reserved Secrets Management Secrets are encrypted at-rest in Git with EYAML and SaaS KMS AWS - https://github.com/adenot/hiera-eyaml-kms GCP - https://github.com/craigwatson/hiera-eyaml-gkms We wrote a helper script to interface with KMS 9 --- profiles::confluence::db_password: ENC[GKMS,CiQAPPX7KHnvqMjmxXUsaIJZil55rm1oBbs=] /etc/puppetlabs/code/data/env/prod/confluence.yaml $ ./eyaml.sh -e prod -a encrypt -v correcthorsebatterystaple ENC[GKMS,CiQAPPX7KHnvqMjmxXUsaIJZil55rm1oBbs=]
10. Copyright © 2020 ForgeRock. All rights reserved Hiera and Instance Metadata Scripts can enumerate metadata and store as static facts for cross-cloud portability 10 for DATA in $(curl http://169.254.169.254/computeMetadata/v1/instance/attributes/); do KEY=$(echo "${DATA}" | sed 's/-/_/g') VALUE=$(curl "http://169.254.169.254/computeMetadata/v1/instance/attributes/${DATA}") echo "${KEY}=${VALUE}" >> /etc/facter/facts.d/metadata.txt done resource "aws_instance" "pdb" { instance_type = "c3.xlarge" availability_zone = "eu-west2a" ... tags = { Name = "puppetdb" role = "puppetdb" } } resource "google_compute_instance" "pdb" { name = "puppetdb" machine_type = "n1-standard-2" region = "europe-west1" ... metadata = { role = "puppetdb" } }
11. Copyright © 2020 ForgeRock. All rights reserved Copyright © 2020 ForgeRock. All rights reserved PuppetDB 11
12. Copyright © 2020 ForgeRock. All rights reserved PuppetDB Overview Central “node database” for Puppet Puppet sends facts, catalog and report for each run Data exposed via Puppetboard UI - thanks to Vox Pupuli! App - https://github.com/voxpupuli/puppetboard Puppet module - https://github.com/voxpupuli/puppet-puppetboard Deployed standalone as a standard “three-tiered” web-application Puppet module - https://forge.puppet.com/puppetlabs/puppetdb Two PuppetDB servers, behind and SSL-terminating load balancer We use Google CloudSQL to provide a SaaS PostgreSQL database 12
13. Copyright © 2020 ForgeRock. All rights reserved PuppetDB Installation Add classes to role via Hiera Configure puppetdb::server::disable_ssl: true puppetdb::server::gc_interval: 1 puppetdb::server::node_ttl: '32m' puppetdb::server::node_purge_ttl: '1s' profiles::nginx_proxy::upstream_port: 8080 13 --- classes: - puppetdb::server - profiles::cloud_sql_proxy - profiles::nginx_proxy
14. Copyright © 2020 ForgeRock. All rights reserved Sending Node Data to PuppetDB (1) Install puppetdb-termini package Configure Puppet’s routes.yaml (YMMV at this point!) 14 --- apply: catalog: terminus: compiler cache: puppetdb resource: terminus: ral cache: puppetdb facts: terminus: facter cache: puppetdb_apply /etc/puppetlabs/puppet/routes.yaml
15. Copyright © 2020 ForgeRock. All rights reserved Sending Node Data to PuppetDB (2) Configure Puppet 15 [main] server_urls = https://puppetdb.example.com:443 soft_write_failure = true verify_client_certificate = false /etc/puppetlabs/puppet/puppetdb.conf [main] report = true reports = puppetdb localcacert = /etc/pki/tls/certs/ca-bundle.crt certificate_revocation = false /etc/puppetlabs/puppet/puppet.conf
16. Copyright © 2020 ForgeRock. All rights reserved Copyright © 2020 ForgeRock. All rights reserved Bolt 16
17. Copyright © 2020 ForgeRock. All rights reserved Bolt Overview Executes tasks over SSH, can use PuppetDB for inventory Handles rich scripts/plans in Puppet SDL, and also allows arbitrary CLI commands We use bolt command run to: Update Puppet code via yum (we package our codebase as an RPM and host on GCS) Run Puppet via puppet apply We deploy a bolt user on each host, and use Jenkins as our Bolt “control node” 17
18. Copyright © 2020 ForgeRock. All rights reserved Connecting Bolt to PuppetDB We use Jenkins as a Bolt control node 18 /var/lib/jenkins/.puppetlabs/bolt/bolt.yaml --- modulepath: '/etc/puppetlabs/code/modules' ssh: host-key-check: false run-as: root user: bolt puppetdb: server_urls: ["https://puppetdb.example.com:443"] cacert: /etc/pki/tls/certs/ca-bundle.crt
19. Copyright © 2020 ForgeRock. All rights reserved Bolt PuppetDB Inventory Template Within the wrapper script, an “inventory template” file is copied to /tmp, edited via sed and passed to bolt 19 version: 2 groups: - name: dynamic targets: - _plugin: puppetdb query: "inventory[certname] {PQL_QUERY_PLACEHOLDER}" target_mapping: name: facts.networking.fqdn uri: facts.networking.ip cp /path/to/template.yaml /tmp/inventory.yaml sed -i "s/PQL_QUERY_PLACEHOLDER/facts.role = 'confluence'/" /tmp/inventory.yaml bolt command run "hostname" --inventory /tmp/inventory.yaml --targets dynamic
20. Copyright © 2020 ForgeRock. All rights reserved Copyright © 2020 ForgeRock. All rights reserved Build Pipeline & Summary 20
21. Copyright © 2020 ForgeRock. All rights reserved Full Orchestration Pipeline 21 Install Modules librarian-puppet Build RPM fpm Download Repo gsutil rsync Add Package createrepo Upload Repo gsutil rsync Run PQL Query bolt-wrapper.sh Return Nodes PuppetDB SSH to each node Bolt Run Command Bolt Install Puppet Code to target instance / Run Puppet Build Package
22. Copyright © 2020 ForgeRock. All rights reserved Final Thoughts Masterless Puppet allows us to scale our Puppet deployment with little overhead Secrets are encrypted at-rest with per-environment KMS keys, decrypted via EYAML Our nodes send facts, catalogs and reports to PuppetDB PuppetDB is deployed as a standard three-tier web-application with LB and SaaS DB As part of our deployment pipeline, Bolt queries PuppetDB for inventory Bolt then connects to each node via SSH and runs the required commands 22
23. Copyright © 2020 ForgeRock. All rights reserved Thank You! craigwatson1987 craigwatson