Puppet at SpaceX: How to stay sane, while sending rockets into space

The much-requested slides from Joachim "Jok" Thuau of SpaceX speaking on "Puppet at SpaceX". From PuppetCamp LA '12. Jok Thuau gave a similar talk you can watch here: http://youtu.be/YcxEmjkulU8

Abstract: SpaceX manages a large number of machines using puppet. A quick overview of where puppet fits and how we came to use it.

Speaker Bio: I have been a system administrator for over 10 years, managing everything from networking to Windows, Linux and Macs. Before joining the SpaceX team, I worked in the video game industry. I have always been fascinated by computers, video games and space travel, and am lucky enough to have gotten to work in all three areas. I enjoy reading, subverting devices and spending time with my family.

Watch other PuppetConf videos you missed out on here: http://www.puppetlabs.com/community/videos/puppetconf

Be the first to know about PuppetConf 2013:
http://info.puppetlabs.com/puppetconf2013-notification.html

Learn more: www.puppetlabs.com

Transcript

1. Puppet at SpaceX
   How to stay sane, while sending rockets into space
   

   View Slide

2. Who’s this guy anyway?
   —  Linux system admin
   —  Came from the video game industry
   —  Started at SpaceX about 9 month ago
   —  Linux System Engineer
   

   View Slide

3. SpaceX
   —  Rocket Company
   —  First private company to send a vehicle to LEO
   —  AND recover the vehicle
   —  Founded in 2002
   —  Falcon1, Falcon 9, Dragon
   

   View Slide

4. Puppet
   —  How it happened
   —  Before Scale 10x
   —  After Scale 10x
   

   View Slide

5. Bootstrapping
   —  What we have
   —  Debian/Ubuntu
   —  Kerberos (windows KDC/Active Directory)
   —  LDAP (windows LDAP/Active Directory)
   —  “Apt” mirror (with debian squeeze/ubuntu LTS)
   —  Tons of scripts!
   

   View Slide

6. Deployment
   —  Using existing scripts and replacing them with
   puppet.
   —  Install packages from our local repo
   —  Push configs with puppet
   —  Using exec (in a couple of places only) to “fix” a
   couple of things.
   —  This simplified things a lot for us
   

   View Slide

7. Where are we at now?
   —  Plug desktop/server in (power/network/keyboard/
   video)
   —  Boot “off the network” (pxe)
   —  Pick on pxe menu what you want to install
   —  We have a number of “presets” we use
   —  Install starts and asks the user for hostname
   —  Machine uses a pre-seed file to setup stuff initially
   —  Installs puppet as part of that pre-seed from our local
   mirror
   —  Generate and registers with puppet server
   —  (login on server and sign cert)
   —  And wait for it! We’re DONE!
   

   View Slide

8. Cool stuff to come
   —  Puppet server infrastructure overview
   —  TheForeman
   —  Really nice and easy to setup
   —  Has “proxies” for handling stuff with DHCP/DNS/PXE
   automation (we’re planning on using this, and it’s a little
   more difficult for us, since that part of the infrastructure is
   windows based)
   —  Nagios monitoring
   —  Host lists/services/dependencies built by puppet
   —  HPC
   —  We have a compute cluster that we are starting to rebuild
   using puppet for automation and validation (test cluster
   with a couple of nodes, validate config, and push to
   production)
   

   View Slide