Puppet at SpaceX: How to stay sane, while sending rockets into space

The much-requested slides from Joachim "Jok" Thuau of SpaceX speaking on "Puppet at SpaceX". From PuppetCamp LA '12. Jok Thuau gave a similar talk you can watch here: http://youtu.be/YcxEmjkulU8

Abstract: SpaceX manages a large number of machines using puppet. A quick overview of where puppet fits and how we came to use it.

Speaker Bio: I have been a system administrator for over 10 years, managing everything from networking to Windows, Linux and Macs. Before joining the SpaceX team, I worked in the video game industry. I have always been fascinated by computers, video games and space travel, and am lucky enough to have gotten to work in all three areas. I enjoy reading, subverting devices and spending time with my family.

Watch other PuppetConf videos you missed out on here: http://www.puppetlabs.com/community/videos/puppetconf

Be the first to know about PuppetConf 2013:
http://info.puppetlabs.com/puppetconf2013-notification.html

Learn more: www.puppetlabs.com

Transcript

1. Puppet at SpaceX How to stay sane, while sending rockets into space
2. Who’s this guy anyway? —  Linux system admin —  Came from the video game industry —  Started at SpaceX about 9 month ago —  Linux System Engineer
3. SpaceX —  Rocket Company —  First private company to send a vehicle to LEO —  AND recover the vehicle —  Founded in 2002 —  Falcon1, Falcon 9, Dragon
4. Puppet —  How it happened —  Before Scale 10x —  After Scale 10x
5. Bootstrapping —  What we have —  Debian/Ubuntu —  Kerberos (windows KDC/Active Directory) —  LDAP (windows LDAP/Active Directory) —  “Apt” mirror (with debian squeeze/ubuntu LTS) —  Tons of scripts!
6. Deployment —  Using existing scripts and replacing them with puppet. —  Install packages from our local repo —  Push configs with puppet —  Using exec (in a couple of places only) to “fix” a couple of things. —  This simplified things a lot for us
7. Where are we at now? —  Plug desktop/server in (power/network/keyboard/ video) —  Boot “off the network” (pxe) —  Pick on pxe menu what you want to install —  We have a number of “presets” we use —  Install starts and asks the user for hostname —  Machine uses a pre-seed file to setup stuff initially —  Installs puppet as part of that pre-seed from our local mirror —  Generate and registers with puppet server —  (login on server and sign cert) —  And wait for it! We’re DONE!
8. Cool stuff to come —  Puppet server infrastructure overview —  TheForeman —  Really nice and easy to setup —  Has “proxies” for handling stuff with DHCP/DNS/PXE automation (we’re planning on using this, and it’s a little more difficult for us, since that part of the infrastructure is windows based) —  Nagios monitoring —  Host lists/services/dependencies built by puppet —  HPC —  We have a compute cluster that we are starting to rebuild using puppet for automation and validation (test cluster with a couple of nodes, validate config, and push to production)