Chef Conf 2013: Tips and Tricks for Automating Windows

Nordstrom has been using Chef to automate Windows environments since October 2012.

In this talk, I"ll cover tips such as:

Using Mixlib::Shellout and PowershellOut to execute Windows tools and scripts as a Domain user.
Windows cookbook improvements, including Printer LWRP
Diskpart cookbook
Chef-keypass for better one-way encryption of data-bag secrets, including certs and passwords
How to use Windows cookbook helpers
Using the new Windows Registry resource in Chef 11
Windows Sysnative for correctly locating Windows programs
Perf improvement numbers for Ruby 1.9.3 in Chef 11 for Windows
Recommended Ohai plugins to disable

1. Tips and Tricks for Automating Windows Doug Ireton Infrastructure Engineering @dougireton / dougireton.com 1
2. Who am I? • Infrastructure Engineer at Nordstrom • I’ve been a tester, a developer and a sysadmin • Working with Windows for 20 years @dougireton 2 Currently an Infrastructure Engineer at Nordstrom working full-time on Chef I’ve been a tester at Microsoft, a Windows Sysadmin, and a .NET developer on Nordstrom.com Altogether, I’ve been working with Windows for 20 years.
3. Infrastructure Engineering 3 And now, a little bit about our team. Our team was formed last August. We grew out of a cross team working group to implement Chef at Nordstrom. Jon actually wrote our first Chef recipes to build a Linux Hadoop cluster during Velocity conf last June. Since then we’ve been primarily focused on Windows automation, since our stores and our website runs on Windows servers.
4. Who are you? 4 1. How many of you would consider yourselves Windows Sysadmins? 2. How many of you would consider yourselves Linux/Unix Sysadmins? 3. How many of you have automated Windows with Chef?
5. Agenda • About Nordstrom • A challenging first project • What we’ve learned from automating Windows • Twitter: #chefconf #winchef 5 First I’m going to spend about 10 seconds talking about Nordstrom, just to set the context. Then I’m going to tell you a story about a major challenge we faced when automating our first Windows project and how we overcame it. Finally, I’m going to dive into some of the technical tips and tricks we’ve learned over the past six months, including how to securely manage service acct passwords, how to run commands as a domain user, how to manage disks and printers, two Windows cookbook helpers, and two tips for decreasing Chef run times.
6. Brick and Mortar still critical 6 Nordstrom has 242 Stores in 31 states, Nordstrom.com, HauteLook.com 65,000 employees, with 1000 in IT We started using Chef in earnest last October to automate building Windows servers critical to our stores’ success. These servers, called PSPs, sit in each store, roll up each register’s transactions, and forward them to our sales and inventory systems in Denver.
7. A complex first project... 7 Unfortunately, these PSPs run a legacy app which only runs on 32-bit Windows 2003 and SQL Server 2005. In many ways this was the worst possible project to learn Chef on. The existing partially automated process was complex, required 15 handoffs between teams and required tribal knowledge to build each PSP.
8. With Good Results... 8 Nevertheless, we were able to able to achieve some pretty satisfactory results with Chef, especially for such a challenging first project.
9. Our First Real Chef Project • Manual Steps: 48 -> 5 • Team Handoffs: 15 -> 1 • Provision Time: 22 hours -> 7 9
10. 10 What prevented us from fully automating the PSP build?
11. No Run As image We Didn’t Have Run As 11 We faced one big blocker: lack of Run As. Until a month ago, there was no way for Chef to run processes as an Active Directory domain user, instead of local System. We needed Run As to move the server’s machine acct in AD, and the app had to be installed by a member of a specific AD group. We worked around this by having someone with the proper domain rights run Chef Solo directly on the box, which is far from ideal. However, since we only need to build 16-20 new store servers per year, it’s a workable compromise.
12. Fast-Forward to... 12 Fast-forward to Jan 2013
13. 13 In January, we started automating the various server roles which make up the Nordstrom.com website, which we are moving to an all Windows 2008 R2 stack. Nordstrom.com production environment runs on roughly 200 servers, across 40 different server roles, including database, web, search, etc. We knew we couldn’t rely on Chef Solo to build these servers. We had to be able to spin up new web servers by assigning them to a role. We had to solve our Run As problem.
14. “I’ve  no)ced  a  considerable  reduc)on  in  deployment  )me  from  base   OS  to  fully  func)onal  app  server.   We  are  also  deploying  a  more  consistent  product  to  our  customers   now  due  to  the  automated  configura)on  management.” -­‐  Harvey  Bendana Nordstrom  WebOps  team 14 We’ve made good progress. Of the 40 server roles, we’ve automated 8 of them and built a foundation of cookbooks to make the next 32 easier. More importantly, by embedding two members of the Web Ops team on our team, and working with them for 4 months, we’ve spread the knowledge to another team. They are now busy writing their own cookbooks and we’ve been able to turn our attention to other challenges. Since January, we’ve solved our Run As problem and along the way we’ve had to solve quite a few other challenges. In the next 20 minutes, I’m going to share some of things we’ve learned about automating Windows using Chef.
15. Windows Cookbook Helpers 15 I’m going to start with some basic helpers available in the Opscode Windows cookbook. Here’s my daughter explaining to me how these helpers work. win_friendly_path() locate_sysnative_cmd()
16. win_friendly_path() #  include  Windows::Helper  from  Opscode  Windows  Cookbook ::Chef::Recipe.send(:include,  Windows::Helper)   #  now  you  can  call  helper  methods  like  win_friendly_path  directly my_batch_file  =  win_friendly_path('c:/temp/foo.bat')   execute  "My  batch  file"  do    command  my_batch_file    #  c:\temp\foo.bat end 16 This simple method will gsub Windows-style backslashes for Unix-style slashes to make Windows command line tools work. Windows is pretty good at understanding forward slashes, but some commands, for example Robocopy, don’t like forward slashes. Use win_friendly_path() for these situations.
17. locate_sysnative_cmd() helper for 64-bit Windows #  include  Windows::Helper  from  Opscode  Windows  Cookbook ::Chef::Recipe.send(:include,  Windows::Helper) locate_sysnative_cmd("dism.exe") 17 Use this for any command in %SystemRoot%\System32 to ensure Chef-Client (32-bit ruby) correctly finds the command. Bypasses Microsoft’s File System Redirector on 64-bit Windows
18. Run Commands As Another User 18 If you’re going to impersonate another user on Windows, the first thing you need is a secure way to manage and distribute passwords. So just use Encrypted Data Bags, right?
19. “The system uses shared-key encryption. An encrypted file can only be decrypted by a node or a user with the same shared- key.” http://docs.opscode.com/ essentials_data_bags_encrypt.html Encrypted Data Bags 19 Encrypted data bags use shared key encryption. So now you have a key distribution problem. You have to get that shared key to all nodes who need it. And furthermore, that shared key must be a file, usually distributed in a cookbook. So, if you’re a bad guy, all you need to do to get the shared key is “knife cookbook download” and boom, you can decrypt the data bag and get the password.
20. “That’s why storing encryption keys on the same system where the protected data resides violates all of the core principles of data protection.” - Patrick Townsend Townsend Security http://web.townsendsecurity.com/bid/23881/PCI-DSS-2-0-and-Encryption-Key-Management 20 So this is bad. So, how do you solve this? You solve it with Public/Private key encryption. Chef Client already has a private key and it’s public key is already on the Chef server. So just encrypt your passwords and certificates with the node’s public key and only that node can decrypt it.
21. http://www.flickr.com/photos/gtarded/2759499462/sizes/l/ Chef-Vault 21 Chef-Vault is a gem, that Kevin Moser, from our team has open sourced to solve this exact problem. Chef-Vault leverages encrypted data bags and encrypts the shared key using the public key of the node which needs to decrypt the service acct password. Only nodes which you specify can later decrypt that data bag. So let’s walk through an example.
22. knife encrypt password Use this knife command to encrypt the username and password that you want to protect. $  knife  encrypt  password  -­‐-­‐search  "role:web_server"        -­‐-­‐username  "mysql_user"  -­‐-­‐password  "P@ssw0rd"        -­‐-­‐admins  "alice,  bob,  carol" 22 Read the slide... In this example, the mysql_user's password will be encrypted using the public keys of the nodes in the web_server role. In addition to the servers in the web_server role, alice, bob, and carol will also be able to decrypt the password.
23. Securely manage passwords for Run As chef_gem  "chef-­‐vault"   require  'chef-­‐vault'   #  given  a  'passwords'  data  bag vault  =  ChefVault.new("passwords")   #  get  the  'mysql_user'  data  bag  item user  =  vault.user("mysql_user")   #  decrypt  the  user's  password password  =  user.decrypt_password #  do  something  with  password 23 And here’s how you would use it in a recipe. So that’s Chef-Vault.
24. Run Commands as Another User ruby_block  "Add  server  to  WSUS  group"  do    block  do        Chef::Resource::RubyBlock.send(:include,  Chef::Mixin::ShellOut)                #  get  password  from  Chef-­‐Vault        password  =  user.decrypt_password          add_group  =  shell_out(            "dsquery.exe  computer  -­‐name  #{  node['hostname']  }  |  dsmod  group   'cn=patch_Tuesday,dc=mycorp,dc=com'  -­‐addmbr",            {                :user          =>  "my_user",                :password  =>  password,                :domain      =>  "mycorp.com",            }        )    end end 24 And here, we’re using the new impersonation features to run a Windows command to add the server to an Active Directory group. Let’s go through this line by line
25. Managing Devices 25 Here are my twins managing a John Deere tractor.
26. Manage disks, partitions, and drives #  Use  Kevin  Moser’s  diskpart  cookbook   diskpart_partition  "create_#{disk[:letter]}:/"  do    disk_number  disk[:number]    letter  disk[:letter]    action  :create end diskpart_partition  "format_#{disk[:letter]}:/"  do    disk_number  disk[:number]    letter  disk[:letter]    action  :format end 26 Using Kevin Moser’s diskpart cookbook, you can easily manage Windows disks, partitions, and drives.
27. Manage Printers and Printer Ports #  https://github.com/opscode-­‐cookbooks/windows   #  create  a  printer windows_printer  'HP  LaserJet  5th  Floor'  do    driver_name  'HP  LaserJet  4100  Series  PCL6'    ipv4_address  '10.4.64.38' end 27 So clearly, you don’t need to manage printers in the cloud. But large enterprises like Nordstrom still have printers. We contributed printer and printer port LWRPs to the Opscode Windows cookbook.
28. Better Performance 28
29. Chef 11: Ruby Performance Improvements 30 - 50% faster Chef Client Run time on Windows 29
30. Ohai Plugins to Disable on Windows Ohai::Config[:disabled_plugins]  =  [ #  The  following  plugins  are  disabled  as  they  are  either  not  needed, #  have  poor  performance,  or  do  not  apply  to  the  Windows  configuration #  we  use.      "c",  "cloud",  "ec2",  "rackspace",  "eucalyptus",  "command",  "dmi",    "dmi_common",  "erlang",  "groovy",  "ip_scopes",  "java",  "keys",    "lua",  "mono",  "network_listeners",  "passwd",  "perl",    "php",  "python",  "ssh_host_key",  "uptime",  "virtualization",    "windows::virtualization",  "windows::kernel_devices" ] 30
31. Summary 31
32. Chef-Vault and Run As moserke / chef-vault Securely store and retrieve certificates and service acct passwords opscode / mixlib-shellout Run commands as another user 32 So I’ve told you a story about how a lack of Run As forced us to compromise on our first Chef project and how that led to the creation of Chef-Vault, and improvements to MixLib::ShellOut to support Windows Impersonation.
33. Manage disks and printers moserke / diskpart-cookbook opscode-cookbooks / windows v1.8.2 has Printer/Printer Port LWRPs 33 • We’ve talked about a new cookbook to manage Windows disks and new LWRPs to manage printers
34. Performance Improvements http://wiki.opscode.com/display/chef/Disabling+Ohai+Plugins 34 • We’ve talked about performance improvements in Chef 11, and how disabling Ohai plugins boost Chef Client performance
35. Call to Action • IIS cookbook not idempotent for options • Better bootstrapping using Kerberos • Better integration with Active Directory 35 And now a call to action
36. Will you join us? http://bit.ly/infeng 36 And finally, a pitch for working with us. 1. At Nordstrom, you have an opportunity to follow your passion 2. You will have an opportunity to use and contribute to open source 3. Our team is small and focused 4. We are at the beginning of a cultural sea change I hope you’ll join us and we’d love to talk with you. Thank you.
37. Go to Adam Edward’s talk right after this • “Cooking on Windows without the Windows Cookbook” • Seacliff A,B,C,D 37
38. http://www.flickr.com/photos/drachmann/327122302/sizes/l/ 38
39. Photo Credits 1.Slide 3: http://www.flickr.com/photos/benedictineuniversity/6021873707/sizes/l/ 2. Slide 4: http://www.flickr.com/photos/kubina/278696130/sizes/l/ 3. Slide 7: http://www.flickr.com/photos/orlando-herb/8167991591/sizes/l/ 4.Slide 9: http://www.flickr.com/photos/ejbsf/8609182524/sizes/h/ 5.slide 10: http://www.flickr.com/photos/ashley-rly/3768328487/sizes/l/ 39