When Chef Runs Riot

A presentation I gave at Riot Games on Chef and transforming culture. Riot is doing a lot to transform gaming culture which I feel gives their tech teams a leg up on understanding tools like Chef. This is a re-crafting of similar points from my Chefconf talk with a new introduction

Transcript

1. when chef runs RIOT Friday, July 5, 13
2. brattyredhead.com @sascha_d The Ship Show Podcast sascha bates Friday, July 5, 13
3. I mess things up so you don’t have to Friday, July 5, 13 Some people call this experience. Fuck up enough and people will start paying you big money to tell them how to not fuck up. None of them will listen, but you’ll still get paid.
4. credentials? making messes with Chef in the Enterprise and at home since 2010 Friday, July 5, 13
5. Friday, July 5, 13 I played an online game once
6. Friday, July 5, 13 I played an online game once
7. Friday, July 5, 13 I played an online game once
8. I’m not much with the gaming but... Friday, July 5, 13
9. I’m not much with the gaming but... Friday, July 5, 13
10. I’m not much with the gaming but... Friday, July 5, 13
11. I’m not much with the gaming but... Friday, July 5, 13
12. WHY AM I HERE? Friday, July 5, 13 When Riot asked me to visit and give a talk, I may have squeed. I’m a big fan of Berkshelf and the Berks team. It’s made a massive impact on the community because it healed a lot of procedural pain points. But also, it’s opinionated. We tend to avoid opinionated in the chef community but sometimes the stress of figuring out what’s best while also trying to support your infrastructure is too hard. So even if it doesn’t work out long term, it gives you a starting point for managing your stuff while figuring out your own opinions. The berks team themselves are always friendly and nice and wicked good toolmakers.
13. LEAGUE OF LEGENDS what I knew and the wrong assumptions I made Friday, July 5, 13 I first heard of LoL because of Berkshelf. I know about Game company culture though. While I’d read the Valve employee handbook I assumed it was an outlier. Despite my love of Bioware games, I’m not under any illusions about what it’s probably like to work there. I know people who’ve worked in the gaming industry: Noah Kantrowicz and Seth Thomas are both full of stories. I’m harshly judgmental of online games. I don’t think there’s much they can do to keep my interest and my aversion to obnoxious online personalities has kept me away. When I hear “PVP” I think “jerks who run around and beat up the new kid for their lunch money.” I assumed the creation and open sourcing of Berkshelf was a fluke, probably born of a death march problem solvable only by rockstar super heroes who were then rewarded with the gift of being able to opensource their code.
14. LEAGUE OF LEGENDS gaming company - death march culture what I knew and the wrong assumptions I made Friday, July 5, 13 I first heard of LoL because of Berkshelf. I know about Game company culture though. While I’d read the Valve employee handbook I assumed it was an outlier. Despite my love of Bioware games, I’m not under any illusions about what it’s probably like to work there. I know people who’ve worked in the gaming industry: Noah Kantrowicz and Seth Thomas are both full of stories. I’m harshly judgmental of online games. I don’t think there’s much they can do to keep my interest and my aversion to obnoxious online personalities has kept me away. When I hear “PVP” I think “jerks who run around and beat up the new kid for their lunch money.” I assumed the creation and open sourcing of Berkshelf was a fluke, probably born of a death march problem solvable only by rockstar super heroes who were then rewarded with the gift of being able to opensource their code.
15. LEAGUE OF LEGENDS gaming company - death march culture online game - bratty PVP what I knew and the wrong assumptions I made Friday, July 5, 13 I first heard of LoL because of Berkshelf. I know about Game company culture though. While I’d read the Valve employee handbook I assumed it was an outlier. Despite my love of Bioware games, I’m not under any illusions about what it’s probably like to work there. I know people who’ve worked in the gaming industry: Noah Kantrowicz and Seth Thomas are both full of stories. I’m harshly judgmental of online games. I don’t think there’s much they can do to keep my interest and my aversion to obnoxious online personalities has kept me away. When I hear “PVP” I think “jerks who run around and beat up the new kid for their lunch money.” I assumed the creation and open sourcing of Berkshelf was a fluke, probably born of a death march problem solvable only by rockstar super heroes who were then rewarded with the gift of being able to opensource their code.
16. LEAGUE OF LEGENDS gaming company - death march culture online game - bratty PVP berkshelf - a fluke what I knew and the wrong assumptions I made Friday, July 5, 13 I first heard of LoL because of Berkshelf. I know about Game company culture though. While I’d read the Valve employee handbook I assumed it was an outlier. Despite my love of Bioware games, I’m not under any illusions about what it’s probably like to work there. I know people who’ve worked in the gaming industry: Noah Kantrowicz and Seth Thomas are both full of stories. I’m harshly judgmental of online games. I don’t think there’s much they can do to keep my interest and my aversion to obnoxious online personalities has kept me away. When I hear “PVP” I think “jerks who run around and beat up the new kid for their lunch money.” I assumed the creation and open sourcing of Berkshelf was a fluke, probably born of a death march problem solvable only by rockstar super heroes who were then rewarded with the gift of being able to opensource their code.
17. Wrong Friday, July 5, 13 I was utterly wrong.
18. TRANSFORMATION welcome to the 21st century Friday, July 5, 13 Berkshelf is transforming how large organizations use Chef. What I learned in my reading is that Berkshelf is awesome because Riot culture, not because of it. You’re not just making games, you’re transforming the way people think about games, gaming communities and game companies.
19. 21st Century Game Shop RIOT CULTURE Friday, July 5, 13 I’ve read a bit about your company culture and I’ve watched Michael Saladino’s presentation on redlining. You are showing people that it’s possible to be awesome and not make every day a death march. You want everyone who works here to love gaming. And that’s because.....
20. 21st Century Game Shop transforming COMMUNITY Friday, July 5, 13 You’re also transforming community. I read the ars technica article on toxic player reform and was impressed. I like that the company is deeply involved in the community and that the answer is not just banning brats, but first attempting to identify and head off bad behavior and then enlisting the community to assist with judging and enforcement.
21. TRANSFORMING how we think about INFRASTRUCTURE Friday, July 5, 13 Why am I talking about this? The transformative thinking required to bootstrap Riot is the same kind of thinking we need when considering so many problems like, how do we scale to thousands of servers all over the world? That one question spawns so many others. In order to scale reliably, how do we cope with distributed performance, reliable configuration? How do we make our apps self aware enough to allow for smart orchestration? And these questions beg the question, how do we as devs and sysadmins change our thinking to encompass all this change? We can’t adopt new tools and process while keeping our beliefs rooted in the previous decade where we were intimately acquainted with all of our hardware.
22. wasn’t it awesome when it took 3-6 weeks to get a dev server and you got to share it with 60 other people? - nobody ever Friday, July 5, 13 A common way we see orgs speeding up workflow is getting access to easy,self service virtualization for dev and systems teams. What I’ve seen is that this experiences a brief honeymoon phase where everyone is thrilled with being able to make their own servers whenever they need one, regardless of how hard or confusing the process is or how unreliable the system is. After this becomes the new normal, we realize that instaserver isn’t actually going to solve all our problems but instead introduces a different kind of uncertainty into our pipeline.
23. how devs think about systems how sysadmins think about code TRANSFORMING Friday, July 5, 13 It used to be we could have anything we wanted if we could wait 6-12 weeks for it. Now we can have anything we want in 5 minutes if we can figure out how to build it, stabilize it and operationalize it. So we’re experiencing an upheaval not just with our tools but with how we think about things. Easy access to virtualization, and accessible tools written in Ruby and Python like Chef are changing us. We are experiencing expansions and reversals of traditional roles.
24. internet buffet overload on-demand, empty servers oh shit empty server halp chef vim java -Dwtf bundle eclipse #!$%#! nginx tomcat omg jboss Friday, July 5, 13 So you can have your own servers whenever you want. But oh shit, they don’t come with anything on them. The flip side of your instant server lifestyle means no one is lovingly handcrafting your individual servers. So you go out and find your own software. But it doesn’t run very well because actually, you don’t really know much about servers and java containers except that they’re the place you put your app. All of the sudden you’re spending a lot more time on stuff you never needed to know before because the responsibility has shifted. You may find yourself tempted to be angry and resentful that you’re no longer just a java dev and have to know all this other stuff too. But you wanted this. With great power comes great responsibility.
25. scm wat? scm wtf? code wat? configuration management wtfucking hell? halp! Friday, July 5, 13 omg code - as a sysadmin, all we used to have to do is lovingly handcraft servers and hand them off to devs or deploy them into production. The hardest part might be documenting the process for newbs on sharepoint or a shared email and then showing them how to do all the stuff you never actually wrote down. I knew a guy who actually kept notes in a dotfile on the root users’s home on the admin server. Now you have to figure out a way to configure a hundred servers at once that doesn’t require you to fix a bunch of stuff by hand later. Or bootstrap a bunch of VMs without doing each one by hand. And we have to put it in version control that we might first have to install and configure and OMG
26. with great power comes great responsibility and even more systems work Friday, July 5, 13 With all these new ideas and concepts coming at you from all sides it’s easy to get caught up in the moment and fall back on old behaviors and beliefs, especially that we can continue to do things the way we’ve always done them, by shoehorning our existing process into new tools because that’s what’s most comfortable to us. What this looks like: developers half-assing non-dev activities and hoping that ops can just cope when it gets to them. Ops trying to directly migrate scripts into configuration management tools and failing to account for new process and learning required to make the tool successful.
27. HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE HUE letting your chef run RIOT Watch out for the angry man with the big knife Friday, July 5, 13 The original title for this was Doom Your Chef in 3 easy steps. I thought it rather incongruous to reference Doom here though. So I want to talk about 3 broad categories of ways we can mess up without really intending to -rather through lack of time or lack of understanding the imperatives. When you lose control of your tools they can take on a life of their own.
28. Friday, July 5, 13
29. ignore your infrastructure Friday, July 5, 13
30. ignore your infrastructure test nothing Friday, July 5, 13
31. ignore your infrastructure test nothing write scripts, not code Friday, July 5, 13
32. a package manager a package repository a substitute for version control chef is not ignore your infrastructure Friday, July 5, 13 But you need these things for successful configuration management
33. ignore your infrastructure insert package repository rant here package repos configuration management Friday, July 5, 13 life is just easier with package and artifact repos. Tools like Chef are designed to work with package managers. When you prefer tarballs to rpms/debs/gems, you are subverting the tool. No idempotence. Newbs don’t know how to write good checks.
34. ignore your infrastructure internet happens Friday, July 5, 13 You can approach internet access in myriad different ways. The important thing is that you need to consider Much of the world assumes unfettered internet access for all your servers. The assumption plays out in different ways. Open source tools like Puppet modules and Chef cookbooks often have internet urls at which to download software and then compile from source. Or they assume EPEL access. Anyone spending more than five minutes in big companies knows this is almost never true. The best you can hope for is a Bastion Host from which you can host package mirrors. Regardless of what the rules are, you need to understand your limitations and establish methods for working with them. This could include things like internal mirrors, custom repos or a tool like Artifactory that lets you not just host yum repos, but caches packages and acts like a proxy if you need it.
35. ignore your infrastructure hand craft all the things Friday, July 5, 13 with great power comes great responsibility and even more systems work. Now you get servers in minutes but that means you need to be able to cope with that reality. Make it a collaborative effort between dev and systems. Or this is why we have tools teams these days. Having a cross functional team own the tools that everyone wants to use is a great way to get automated, repeatable results from your virtualization
36. ignore your infrastructure artisan dev environments are awesome hand craft all the things Friday, July 5, 13 with great power comes great responsibility and even more systems work. Now you get servers in minutes but that means you need to be able to cope with that reality. Make it a collaborative effort between dev and systems. Or this is why we have tools teams these days. Having a cross functional team own the tools that everyone wants to use is a great way to get automated, repeatable results from your virtualization
37. ignore your infrastructure artisan dev environments are awesome don’t automate: write pages of dev env instructions on the wiki hand craft all the things Friday, July 5, 13 with great power comes great responsibility and even more systems work. Now you get servers in minutes but that means you need to be able to cope with that reality. Make it a collaborative effort between dev and systems. Or this is why we have tools teams these days. Having a cross functional team own the tools that everyone wants to use is a great way to get automated, repeatable results from your virtualization
38. ignore your infrastructure artisan dev environments are awesome don’t automate: write pages of dev env instructions on the wiki bonus points if the wiki is Sharepoint hand craft all the things Friday, July 5, 13 with great power comes great responsibility and even more systems work. Now you get servers in minutes but that means you need to be able to cope with that reality. Make it a collaborative effort between dev and systems. Or this is why we have tools teams these days. Having a cross functional team own the tools that everyone wants to use is a great way to get automated, repeatable results from your virtualization
39. ignore your infrastructure artisan dev environments are awesome don’t automate: write pages of dev env instructions on the wiki bonus points if the wiki is Sharepoint don’t homogenize: ensure everyone has a slightly differerent version of everything. This helps keep things interesting hand craft all the things Friday, July 5, 13 with great power comes great responsibility and even more systems work. Now you get servers in minutes but that means you need to be able to cope with that reality. Make it a collaborative effort between dev and systems. Or this is why we have tools teams these days. Having a cross functional team own the tools that everyone wants to use is a great way to get automated, repeatable results from your virtualization
40. ignore your infrastructure test nothing write scripts, not code Friday, July 5, 13
41. Who cares if I bring down all the other people developing on Chef-provisioned VMs and servers? TEST NOTHING testing is dumb Friday, July 5, 13 Who here actually thinks testing is dumb? Be careful. Jamie’s head will asplode. when you are writing provisioning code or deployment code that affects 10, 20, 100, 500 other people, it pays to be careful.
42. TEST NOTHING testing is dumb I’m just fixing a typo ....with another typo It’s just new functionality ...with more logic Friday, July 5, 13 When you think your changes are too small to warrant testing because they’re totally straightforward and won’t hurt anything No really, think again. I can’t tell you how many times I’ve seen where we’ve just checked in without any testing and then had someone at another table look up an hour later and say, “did you guys change something in the common_collection?”
43. TEST NOTHING baby steps Syntax testing with Knife - does it parse? Linting with Foodcritic - is it pretty and resilient? Convergence testing with Vagrant - does it build? Friday, July 5, 13 You know all about these right? With Riot’s automation team leading the charge, I imagine you’re well set up with basic local verification testing. So I’ll skim. Don’t upload or check in your cookbooks before running these basic checks. Some syntax errors will actually keep you from uploading although not checking in to source. Seriously. If you “just fix a typo” and check it in without testing, it will bite you in the ass when you least expect it and other people around you. I’ve done it and other people around me have done it. If you upload it to Chef without versioning and workflow management, anyone using that cookbook will immediately get your change and, if it breaks something, you could have 50 pissed off people just like that.
44. sysadmins & testing wtf is testing? “what do you mean, write a test?” “everything starts, what else do you want?” Friday, July 5, 13 I’ve always understood that things need to be verified. But if you’re a dev, perhaps you don’t know how entirely foreign the idea of a unit test is to a sysadmin? Or that there are different kinds of tests for the same code? I really only started wrapping my head around unit tests in the last year and have only tried to write some in the last few months. Unit tests for cookbooks are gaining acceptance because it’s far faster to verify your code is at least correct with unit tests than it is to spin up a vagrant VM and run an actual converge. I’ve been trying to think about unit tests for libraries and chef resources and this generally causes an epic existential crisis where nothing gets done and I over analyze the hell out of everything.
45. sysadmins & testing we understand test -f /some/file/somewhere Friday, July 5, 13
46. functional testing it "creates directories" do directory("/etc/").must_exist.with(:owner, "root") assert_directory "/etc", "root", "root", 0755 end level up Friday, July 5, 13 Even when it seems obvious what to be testing, I’ve struggled with over analytical existentialism. Should I use node attributes in my functional/integration testing? I generally opt for NO because with functional tests I want to verify that chef is matching up what should be in place. If I check that chef set the attribute and the attribute has a typo, what have I really just tested? Nothing at all. On the other hand, a way I work with this limitation is to not just write tests against actual chef blocks, but, for example, if I am testing that I correctly set a static route, I also write a test to see if the server can connect to something on the other end of that route. You can do that in multiple ways: a tcp socket connection, curl, net/http.
47. test nothing advanced level up TDD Baby! write your tests first Friday, July 5, 13 Anyone doing this? Yeah I’m not there either, although I flirt with it from time to time. This is an awesome place to be and one you should strive for. It’s a real maturity indicator, both personally and organizationally (I hear).
48. ignore your infrastructure test nothing write scripts, not code Friday, July 5, 13
49. write scripts, not code learn chef, don’t just do it porting scripts into Chef is a recipe for eternal sadness Friday, July 5, 13 Know how you can tell the diff between a place who is “doing agile” and actually is agile? It’s the same with Chef. When adopting a configuration management tool, it’s important to understand the way it’s designed to work and not just shoehorn you existing process into it.
50. bash "update_ssh" do code <<-EOH sed -i -e 's/AuthorizedKeysFile.*authorized_keys/ AuthorizedKeysFile \\/\\.keys\\/%u\\/ authorized_keys/g' /etc/ssh/sshd_config EOH end bash “ssh_dns” do code <<-EOH sed -i -e 's/#UseDNS.yes/UseDNS no/g' / etc/ssh/sshd_config EOH end why we hate script ports Friday, July 5, 13
51. write scripts, not code wtf was that?! Friday, July 5, 13
52. write scripts, not code learn the primitives template file package Friday, July 5, 13
53. write scripts, not code dsl trumps scripts package "ssh" do action :install end service "sshd" do action [:enable, :start] end template "/etc/ssh/sshd_config" do action :create mode 0644 notifies :restart,"service[sshd]" end Friday, July 5, 13
54. # Cookbook Name:: keys # Recipe:: common # Author:: Sascha Bates keys = [] search('public_keys',"tags:common").each { |k| keys << k } search('public_keys',"tags:chef AND tags:#{node.env}").each { |k| keys << k } keys.each do |k| key_type, key_part, key_comment = k['pub_key'].split(' ') ruby_block "root_keys_#{k['id']}" do Chef::Log.debug("test condition: grep #{key_part} #{keyfile}") not_if "grep #{key_part} #{keyfile}" block do File::open(keyfile, 'a') do |f| Chef::Log.debug("Adding #{key_comment} to #{f.path}") f << k["pub_key"] << "\n" end end end end Friday, July 5, 13 One of the first things I wrote. Represents a mighty step forward in my ruby ability and a giant cluster for someone to read when trying to debug a recipe.
55. write scripts, not code dsl trumps code # Cookbook Name:: keys # Recipe:: common # Author:: Sascha Bates authkey “common_key” do action :add user “root” end Friday, July 5, 13 Keep code out of your recipes.
56. you don’t have to write code to read it write scripts, not code read the source code Friday, July 5, 13 sometimes the only way to know what’s going on, especially with younger tooling, is to dig in and read the source code. Even if you don’t know *exactly* what it’s doing, you can often get a better feel for it than just reading the documentation which can be wrong. I learned more from reading the feature specs for Berkshelf than I ever did using the `help` command.
57. but when chef runs RIOT happy chefs abound Friday, July 5, 13 When you’re in control, when you understand the consequences of technical decisions by taking the time to understand the tool
58. the talk I was supposed to give http://www.youtube.com/watch? v=pHmU0aNkENchyperlink if you really want to just see my chef conf talk... Friday, July 5, 13