June 21, 13 I’ve made some horrible messes in my day. Many resulting from not understanding the consequences of decisions I made or the technical debt I was incurring and the resulting mess, not just of my own work, but others as well. Configuration management tooling is a rich and complex topic. The culture around the tools is still fairly young and without established “rules” or precedents. In lieu of those, I’d like to share what I think are some of the most important things you need to consider during an implementation.
out of the way, let’s get to know each other a bit. I know people are ambivalent about the whole “raise your hand” thing, but I want to know who’s here and why. So humor me. Who here is ops? Dev? Tools or in between or consultants? Management/Leadership?
out of the way, let’s get to know each other a bit. I know people are ambivalent about the whole “raise your hand” thing, but I want to know who’s here and why. So humor me. Who here is ops? Dev? Tools or in between or consultants? Management/Leadership?
out of the way, let’s get to know each other a bit. I know people are ambivalent about the whole “raise your hand” thing, but I want to know who’s here and why. So humor me. Who here is ops? Dev? Tools or in between or consultants? Management/Leadership?
out of the way, let’s get to know each other a bit. I know people are ambivalent about the whole “raise your hand” thing, but I want to know who’s here and why. So humor me. Who here is ops? Dev? Tools or in between or consultants? Management/Leadership?
out of the way, let’s get to know each other a bit. I know people are ambivalent about the whole “raise your hand” thing, but I want to know who’s here and why. So humor me. Who here is ops? Dev? Tools or in between or consultants? Management/Leadership?
components Friday, June 21, 13 the short version: build everything the same every time and free yourself to do more important work and from problems caused by system inconsistencies.
who’s read my blog or heard me on the podcast knows that I have opinions on just about everything. I think some tools are better than others and that there are certain correct ways to use those tools. But if you may have come here expecting me to discuss the pros and cons of different configuration management tools, you will be disappointed. When it comes to CM tools, the first answer is always YES. Of course you want to use one. Because why wouldn’t you? The second answer can be the result of a tool debate but I’m not here to have one of those today and in general, I feel that answer is, pick what you think is right for you. Because as long as you don’t fight me on the first answer, it’s all good.
21, 13 Anyone who’s read my blog or heard me on the podcast knows that I have opinions on just about everything. I think some tools are better than others and that there are certain correct ways to use those tools. But if you may have come here expecting me to discuss the pros and cons of different configuration management tools, you will be disappointed. When it comes to CM tools, the first answer is always YES. Of course you want to use one. Because why wouldn’t you? The second answer can be the result of a tool debate but I’m not here to have one of those today and in general, I feel that answer is, pick what you think is right for you. Because as long as you don’t fight me on the first answer, it’s all good.
with opinions on what and how Friday, June 21, 13 Anyone who’s read my blog or heard me on the podcast knows that I have opinions on just about everything. I think some tools are better than others and that there are certain correct ways to use those tools. But if you may have come here expecting me to discuss the pros and cons of different configuration management tools, you will be disappointed. When it comes to CM tools, the first answer is always YES. Of course you want to use one. Because why wouldn’t you? The second answer can be the result of a tool debate but I’m not here to have one of those today and in general, I feel that answer is, pick what you think is right for you. Because as long as you don’t fight me on the first answer, it’s all good.
with opinions on what and how When it comes to CM tools... Friday, June 21, 13 Anyone who’s read my blog or heard me on the podcast knows that I have opinions on just about everything. I think some tools are better than others and that there are certain correct ways to use those tools. But if you may have come here expecting me to discuss the pros and cons of different configuration management tools, you will be disappointed. When it comes to CM tools, the first answer is always YES. Of course you want to use one. Because why wouldn’t you? The second answer can be the result of a tool debate but I’m not here to have one of those today and in general, I feel that answer is, pick what you think is right for you. Because as long as you don’t fight me on the first answer, it’s all good.
with opinions on what and how When it comes to CM tools... the first answer is YES Friday, June 21, 13 Anyone who’s read my blog or heard me on the podcast knows that I have opinions on just about everything. I think some tools are better than others and that there are certain correct ways to use those tools. But if you may have come here expecting me to discuss the pros and cons of different configuration management tools, you will be disappointed. When it comes to CM tools, the first answer is always YES. Of course you want to use one. Because why wouldn’t you? The second answer can be the result of a tool debate but I’m not here to have one of those today and in general, I feel that answer is, pick what you think is right for you. Because as long as you don’t fight me on the first answer, it’s all good.
to help it. But the tools don’t require you to understand object oriented programming. Understanding basic primitives like hashes, arrays and strings is very helpful however...
won’t be able to help it. But the tools don’t require you to understand object oriented programming. Understanding basic primitives like hashes, arrays and strings is very helpful however...
You won’t be able to help it. But the tools don’t require you to understand object oriented programming. Understanding basic primitives like hashes, arrays and strings is very helpful however...
right? In the last few years many of us have been transformed by the explosion of open source into mainstream cultures. This includes exposure to Agile and DevOps methodologies. But there was a time when we hadn’t heard of things. And some of us are more clueless than others. It wasn’t until 2010 that I even heard of configuration management as a “thing.”
21, 13 When I first encountered configuration management tools, I had years of infrastructure experience. As a consultant, I walked into new clients all the time and built the same things by hand over and over. I often despaired, not just at hand-crafting, but at how each consultant built things slightly differently. I’d never heard of configuration management tooling although several of us had experimented with centrally located files and ssh push scripting. Unfortunately we could never get good client adoption of these little hacks and every project eventually devolved into wiki docs and tribal knowledge of the inheriting support teams.
often wished for tools that could help homogenize installs and configuration across clients. I often brought up the idea of a common kickstart and package repo collection, but it was met with obstacles and objections mainly revolving around operating system licensing or client constraints or the inability of connecting client networks to internal network locations.
became a “thing” I tried advocating a place on the internet that was secure but connectable and was met with the same lack of interest in changing how things worked. I just didn’t understand why it was so hard to figure out a way to create a repeatable build process across client infrastructures.
21, 13 And I was so bored. You can only install Websphere and JBoss so many different ways and explain connection pooling so many times before you start wondering what else there is to life. I didn’t want to spend the rest of my life, building infrastructure by hand. I wanted OS-level management to be a solved problem so that I could get on with the real fun in life: working with dev teams to optimize and operationalize web apps. I thought about leaving my consulting job but didn’t know what I wanted to do. And the money was great. So I stuck around and eventually I ended up on a project that transformed the way I thought about, well, everything.
was persuaded to return to a very large retail client where I’d designed much of the infrastructure and truthfully never wanted to work on again. But I have a friend, his name is Mike Nygard. He’s brilliant and loves to solve hard problems. You know how you have that favorite band or book author where you don’t need to know anything about their new release before buying it? Well, I respect him immensely and any project he chooses is something I would blindly sign up for.
13 In 2010, I was persuaded to return to a very large retail client where I’d designed much of the infrastructure and truthfully never wanted to work on again. But I have a friend, his name is Mike Nygard. He’s brilliant and loves to solve hard problems. You know how you have that favorite band or book author where you don’t need to know anything about their new release before buying it? Well, I respect him immensely and any project he chooses is something I would blindly sign up for.
I started on a “Platform Transformation” project where the object was taking everything done in the last 10 years and re-working web site architecture, applications and culture all at once. He showed me Chef and said, “go read some ruby tutorials, learn Chef and now write me some cookbooks.” And then I started looking at Chef and trying to figure out how to work it into the existing architecture
June 21, 13 As you accumulate experience, you tend to accumulate baggage. Bias on the “right way” to do things. Right now it’s just weighing you down. Let it go.
attitude: free yourself Friday, June 21, 13 Why are you automating? To remove some level of chance and uncertainty from your world. I love the challenge of automation for itself. Even if you don’t though, the idea that can you free yourself from trivial day to day repetition should be a powerful argument. I have heard there are people who feel their jobs are threatened by automation and configuration management. Never worry about this. This tooling empowers you and frees you from trivia. If you actually manage to automate yourself out of a job, and honestly I’ve never actually heard of it happening, I can guarantee you that your employer will a new challenge for you immediately. And if they don’t, do you have any idea how much people pay for someone who knows an automation tool?
a promise? a fact? rbd ebs nosql orm Friday, June 21, 13 The first time I called bullshit on someone else, I knew I’d had a leveling up moment. I said, “I have no idea what you just said and I don’t think you do either.” Too often we are content to nod like we understand what someone is saying when we have no idea really. How is a defined type different from a parameterized class in Puppet? I had real trouble with that. In 2010 I didn’t know what a Rubygem was and struggled to wrap my head around the idea of a packaging system un-connected to any known operating system. When you can say, “I don’t know,” What you’re really saying is “I know so many other things that I’m entirely comfortable exposing my ignorance on this subject to you.” It’s an empowering feeling. People who can’t admit ignorance are usually afraid because they’re worried about being perceived as not knowing anything. You know things. But it doesn’t matter. Our value isn’t knowing THINGS it in our ability to figure stuff out. Collecting facts is useless without the ability to correlate and extrapolate. Why am going on about this? When you pick up something like configuration management or automation, you will be exposed to a host of new terms, acronyms, processes. It’s ok to not be an instant expert.
management is complex and it can sometimes feel like you’re trapped in a dream with too many targets that keep getting farther away whenever you move toward one. I think in some ways this is because we often operate on instinct. When you attempt to codify and corral infrastructure that you understand best at a visceral level, you realize that it’s harder than you think to quantify and qualify these things. At the same time, you’re learning something new. If your background is systems, it can take a while to get used to the conventions around treating CM like source code. I walked into a client where I knew the infrastructure, but nothing about my configuration management tool. I was learning the tool, conventions, terminology all at once and it was intimidating.
that I had an idea of what it did, I struggled with writing something/anything that made sense and I seriously spent the first couple weeks learning chef not being able to remember the difference between arrays and hashes and constantly mixed up the notation.
program. I’m not a developer. I said, “I hope you don’t expect to write code. Because I can’t.” Years of staring at other people’s perl scripts and some dabbling in java had convinced me that I was a failure at any kind of development work. I purposely stuck to bash scripts after that to avoid embarrassing myself or that sinking feeling of not understanding what I was doing.
WRITE CODE. I can’t program. I’m not a developer. I said, “I hope you don’t expect to write code. Because I can’t.” Years of staring at other people’s perl scripts and some dabbling in java had convinced me that I was a failure at any kind of development work. I purposely stuck to bash scripts after that to avoid embarrassing myself or that sinking feeling of not understanding what I was doing.
13 I DON’T WRITE CODE. I can’t program. I’m not a developer. I said, “I hope you don’t expect to write code. Because I can’t.” Years of staring at other people’s perl scripts and some dabbling in java had convinced me that I was a failure at any kind of development work. I purposely stuck to bash scripts after that to avoid embarrassing myself or that sinking feeling of not understanding what I was doing.
DEVELOPER Friday, June 21, 13 I DON’T WRITE CODE. I can’t program. I’m not a developer. I said, “I hope you don’t expect to write code. Because I can’t.” Years of staring at other people’s perl scripts and some dabbling in java had convinced me that I was a failure at any kind of development work. I purposely stuck to bash scripts after that to avoid embarrassing myself or that sinking feeling of not understanding what I was doing.
what happened? Configuration management is insidious. You truly don’t need to be a dev to use configuration management. Actually I’ve know plenty of devs who have had real trouble with it. Regardless, you tend to absorb some key tenets of programming without actually writing code.
13 Bad attitude can hold you back. Resentment, surliness, condescension and passive aggressive nonsense can manifest in any number of ways. Don’t let it make you “that jerky team member no one wants to talk to because they’re always complaining about something.”
*is* serious business. The only time I hear CM code referred to as scripts is when it’s discussed by people who don’t take it seriously. They often resent being made to use it and will pawn it off on their most junior, least capable team member. They will not take the time to understand basic tenets and then wonder why nothing works later. Calling CM code scripts is passive aggressive dismissal and belittles the effort required to learn and implement intelligent tooling.
systems doesn’t mean you understand configuration management Friday, June 21, 13 just because you know systems doesn’t mean you understand configuration management. I often think I know best about something and then have my mind opened to possibilities that I haven’t thought of. Keep an open mind, learn the conventions of your tooling and, instead of immediately dismissing what you’re told, ask why this over that.
the whole world Pick something small but impactful Prefer something light on data Friday, June 21, 13 When picking a first use case, pick something that makes sense, but also allows you to learn the tool without saddling you with too many problems to solve at once. I often suggest starting with small operating system level configurations. However, I once recommended someone start with Tomcat because they were in need of the ability to stabilize their application environment stat. You also want to pick something that is impactful. If you configure sshd on all the servers and it’s been working fine, no one will notice. But if you have been encountering instability due to config file variance, then this makes sense and you can put a tick mark in the value column. You also want to prefer something light on data.
NOT just migrate your scripts directly into the tool Devs: Do NOT just write a bash block to install your app Friday, June 21, 13 Primitives are developer-speak for the smallest building blocks in a language. These are commonly described as strings, hashes, arrays and similar in programming. With configuration management, the primitives are things like packages, files, users, templates. These primitives are specifically designed to configure state without hand crafted bash scripts doing the work.
readability primitives ensure operating system cross- functionality Friday, June 21, 13 Primitives are developer-speak for the smallest building blocks in a language. These are commonly described as strings, hashes, arrays and similar in programming. With configuration management, the primitives are things like packages, files, users, templates. These primitives are specifically designed to configure state without hand crafted bash scripts doing the work.
Chef Friday, June 21, 13 Primitives are developer-speak for the smallest building blocks in a language. These are commonly described as strings, hashes, arrays and similar in programming. With configuration management, the primitives are things like packages, files, users, templates. These primitives are specifically designed to configure state without hand crafted bash scripts doing the work.
package_version => "2.2.00", package_select => ">=", package_policy => "addupdate"; } CFEngine Friday, June 21, 13 Primitives are developer-speak for the smallest building blocks in a language. These are commonly described as strings, hashes, arrays and similar in programming. With configuration management, the primitives are things like packages, files, users, templates. These primitives are specifically designed to configure state without hand crafted bash scripts doing the work.
alias => openssh, require => Package[openssl] } Puppet Friday, June 21, 13 Primitives are developer-speak for the smallest building blocks in a language. These are commonly described as strings, hashes, arrays and similar in programming. With configuration management, the primitives are things like packages, files, users, templates. These primitives are specifically designed to configure state without hand crafted bash scripts doing the work.
the tool: read the source code Friday, June 21, 13 sometimes the only way to know what’s going on, especially with younger tooling, is to dig in and read the source code. Even if you don’t know *exactly* what it’s doing, you can often get a better feel for it than just reading the documentation which can be wrong.
21, 13 I have a whole talk I’m preparing just on toolchaining and testing. If you are using CM, starting getting familiar with testing tools. It’s an enormous topic and one we in the community are trying do more talking about.
a vacuum. It should be one of many pieces in your curated infrastructure. Own your entire world or you will be owned by it. And you won’t be happy. What am I talking about with this?
are more than just the sum of their parts. Together they should make up a high functioning ecosystem with logical, consistent integration points and dedicated curators. I’m talking about things like systems access to the internet, packages and repositories, artifacts, version control and environment management.
about technical debt. We talk about it a lot. Accepting that we are going to create technical debt is part of the creative process. We don’t always know everything we need to at the beginning of a project and we know we’ll have to rework things as we learn. Here’s the thing. There’s a big diff between ....
bad. I plugged a few things in. It’s a bit messy but I can probably fix it without too much trouble with some recabling and maybe a few zip ties. On the other hand....
understand what we can do that won’t lead us to bigger, unsolvable problems later. What are you going to do with this? Burn it down? You need to take a few months and redo everything. There is nothing redeemable about this. Someone should have seen what was happening with the very first rack and started figuring things out then. But it’s really easy to lose sight of problems over in the data center when you’re dealing with stuff at your desk. You need to be aware things that aren’t right in front of your face, before they bite you in the ass. Right?
understand what we can do that cause us to be carrying this baggage around for the rest of our careers. And you will be known by the things that you create.
tool is going to be all rainbows and unicorns, well, it might. But you have to be careful, or just when you think you’re going to collect your awesome toy surprise....
stuff is complex. Chef can’t live by itself in a vacuum. You need to think about how you’re going to approach it. If you just think about the CM tool and ad hoc everything else, this guy is going to be the least of your problems.
Internet Friday, June 21, 13 Much of the world assumes unfettered internet access for all your servers. The assumption plays out in different ways. Open source tools like Puppet modules and Chef cookbooks often have internet urls at which to download software and then compile from source. Or they assume EPEL access. Anyone spending more than five minutes in the enterprise knows this is almost never true. The best you can hope for is a Bastion Host from which you can host package mirrors.
13 Configuration management tools were made to work with package managers. When implemented with package repositories available you will write significantly less configuration code to achieve the same results.
hint: you do Friday, June 21, 13 ---who cares where we keep our packages? I give you a hint: you do, because 6 months of not caring and you will have a guaranteed hot mess on your hands. I kept thinking that I should care more about software storage on my first project, but I was so busy with Chef, that I never took the time to create a system that was easy to use for anyone. This means that everyone else using chef just did what they needed at that moment to cope. In the days before we had the omnibus installer, we needed Ruby, Rubygems and access to rubygems.org because that’s where gems come from. RH 5 shipped with ruby 1.8.5, pretty much unusable with Chef RH 6 ships with ruby 1.8.7 which is less traumatic today because we have Omnibus. But if you want one single other thing that isn’t Chef, you still need all these things. what do you do when your servers can’t access the internet? If you’re dumb like us, you download ruby source and rubygems tarball and install them from scratch on every server you build. I worked with a colleague last year to eliminate the Ruby compile from source on our VM builds and decreased build time by 5 minutes. If you’re smart, you create a central internal “extras” repository and use FPM to create precompiled packages accessible to your OS package manager This not only speeds up build time, but it also makes it REALLY OBVIOUS to anyone with a brain that these things are available to them. So this then also avoids people getting new binaries and dropping them all over
did why you did it and so does the person who replaces you next year when you take off to bootstrap your startup Own Your Version Control Friday, June 21, 13 CM code is code. Don’t let anyone tell you differently.
and maintain system state use different provisioning processes in Dev vs Prod deploy apps differently in different environments Own Your Integrity Friday, June 21, 13 * Don’t turn off your chef/puppet/cfengine agents. If you use push-based tooling, don’t cherrypick or waffle about what gets managed that way. I’ve seen people turn off chef clients. I’ve seen them only enable chef clients in Stage and Production for deployments and then be sad and confused when things don’t work. * Don’t provision your servers with one set of configuration data and then expect your tool to come through and change things. While in theory this should work, it’s a dreadful idea. Why isn’t the configuration data centrally managed? For example, I have seen where people use Cobbler to do a heavy amount of configuration during bootstrap and then use Chef to come through and “enforce” these configurations. Can anyone see a problem with this?
sascha_d
aarron
madoxten
davidcarrasco
addyosmani
jct
sstephenson
csswizardry
maltzj
bcantrill
notwaldorf
inesmontani
gurzu
