Changing the Laws of Engineering with GitHub Pull Requests (DevWeek 2015)

For an engineering team to sustain its culture as it grows, even more important than what processes it adopts is how those processes undergo change. At New Relic, we have a culture of openness that we intend to keep, so we've been using the same process to change our culture that we use to change our code: GitHub Pull Requests. Learn why and how we opened up our Engineering Handbook.

Transcript

1. Changing the Laws of Engineering With GitHub Pull Requests Monday, February 9, 15 30 minutes Process, documentation, and culture -When you leave, I hope you'll be thinking about your engineering processes. -Asking yourself why and how you change them. -You will have more powerful tools for sustaining your engineering culture.
2. Ralph Bodenner Director of Engineering New Relic @ralphbod Monday, February 9, 15 But I’ll start with a story about another company.
3. Monday, February 9, 15 Back in 2012, this document was leaked. Valve, the famous game developer, had a flat org structure.
4. Monday, February 9, 15 Their handbook paints a picture a lovely culture of autonomy and shipping awesome software. Described how to do it. Processes. We were inspired by that at New Relic. We wanted our culture to look like that.
5. Processes Policies Best Practices Monday, February 9, 15 We aren’t Valve. We are not flat. We’re not a game company, we do enterprise software services. But we value autonomy, innovation, and boldness. How do we get that?
6. Process: How to recognize people’s contributions? Monday, February 9, 15 We value experimentation, so we’ll try an experiment in making our processes more like Valve’s.
7. Title Salary Promotions Reviews Praise Monday, February 9, 15
8. The Great Title Debate Monday, February 9, 15 Right around the time of Valve handbook, engineering management discussed, in meetings over the course of many weeks, what our titles should be.
9. Engineer Monday, February 9, 15 Flat. Simplicity. Recognition of merit over seniority.
10. Monday, February 9, 15 Engineer who coined “software engineering” Margaret Hamilton, lead flight software engineer on Apollo mission That’s her code
11. Software Engineer Senior Software Engineer Lead Software Engineer Principal Software Engineer Distinguished Software Engineer Engineering Manager Senior Engineering Manager Director of Engineering Monday, February 9, 15 Lots of reasons for what we ended up with. Not ideal (as in idealistic). How you promote and recognize people embodies your culture.
12. “Process” == What you do because it works Monday, February 9, 15 Not about which ones you’re using. It works for the people, it works for the business. You want to keep doing what works. It embodies your culture. Write it down.
13. Monday, February 9, 15 So here’s your process. It works. Need a manual. Everyone new and old can read it and do things well.
14. HOW ABOUT A WIKI? Monday, February 9, 15 - Backpack - Wikispaces - Confluence This is Ward.
15. The Append-only Wiki Monday, February 9, 15 People just added on, they didn’t change or remove. There were comments. Big pile of crap. We had to do annual purges. Became untrustworthy. We have Ward!
16. Inertia Monday, February 9, 15 Not just the time spent rewriting that slows you down. People read stuff and believe it. You continue in wrong directions. Things that are written down have weight. > Dunbar’s Number. You’re too big to convey all the nuances of how things have changed since you wrote it down, since the changes are only in your heads. Multiple offices in multiple timezones. Slow-to-change written word is the medium.
17. The manual tests that wouldn’t die Monday, February 9, 15 Another deployment story. We used to do manual tests after every production deploy. We had a wiki page on it, a list of accounts to test with. New engineers would read about the tests and take them really literally. We never found problems, but spend hundreds of person-hours doing it. Can’t afford that. We deleted the wiki pages.
18. Monday, February 9, 15 Why delete them? We didn’t want our process to become untrustworthy. Misleading is worse than missing. Read well-meaning but wrong stuff too many times and it loses any credibility it might have had.
19. There was a good reason for the crap that doesn’t work anymore. Monday, February 9, 15 Well-meaning! But it’s confusing. We care about our customers’ experience. We operate a SaaS, where production is the one true reality. We had three different docs on our wiki for deploying one site. More than a dozen slightly different ones on onboarding and a equal number on hiring.
20. Culture > Process Monday, February 9, 15 If we had left the process the same, people would have continued to waste time on things that didn’t have value. We value spending time valuably, so with this process, that aspect of culture was being eroded.
21. Culture is the reason for Process Monday, February 9, 15 This is the core problem: We care about our culture. We want it to live and thrive. That’s why we have processes. But if those processes don’t change to close the gap with your culture, you lose.
22. You need a way to close the culture/ process gap. Monday, February 9, 15 Metaprocess! We tried again.
23. The Engineering Book Monday, February 9, 15 Growing. 20 five years ago. 500-something employees by the time we filed our S-1 last year. We needed to double-down on our culture of writing things down. It didn’t go anywhere.
24. Same metaprocess, same result Monday, February 9, 15 Same process for changing process that we had before.
25. A spark: Architecture reviews Monday, February 9, 15 Time passes. A company we admire, Simple, doing engineering in Portland. They wrote architecture docs in GitHub. Commit the docs first, then the software becomes real after review. Moved our operational docs into READMEs, some repos just about describing our systems.
26. Monday, February 9, 15 Change Comment Yes/no (merge)
27. git commit -am “Change our values” Monday, February 9, 15 Culture as code. Same workflow as for code. That works for engineers. Lower mental barrier to contribution. Wiki is “a manager tool”.
28. Monday, February 9, 15
29. Monday, February 9, 15
30. Monday, February 9, 15 Our version of this
31. Metrics • 73 PRs merged since July 2014 • 5 engineers contributing (16 managers) Monday, February 9, 15 Wouldn’t be a New Relic presentation without metrics. +2 people outside Engineering
32. One change per day Monday, February 9, 15 Good amount of activity. What did we learn?
33. Don’t need meetings Monday, February 9, 15 You don’t have to have a meeting. People get left out who can’t (remote) or shouldn’t (engineers) be in the meeting. No conversation on the side for those privileged by title or location. Reinforces a meeting-averse culture.
34. Do write together Monday, February 9, 15 A focused conversation of a few people generates a proposal that is worth sharing. Informal group of managers, engineers, support staff get together to write PRs together.
35. Monday, February 9, 15 Process pairing. Looks a lot like this. But we’re not the idealized version of Valve. We don’t just work completely independently. We opened up our Book of Engineering Laws to everyone, but still maintain executive veto power.
36. Open Source is not a democracy. Monday, February 9, 15 We have a few committers, everyone’s a contributor. Everyone has a voice, but not a vote.
37. “Lowers the contribution barrier...” Monday, February 9, 15 Everyone has to understand git/GitHub to contribute. No Edit button. Wiki or Quip is better at this right now. Do anything you can to lower barrier.
38. git clone processes Monday, February 9, 15 You can edit the whole world and propose something new. No need to get it right before you edit, or need to get rights. Freedom to think of the world as it should be.
39. Monday, February 9, 15 Another culture example. Someone else’s reimagining of the world. How/why did they do it this way? This document tries to explain the why. It’s pretty good, actually.
40. Why did you think this was a good idea? Monday, February 9, 15 What did you know at the time? What assumptions went unchallenged? What was important at the time?
41. Why > What Monday, February 9, 15 It’s hard to read the why in a document.
42. Junior Software Engineer Software Engineer Senior Software Engineer Principal Software Engineer Distinguished Software Engineer Engineering Manager Senior Engineering Manager Director of Engineering Monday, February 9, 15
43. Software Engineer Senior Software Engineer Lead Software Engineer Principal Software Engineer Distinguished Software Engineer Engineering Manager Senior Engineering Manager Director of Engineering Monday, February 9, 15
44. Pull Request knows Why Monday, February 9, 15 It also knows who and when.
45. Ask why. Monday, February 9, 15 Everyone should be able to ask why. Our culture (autonomy, boldness, urgency, customer focus) demands it.
46. Thank you! Slides: http://bit.ly/pull-request-your-process Monday, February 9, 15 http://bit.ly/pull-request-your-process
47. References • Valve Handbook for New Employees http://www.valvesoftware.com/company/ Valve_Handbook_LowRes.pdf • GitHub: Quick Pull Requests https://github.com/blog/1945-quick-pull-requests • Netflix culture deck http://www.slideshare.net/reed2001/culture-2009 Monday, February 9, 15
48. Image References • Valve Working without a Boss http://www.blogcdn.com/www.joystiq.com/media/2012/04/screen-shot-2012-04-23-at-11.24.33-am.jpg • Valve cover http://i.kinja-img.com/gawker-media/image/upload/s--ANyPdiwb--/17k9nffv4tn36jpg.jpg • Margaret Hamilton with her code http://upload.wikimedia.org/wikipedia/commons/2/2e/Margaret_Hamilton.gif • Employee Handbook http://www.hrhero.com/hl/articles/wp-content/uploads/2013/09/EmployeeHandbook.jpg • Ward Cunningham http://upload.wikimedia.org/wikipedia/commons/3/30/Ward_Cunningham_1.jpg • CA carcinogen warning sign http://images.mysafetysign.com/img/lg/S/chemical-cause-cancer-warning-sign-s-0289.png • Tip! http://www.mobileeaglemedia.com/wp-content/uploads/2014/04/Tips-voor-Makelaars.jpg • Pairon chair http://www.openscience.org/blog/wp-content/uploads/2009/06/pairon.jpg Monday, February 9, 15