Tweet
Tweet

Slide 1

Slide 1 text

Changing the Laws of Engineering With GitHub Pull Requests Ralph Bodenner Director of Engineering, New Relic @ralphbod Slides https://bit.ly/pull-request-your-culture Wednesday, May 27, 15 30 minutes Culture. Relation to process, documentation -When you leave, you'll be thinking about your engineering processes. -Asking yourself why and how you change them. -You will have more powerful tools for sustaining your engineering culture.

Slide 2

Slide 2 text

Wednesday, May 27, 15 Tell our story of how New Relic Engineerings makes our processes serve our culture. Our story starts with a story about another company.

Slide 3

Slide 3 text

Copyright 2015 Valve Corporation Wednesday, May 27, 15 Inspiration is this. (Read it?) Back in 2012, this document was leaked. Valve, the famous game developer, had a flat org structure.

Slide 4

Slide 4 text

Copyright 2015 Valve Corporation Wednesday, May 27, 15 Their handbook paints a picture a lovely culture of autonomy and shipping awesome software. Described how to do it. Methods/processes.

Slide 5

Slide 5 text

LET’S DO THIS Wednesday, May 27, 15 In our way at the time, with managers in a room... 3 years later

Slide 6

Slide 6 text

Wednesday, May 27, 15 This is ours! That is not an image of an insignia from a space-traveling TV show owned by a multinational media conglomerate. But the artifact is not as important as how we create it.

Slide 7

Slide 7 text

Wednesday, May 27, 15 And our way of writing it: “Process PRs” ... After much debate and learning. Needed the document, but the process of writing is the most important

Slide 8

Slide 8 text

Culture Why you do things. Metaprocess How you change. Process What you do, today. Entropy Holding you down. Wednesday, May 27, 15 Terms Aspirational

Slide 9

Slide 9 text

1,054 64 187 25 comments commenters (~30% of Engineering) PRs merged (+30 closed) engineers committing (+25 others) Wednesday, May 27, 15 Results since July 2014 (10 months) More engineers than managers are commenting (38 vs. 26)

Slide 10

Slide 10 text

Processes embody culture Processes are repeatable (efficient, fair) Transparent Inclusive Discoverable Single source of truth, written down Update/delete easily Bias for action Autonomy Culture > Process What we want Wednesday, May 27, 15 Learned what we want today after 5 years of experimentation

Slide 11

Slide 11 text

The Great Title Debate 1 / 5 Wednesday, May 27, 15 Right around the time of Valve handbook, engineering management discussed, in meetings over the course of many weeks, what our titles should be.

Slide 12

Slide 12 text

Titles: Method to recognize people’s contributions Wednesday, May 27, 15 Why titles?

Slide 13

Slide 13 text

Titles Compensation Promotions Reviews Wednesday, May 27, 15 We want these to be consistent and understood by everyone. That would embody our culture, of transparency and equity. Anyone should be able to read what titles are defined by.

Slide 14

Slide 14 text

Engineer Wednesday, May 27, 15 Flat. Simplicity. Recognition of merit over seniority.

Slide 15

Slide 15 text

Wednesday, May 27, 15 Engineer who coined “software engineering” Margaret Hamilton, lead flight software engineer on Apollo mission What did we end up with?

Slide 16

Slide 16 text

Software Engineer Senior Software Engineer Lead Software Engineer Principal Software Engineer Distinguished Software Engineer Engineering Manager Senior Engineering Manager Director of Engineering Wednesday, May 27, 15 We didn’t put on the cowboy hat. Lots of reasons for what we ended up with. Not ideal (as in idealistic). How you promote and recognize people embodies your culture.

Slide 17

Slide 17 text

Process == What you do because it works at the time. Wednesday, May 27, 15 This talk: Not about which ones you’re using. It works for the people, it works for the business. It works now. You want to keep doing what works. It embodies your culture.

Slide 18

Slide 18 text

Processes embody culture Processes are repeatable (efficient, fair) What we want Wednesday, May 27, 15

Slide 19

Slide 19 text

Ye Olde Metaprocess Wednesday, May 27, 15 Meeting + Email + Wiki People get left out who can’t (remote) or shouldn’t (engineers) be in the meeting. Conversation happening for those privileged by role or location or start date.

Slide 20

Slide 20 text

Processes embody culture Processes are repeatable (efficient, fair) Transparent Inclusive What we want Wednesday, May 27, 15

Slide 21

Slide 21 text

150 Wednesday, May 27, 15 Dunbar’s Number, which taught us that we must write things down. We’re too big to convey all the nuances of how things have changed since you wrote it down, since the changes are only in your heads. Multiple offices in multiple timezones. Oral culture --> written culture. Slow-to-change written word is the medium.

Slide 22

Slide 22 text

Processes embody culture Processes are repeatable (efficient, fair) Transparent Inclusive Discoverable Single source of truth, written down What we want Wednesday, May 27, 15 We’re well beyond “oral tradition” scale. How will we write it down?

Slide 23

Slide 23 text

HOW ABOUT A WIKI? Wednesday, May 27, 15 This is Ward. - Backpack - Wikispaces - Confluence

Slide 24

Slide 24 text

The Append-only Wiki 2 / 5 Wednesday, May 27, 15 People just added on, they didn’t change or remove. There were comments. Big pile of crap hiding the gems. We had to do annual purges. Became untrustworthy. But we have Ward!

Slide 25

Slide 25 text

Copypasta feeds inertia Wednesday, May 27, 15 Not just the time spent rewriting that slows you down. People read stuff and believe it. You continue in wrong directions.

Slide 26

Slide 26 text

Processes embody culture Processes are repeatable (efficient, fair) Transparent Inclusive Discoverable Single source of truth, written down Update/delete easily What we want Wednesday, May 27, 15 “Easily” not just in UI, but organizationally Reinforced by our next story...

Slide 27

Slide 27 text

The Zombie Manual Tests 3 / 5 Wednesday, May 27, 15 We used to do manual tests after every production deploy. We had a wiki page on it, a list of accounts to test with. New engineers would read about the tests and take them really literally. We never found problems, but spend hundreds of person-hours doing it. Can’t afford that. We deleted the wiki pages.

Slide 28

Slide 28 text

Wednesday, May 27, 15 There was a good reason for the crap that doesn’t work anymore. Well-meaning! But confusing. We care about customer experience. We operate a SaaS, where production is the one true reality. (I know the appendix is actually useful... but this is funny.)

Slide 29

Slide 29 text

Wednesday, May 27, 15 Why delete them? We didn’t want our process to become untrustworthy. Misleading is worse than missing. Read well-meaning but wrong stuff too many times and it loses any credibility it might have had.

Slide 30

Slide 30 text

Culture > Process Wednesday, May 27, 15 If had left process the same, people would’ve continued to waste time on things that didn’t have value. We value spending time... valuably. With this process, that aspect of culture was being eroded. Culture should outlast Process.

Slide 31

Slide 31 text

Culture is the reason for Process. Wednesday, May 27, 15 This is the core problem: We care about our culture. We want it to live and thrive. That’s why we have processes.

Slide 32

Slide 32 text

Entropy Wednesday, May 27, 15 As soon as you write it, it’s out of date. If processes don’t change to close the gap with your culture, you lose to entropy.

Slide 33

Slide 33 text

You need a way to close the Culture/Process gap. Wednesday, May 27, 15 Metaprocess! If that dev had updated the manual test doc, we’d have wasted less time.

Slide 34

Slide 34 text

Processes embody culture Processes are repeatable (efficient, fair) Transparent Inclusive Discoverable Single source of truth, written down Update/delete easily Autonomy Bias for action What we want Wednesday, May 27, 15 “Easily” not just in UI, but organizationally

Slide 35

Slide 35 text

This document and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission. Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,” “expects” or words of similar import. Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings we make with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov. New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to the information provided. Copyright 2008-2015 New Relic, Inc. All rights reserved. Our business is not a democracy. 4 / 5 Wednesday, May 27, 15 Laws, boards of directors get to say no.

Slide 36

Slide 36 text

The Boss Wednesday, May 27, 15 Can’t have just anyone change processes, in particular those that affect money or private information.

Slide 37

Slide 37 text

Open Source is not a democracy. Wednesday, May 27, 15 We have a few committers, everyone’s a contributor. Everyone has a voice, but not a vote.

Slide 38

Slide 38 text

Processes embody culture Processes are repeatable (efficiency) Transparent Inclusive Discoverable Single source of truth, written down Update/delete easily Autonomy Bias for action Editorial control What we want Wednesday, May 27, 15

Slide 39

Slide 39 text

The idea 5 / 5 Wednesday, May 27, 15 Growing. 20 five years ago, to 500-something employees by the time we filed our S-1 last year. A company we admire, Simple, doing engineering in Portland. They wrote architecture docs in GitHub. Commit the docs first, then the software becomes real after review. Moved our operational docs into READMEs, some repos just about describing our systems.

Slide 40

Slide 40 text

Culture as Code Wednesday, May 27, 15 Culture as code. Same workflow as for code. That works for engineers. Lower mental barrier to contribution. Wiki was “a manager tool”.

Slide 41

Slide 41 text

Wednesday, May 27, 15 Going all in, ditched the wiki

Slide 42

Slide 42 text

GitHub Pages Jekyll Markdown + HTML Wednesday, May 27, 15 What did we get?

Slide 43

Slide 43 text

Processes embody culture Processes are repeatable (efficient, fair) Inclusive Transparent Discoverable Single source of truth, written down Update/delete easily Bias for action Autonomy Editorial control What we got Wednesday, May 27, 15 Not working perfectly for non-technical people UI could be easier What did we learn?

Slide 44

Slide 44 text

Do write together Wednesday, May 27, 15 Pair programming. Focused conversation of a few people generates a proposal that is worth sharing. Draft outside the PR.

Slide 45

Slide 45 text

Actively discourage email discussion Wednesday, May 27, 15 Structure your repos/orgs so people can use notifications. Close email threads with BCC and paste them into PRs.

Slide 46

Slide 46 text

Keep comments open after merge ~50% of discussion after merge Wednesday, May 27, 15 I typed into a comment field while someone was talking to me.

Slide 47

Slide 47 text

“Lowers the contribution barrier...” Wednesday, May 27, 15 Everyone has to understand git/GitHub to contribute. No Edit button. Wiki or Quip is better at this. Do anything you can to lower barrier. Train your non-technical staff. Train your technical staff.

Slide 48

Slide 48 text

git clone culture 6 / 5 Wednesday, May 27, 15 You can edit the whole world and propose something new. No need to get it right before you edit, or need to get rights. Freedom to think of the world as it should be.

Slide 49

Slide 49 text

Wednesday, May 27, 15 Titles again! Files changed: 4 Look at the why!

Slide 50

Slide 50 text

Why did we think this was a good idea? Wednesday, May 27, 15 What did you know at the time? What assumptions went unchallenged? What was important at the time?

Slide 51

Slide 51 text

Why > What Wednesday, May 27, 15 It’s hard to read the why in a document meant to efficiently describe a process.

Slide 52

Slide 52 text

Junior Software Engineer Software Engineer Senior Software Engineer Principal Software Engineer Distinguished Software Engineer Engineering Manager Senior Engineering Manager Director of Engineering Wednesday, May 27, 15

Slide 53

Slide 53 text

Software Engineer Senior Software Engineer Lead Software Engineer Principal Software Engineer Distinguished Software Engineer Engineering Manager Senior Engineering Manager Director of Engineering Wednesday, May 27, 15

Slide 54

Slide 54 text

Pull Request knows Why Wednesday, May 27, 15 It also knows who and when.

Slide 55

Slide 55 text

Ask why. Wednesday, May 27, 15 Everyone should be able to ask why. Our culture (autonomy, boldness, urgency, customer focus) demands it.

Slide 56

Slide 56 text

Thank you! @ralphbod Slides https://bit.ly/pull-request-your-culture Wednesday, May 27, 15 https://bit.ly/pull-request-your-culture

Slide 57

Slide 57 text

Acknowledgements This talk was the work of at least 18 people. @belindarunkle @danarelic @feministy @foliosus @kfrugia @kwugirl Matt Otis @mewzherder @mflaming @nerdygirl @nicbenders @qkate @relistan @spkane @SXavier_NwRelic @WardCunningham @xensesthegreat Thanks for your help in shaping this talk and in making New Relic awesome! Wednesday, May 27, 15

Slide 58

Slide 58 text

References • Valve Handbook for New Employees http://www.valvesoftware.com/company/ Valve_Handbook_LowRes.pdf • GitHub: Quick Pull Requests https://github.com/blog/1945-quick-pull-requests Wednesday, May 27, 15

Slide 59

Slide 59 text

Image References • Valve Working without a Boss http://www.blogcdn.com/www.joystiq.com/media/2012/04/screen-shot-2012-04-23-at-11.24.33-am.jpg • Valve cover http://i.kinja-img.com/gawker-media/image/upload/s--ANyPdiwb--/17k9nffv4tn36jpg.jpg • Constitutional Convention, from http://en.wikipedia.org/wiki/Convention_to_propose_amendments_to_the_United_States_Constitution (Public Domain) • Margaret Hamilton with her code http://upload.wikimedia.org/wikipedia/commons/2/2e/Margaret_Hamilton.gif (Public Domain) • Ward Cunningham http://upload.wikimedia.org/wikipedia/commons/3/30/Ward_Cunningham_1.jpg (Used with permission) • CA carcinogen warning sign http://images.mysafetysign.com/img/lg/S/chemical-cause-cancer-warning-sign-s-0289.png • Scumbag appendix http://imgur.com/gallery/254ANHP • Pair keypunching http://en.wikipedia.org/wiki/File:Keypunching_at_Texas_A%26M2.jpg Wednesday, May 27, 15