Open-source as a project model for internal work (w/ speaker notes)

Open-source as a project model for internal work (w/ speaker notes)

Even if you're working an entirely closed-source codebase, the chances are high that you're pulling in an open-source project. The popularity of jQuery, Node.js (and io.js), and many others speaks to how well open-source works, when done right.

But the truth is, these projects face the same troubles that many workplaces struggle with. How do they encourage contributions when most contributors only have a few minutes to spare? How do they manage to collaborate on a codebase when everyone lives in a different city and works on different teams? How do they get developers to RTFM?

While not everyone project can be open-source, all of them can benefit from taking an "open-source" approach. In this talk, we'll look at how to foster a codebase and a culture that's open and encouraging to everyone. We'll cover what works and what surprisingly doesn't, all in an effort to foster a more open, maintainable and friendly project.

Interested in more?
Read the series of blog posts at http://blog.kevinlamping.com/open-source-as-a-project-model-for-internal-work/
Sign up for new blog posts at http://open.klamp.in

6823d6d1ee14bd068007bd60ddb6a41a?s=128

Kevin Lamping

March 26, 2015
Tweet

Transcript

  1. 4.

    This talk stems from the work I was part of

    at USAA and Rackspace, two very large companies. Not a representative of either company
  2. 6.

    In our role at the time, we were tasked with

    developing and supporting an infrastructure framework for over 300 developers. His focus was on the server-side, mine on the front-end. Supporting a growing team is tough
  3. 7.

    Supporting a growing team is tough After a certain number

    of users, you won't be able to keep up with the constant flow of "drive-by" questions and e-mails. You must balance time spent supporting your fellow teams with time spent developing new features.
  4. 8.

    Teams and project will always be understaffed and there will

    always be more to do than people to do it
  5. 9.

    Most projects are understaffed You can’t waste your users time

    (both consuming and contributing). Consumers of your project will likely be very busy with their own work and will have little time for contributing back.
  6. 12.

    Even co-located teams face physical constraints You can’t be everywhere

    at once, or have everyone next to you at once. At a certain size, you're essentially working as a distributed team due to space constraints.
  7. 13.

    While mulling on these issues, we became interested in how

    open-source projects run their show. jQuery has hundreds of thousands of users and contributors from all over the world who mostly work for free.
  8. 14.

    Open-source projects have to be good at scaling How do

    they support their ever growing user base while staying adaptable? A projects success can also be its demise
  9. 15.

    Open-source projects have to rely mostly on volunteers How do

    they get people to volunteer their time? OSS doesn’t have a big budget
  10. 16.

    Open-source projects have to have distributed teams Most open-source projects

    don't have dedicated, co-located teams. How do they cope?
  11. 17.

    Yet they manage. Quite well in fact. While there are

    a million OS projects that fail, many are outstanding successes, because they overcome the problems they face and are stronger for it.
  12. 18.

    How to foster an Open-source culture at work? So can

    we foster a more open, maintainable and friendly project by taking an "open-source" approach at work?
  13. 19.

    Reduce Friction to Contribute How do you get contributions from

    people who are likely already overworked?
  14. 20.

    Practice empathetic code management How can you continue to evolve

    your product without causing endless headaches for those who have helped you get to where you are?
  15. 21.

    Embrace the human community at work Just because it's work,

    doesn't mean it's not human. How do you build a community that’s supportive and encouraging and avoid the politics?
  16. 22.

    Reduce the friction to contribute Everyone is busy. No one

    has time to waste. Open-source projects face this trouble every day. They don’t pay most of their contributors. They have to rely on their contributors taking time out of their day. Let’s do the same at work.
  17. 23.

    "Better to have people think you're a fool than to

    open your mouth and remove all doubt." Maybe have been lincoln that said this. May have been Mark Twain
  18. 24.

    I know it was said by wilson. w. wilson though

    on Home Improvement though. Idea is, assume the developers are uncomfortable/intimidated when trying to contribute code.
  19. 25.

    The #1 friction is fear Fear of looking foolish Fear

    of breaking something Those who are fearless suffer from the Dunning–Kruger effect
  20. 27.

    It’s nicer to hear from a bot that you’re wrong

    than a human. As a human, you’re less strict than bots, which hurts in the long run.
  21. 28.

    "pre-commit": [ "grunt lint" ], "pre-push": [ "grunt build", "grunt

    test" ] makes PRs much easier b/c you’re not having to be nitpick
  22. 29.

    Automate testing Provides newcomers with a set of requirements to

    test their code changes against Allows much more confidence for newcomers that their changes haven't broken anything Unit, Midway, Visual Makes your job easier, as you can have a better trust that changes won’t break stuff.
  23. 30.

    Let developers mess up Spend time trying to prevent it,

    but also spend time planning for recovery from mistakes. Expect failure. Plan for it.
  24. 31.

    Write docs, but don't expect them to be read It’s

    tough to read all the documentation out there and not everyone has time. Docs should be more useful to the main dev team, as a quick way of pointing people in the right direction. Saves you the trouble of writing the same email explaining the same concept again and again. Be friendly with your RTFM.
  25. 36.

    ack [options] PATTERN [FILE...] WHY ARE YOU YELLING AT ME?

    What is pattern? File…? What about searching folders?
  26. 37.

    Options -a, --all Operate on all files, regardless of type

    (but still skip directories like blib, CVS , etc.) -A NUM , --after-context= NUM Print NUM lines of trailing context after matching lines. -B NUM , --before-context= NUM Print NUM lines of leading context before matching lines. -C [ NUM ], --context[= NUM ] Print NUM lines (default 2) of context around matching lines. -c, --count Suppress normal output; instead print a count of matching lines for each input file. If -l is in effect, it will only show the number of lines for each file that has lines matching. Without -l, some line counts may be zeroes. If combined with -h (--no-filename) ack outputs only one total count. --color, --nocolor --color highlights the matching text. --nocolor supresses the color. This is on by default unless the output is redirected. On Windows, this option is off by default unless the Win32::Console::ANSI module is installed or the "ACK_PAGER_COLOR" environment variable is used. --color-filename=color Sets the color to be used for filenames. --color-match=color Sets the color to be used for matches. --color-lineno=color Sets the color to be used for line numbers. --column Show the column number of the first match. This is helpful for editors that can place your cursor at a given position. --env, --noenv --noenv disables all environment processing. No .ackrc is read and all environment variables are ignored. By default, ack considers .ackrc and settings in the environment. --flush --flush flushes output immediately. This is off by default unless ack is running interactively (when output goes to a pipe or file). -f Only print the files that would be searched, without actually doing any searching. PATTERN must not be specified, or it will be taken as a path to search. --follow, --nofollow Follow or don't follow symlinks, other than whatever starting files or directories were specified on the command line. This is off by default. -G REGEX Only paths matching REGEX are included in the search. The entire path and filename are matched against REGEX , and REGEX is a Perl regular expression, not a shell glob. The options -i, -w, -v, and -Q do not apply to this REGEX . -g REGEX Print files where the relative path + filename matches REGEX . This option is a convenience shortcut for -f -G REGEX . The options -i, -w, -v, and -Q do not apply to this REGEX . --group, --nogroup --group groups matches by file name with. This is the default when used interactively. --nogroup prints one result per line, like grep. This is the default when output is redirected. -H, --with-filename Print the filename for each match. -h, --no-filename Suppress the prefixing of filenames on output when multiple files are searched. --help Print a short help statement. -i, --ignore-case Ignore case in the search strings. This applies only to the PATTERN , not to the regexes given for the -g and -G options. --[no]ignore-dir= DIRNAME Ignore directory (as CVS , .svn, etc are ignored). May be used multiple times to ignore multiple directories. For example, mason users may wish to include --ignore-dir=data. The --noignore-dir option allows users to search directories which would normally be ignored (perhaps to research the contents of .svn/props directories). The DIRNAME must always be a simple directory name. Nested directories like foo/bar are NOT supported. You would need to specify --ignore-dir=foo and then no files from any foo directory are taken into account by ack unless given explicitly on the command line. --line= NUM Only print line NUM of each file. Multiple lines can be given with multiple --line options or as a comma separated list (--line=3,5,7). --line=4-7 also works. The lines are always output in ascending order, no matter the order given on the command line. -l, --files-with-matches Only print the filenames of matching files, instead of the matching text. -L, --files-without-matches Only print the filenames of files that do NOT match. This is equivalent to specifying -l and -v. --match REGEX Specify the REGEX explicitly. This is helpful if you don't want to put the regex as your first argument, e.g. when executing multiple searches over the same set of files. # search for foo and bar in given files ack file1 t/file* --match foo ack file1 t/file* --match bar -m= NUM , --max-count= NUM Stop reading a file after NUM matches. --man Print this manual page. -n, --no-recurse No descending into subdirectories. -o Show only the part of each line matching PATTERN (turns off text highlighting) --output=expr Output the evaluation of expr for each line (turns off text highlighting) --pager=program Direct ack's output through program. This can also be specified via the "ACK_PAGER" and "ACK_PAGER_COLOR" environment variables. Using --pager does not suppress grouping and coloring like piping output on the command-line does. --passthru Prints all lines, whether or not they match the expression. Highlighting will still work, though, so it can be used to highlight matches while still seeing the entire file, as in: # Watch a log file, and highlight a certain IP address $ tail -f ~/access.log | ack --passthru 123.45.67.89 --print0 Only works in conjunction with -f, -g, -l or -c (filename output). The filenames are output separated with a null byte instead of the usual newline. This is helpful when dealing with filenames that contain whitespace, e.g. # remove all files of type html ack -f --html --print0 | xargs -0 rm -f -Q, --literal Quote all metacharacters in PATTERN , it is treated as a literal. This applies only to the PATTERN , not to the regexes given for the -g and -G options. -r, -R, --recurse Recurse into sub-directories. This is the default and just here for compatibility with grep. You can also use it for turning --no-recurse off. --smart-case, --no-smart-case Ignore case in the search strings if PATTERN contains no uppercase characters. This is similar to "smartcase" in vim. This option is off by default. -i always overrides this option. This applies only to the PATTERN , not to the regexes given for the -g and -G options. --sort-files Sorts the found files lexically. Use this if you want your file listings to be deterministic between runs of ack. --show-types Outputs the filetypes that ack associates with each file. Works with -f and -g options. --thpppt Display the all-important Bill The Cat logo. Note that the exact spelling of --thpppppt is not important. It's checked against a regular expression. --type=TYPE, --type=noTYPE Specify the types of files to include or exclude from a search. TYPE is a filetype, like perl or xml. --type=perl can also be specified as --perl, and --type=noperl can be done as --noperl. If a file is of both type "foo" and "bar", specifying --foo and --nobar will exclude the file, because an exclusion takes precedence over an inclusion. Type specifications can be repeated and are ORed together. See ack --help=types for a list of valid types. --type-add TYPE =.EXTENSION[,.EXT2[,...]] Files with the given EXTENSION (s) are recognized as being of (the existing) type TYPE . See also "Defining your own types". --type-set TYPE =.EXTENSION[,.EXT2[,...]] Files with the given EXTENSION (s) are recognized as being of type TYPE . This replaces an existing definition for type TYPE . See also "Defining your own types". -u, --unrestricted All files and directories (including blib/, core.*, ...) are searched, nothing is skipped. When both -u and --ignore-dir are used, the --ignore-dir option has no effect. -v, --invert-match Invert match: select non-matching lines This applies only to the PATTERN , not to the regexes given for the -g and -G options. --version Display version and copyright information. -w, --word-regexp Force PATTERN to match only whole words. The PATTERN is wrapped with "\b" metacharacters. This applies only to the PATTERN , not to the regexes given for the -g and -G options. -1 Stops after reporting first match of any kind. This is different from --max-count=1 or -m1, where only one match per file is shown. Also, -1 works with -f and -g, where -m does not. Sure, I could have read through the man page and viewed all the options.
  27. 38.

    $ ack simpleTerm $ ack “\.escape” $ ack -w onlywords

    show me examples instead. Lots and lots of examples
  28. 40.

    Good variable naming & organization is a TL;DR for your

    code Focus on readable code over clever/small code
  29. 41.

    Code comments https://twitter.com/nzkoz/status/538892801941848064 Credit goes to Michael Koziarski (@nzkoz) Keep

    them up-front and center, so as to make it obvious when they're wrong. Don't hide docs in a wiki where people have to search to find them Apply DRY rules to docs as well, otherwise. JSDoc has this problem
  30. 42.

    Know that many people want to help, but won't Time

    is limited, making a contribution to a big framework takes a lot of time that people with project deadlines don't have Don't fret too much about it. Be opening and encouraging, but not pushy Don't try gimmicks to get contributions. Contests only serve to undermine efforts -> Pumpkin contest, 4 entries out of ~200 people despite amazing prizes. People didn’t trust they could win, so they didn’t risk the effort.
  31. 44.

    better != together Contrary to what Jack Johnson sings, it’s

    not always better when we’re together may work when small, but won’t scale up. Longer release cycles regression testing everything Responsibility becomes vague. Who fixes the bugs? What if team A breaks team B’s code with update?
  32. 45.

    App includes set of common features - site template, -

    re-usable components, - common styles "infrastructure code” Common to live together with the functional codebase
  33. 46.
  34. 47.
  35. 48.
  36. 49.
  37. 50.

    Starks have been working 3 weeks on this new feature.

    A feature that requires some common code to behave a specific way. They’re a day away from releasing…
  38. 51.

    Unfortunately, yesterday, lannisters released update w/modification to code that Team

    stark was expecting to work a specific way. Except it no longer does
  39. 52.

    Now team in scramble mode They hadn’t told the lannisters

    about their work (why would they need to know)?
  40. 53.
  41. 55.
  42. 57.

    The king is just “work the weekend m’kay” and we’ll

    do a “tiger team” to discuss this on 8am monday morning
  43. 58.

    The Case for a Separate Infrastructure This could be avoided.

    Issue was two-fold No one responsible for infrastructure code and communication. infrastructure code being “pushed” out to teams
  44. 59.

    Separate team Separate codebase Need separate infrastructure team, even if

    one person, to manage changes. Need a separate codebase to manage upgrades
  45. 60.

    Most of the time, the common codebase gets pushed out

    to all the apps on update. You want to get features out there, right?
  46. 61.
  47. 62.

    But it’s a bad idea, and the open source world

    has figured that out. You will break prod with this model. Again and again and again.
  48. 63.

    Pull your updates Allow the apps the room to "pull"

    in the code when their ready. This isn't an uncommon model. Almost all open-source projects out there follow it.
  49. 64.

    Teams go to the common repo when they’re ready to

    upgrade. So app a can be upgraded separate from app b
  50. 65.

    Then when app b has finished their dev cycle, they

    can upgrade on their own time. Package managers like NPM follow this model (unless you're using the latest flag, which again, you probably shouldn't be)
  51. 66.

    Lower risk of breaking production with a release Teams able

    to fully regression test code, at their own pace (no weekends) Test changes with small pilot groups If something breaks due to the release, each team has option to delay upgrade until later Ability to rollback an update because infrastructure is versioned
  52. 67.

    Faster iteration due to smaller testing efforts Able to release

    code quicker. Shortened release cycles, each version change will be more bite-sized, allowing teams an easier upgrade. Doesn't mean code isn't tested. infrastructure team should have regression tests they run. Also doesn't mean app teams on their own, or that the infrastructure team is free to make any change they want. Teams should be there for each other (empathy)
  53. 68.

    Slower propagation of features Drawbacks: takes effort for a team

    to pull down code, compared to it automatically being pushed to them trade-off of the model and needs to be understood and accepted as part of the deal. Easy to blame pull model, therefore important to look deeper at the problem before abandonment
  54. 69.

    Visual Inconsistencies hardest pill to swallow, effects app end-user If

    your infrastructure code includes visual design, splintering versions means having an inconsistent experience across apps which are supposed to look similar.
  55. 70.

    Plan Ahead by Bundling Visual Changes If you know you

    have a series of visual updates that need consistency across the site, go through the effort to plan a combined release of these changes. Along with this, get in touch with the app teams and see if you can do a coordinated release for all of the apps. It does mean letting go of the benefits described above, but hopefully this isn't a common occurance.
  56. 72.

    https://twitter.com/sosowski/status/517005067334205441 Many ways to do versioning, including the microsoft way.

    Other’s include dates. But what if we can send information for free with every update?
  57. 74.

    Major.Minor.Patch MAJOR version when you make incompatible API changes, MINOR

    version when you add functionality in a backwards-compatible manner, and PATCH version when you make backwards-compatible bug fixes.
  58. 78.

    Node Stability Index Allows you to introduce new components with

    a disclaimer. Add index with component documentation.
  59. 79.

    0 - Deprecated 1 - Experimental 2 - Unstable 3

    - Stable 4 - API Frozen 5 - Locked Can have an experimental component live next to a stable one Allows more flexibility
  60. 80.

    Update them regularly Update regularly, otherwise it’s meaningless and not

    worth it. Mgmt might also not like it b/c it reveals a certain amount of hesitation around code :/
  61. 81.

    keepachangelog.com Change log is a file which contains a curated,

    chronologically ordered list of notable changes for each version of an open source project. Make it easier for users and contributors to see precisely what notable changes have been made between each release (or version) of the project. Highlight breaking changes Auto-generate changelogs from commit messages. Side benefit is encouraging useful commit messages. Angular.js has an interesting approach to formatting their messages. They follow this model:
  62. 82.

    typeOfChange(component): Description of change Angular follows as commit message format.

    Where changeType can be ‘feat’, ’style’, ‘tests’, 'fix',
  63. 84.

    fix(pen): use blue ink instead of red ink BREAKING CHANGE:

    Pen now uses blue ink instead of red. To migrate, change your code from the following: `pen.draw('blue')` To: `pen.draw('red')` Can include breaking change details
  64. 88.

    Don't get good at answering e-mails & attending meetings These

    behaviors build silos by keeping communication hidden. Unless publicly recorded, meetings and emails are “closed- source”. Meetings & emails don’t scale.
  65. 89.

    Silos encourage politics & discourage outside voices How is someone

    supposed to help when they don’t know the details? Communicate in the open, via github, chat rooms, message boards
  66. 90.

    Have an open, public chat room Slack/Hipchat/gitter.im Chat history that’s

    searchable Easier for quick discussion. Team can help out, instead of an individual. People can lurk.
  67. 91.
  68. 94.

    Keep an ever-growing backlog Don't fear an endless backlog. It's

    okay to have too much work to do. It’s better to track the work than forget it.
  69. 95.

    Fit in time to write Blog posts. Weekly ‘write ups’.

    A-ha’s. Keep a history of why decisions were made, you'll forget why a month later otherwise when someone asks.
  70. 96.
  71. 97.

    Set timelines for Code Reviews People are impatient and that’s

    okay. If they take the time to reach out to you, set expectations. If you know you don't have time to review a PR right away, make a quick comment saying so. Everyone understands people get busy Going 24 hours before reviewing code is not unreasonable to ask for small code reviews. Bigger PRs require more time, and likely more cycles of feedback. Make sure both sides understand this. Same for emails
  72. 98.

    Reduce variability, yet embrace it Reducing is good for things

    like code syntax/style, release cycles, code standards. Embracing is good for things like experimental work, new libraries/techniques/languages
  73. 99.

    Define community guidelines sass-lang.com/community-guidelines The best OSS projects are driven

    by their community. Go to a conference for Drupal or Angular or React or Sass and people will be most excited by the community around them. Foster that with guidelines that clearly define acceptable behavior.
  74. 100.

    You want thinkers, not doers contributors are volunteers when it

    comes to their thoughts. You can't force good behavior. You have to foster it.
  75. 101.

    1) Reduce the friction to contribute 2) Practice empathetic code

    management 3) Embrace the Human Community at Work The problems that OSS projects face mirror the same problems workplaces face and revolve more around human interaction than the code itself. By following the lead of what the open-source world has done, you can ensure a more open, maintainable and friendly project, even if the work you do is closed-source.