Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Open-source as a project model for internal work (w/ speaker notes)

Open-source as a project model for internal work (w/ speaker notes)

Even if you're working an entirely closed-source codebase, the chances are high that you're pulling in an open-source project. The popularity of jQuery, Node.js (and io.js), and many others speaks to how well open-source works, when done right.

But the truth is, these projects face the same troubles that many workplaces struggle with. How do they encourage contributions when most contributors only have a few minutes to spare? How do they manage to collaborate on a codebase when everyone lives in a different city and works on different teams? How do they get developers to RTFM?

While not everyone project can be open-source, all of them can benefit from taking an "open-source" approach. In this talk, we'll look at how to foster a codebase and a culture that's open and encouraging to everyone. We'll cover what works and what surprisingly doesn't, all in an effort to foster a more open, maintainable and friendly project.

Interested in more?
Read the series of blog posts at http://blog.kevinlamping.com/open-source-as-a-project-model-for-internal-work/
Sign up for new blog posts at http://open.klamp.in

Kevin Lamping

March 26, 2015
Tweet

More Decks by Kevin Lamping

Other Decks in Technology

Transcript

  1. Open-source as a project
    model for internal work
    opentalk.klamp.in
    Welcome!

    View Slide

  2. Front-end Developer. Mostly.

    View Slide

  3. Currently work at InVision

    View Slide

  4. This talk stems from the work I was part of at USAA and Rackspace, two very large companies.

    Not a representative of either company

    View Slide

  5. While at USAA, I’d have frequent conversations about infrastructure work with co-workers

    View Slide

  6. In our role at the time, we were tasked with developing and supporting an infrastructure framework for over 300 developers.
    His focus was on the server-side, mine on the front-end. Supporting a growing team is tough

    View Slide

  7. Supporting a growing team
    is tough
    After a certain number of users, you won't be able to keep up with the constant flow of "drive-by" questions and e-mails.
    You must balance time spent supporting your fellow teams with time spent developing new features.

    View Slide

  8. Teams and project will always be understaffed and there will always be more to do than people to do it

    View Slide

  9. Most projects are
    understaffed
    You can’t waste your users time (both consuming and contributing). Consumers of your project will likely be very busy with
    their own work and will have little time for contributing back.

    View Slide

  10. While we can easily support those around us

    View Slide

  11. Even co-located teams face physical constraints

    View Slide

  12. Even co-located teams face
    physical constraints
    You can’t be everywhere at once, or have everyone next to you at once. At a certain size, you're essentially working as a
    distributed team due to space constraints.

    View Slide

  13. While mulling on these issues, we became interested in how open-source projects run their show. jQuery has hundreds of
    thousands of users and contributors from all over the world who mostly work for free.

    View Slide

  14. Open-source projects
    have to be good at
    scaling
    How do they support their ever growing user base while staying adaptable? A projects success can also be its demise

    View Slide

  15. Open-source projects
    have to rely mostly on
    volunteers
    How do they get people to volunteer their time? OSS doesn’t have a big budget

    View Slide

  16. Open-source projects
    have to have distributed
    teams
    Most open-source projects don't have dedicated, co-located teams. How do they cope?

    View Slide

  17. Yet they manage.
    Quite well in fact.
    While there are a million OS projects that fail, many are outstanding successes, because they overcome the problems they
    face and are stronger for it.

    View Slide

  18. How to foster an
    Open-source culture
    at work?
    So can we foster a more open, maintainable and friendly project by taking an "open-source" approach at work?

    View Slide

  19. Reduce Friction to
    Contribute
    How do you get contributions from people who are likely already overworked?

    View Slide

  20. Practice empathetic
    code management
    How can you continue to evolve your product without causing endless headaches for those who have helped you get to
    where you are?

    View Slide

  21. Embrace the human
    community at work
    Just because it's work, doesn't mean it's not human. How do you build a community that’s supportive and encouraging and
    avoid the politics?

    View Slide

  22. Reduce the friction
    to contribute
    Everyone is busy. No one has time to waste. Open-source projects face this trouble every day. They don’t pay most of their
    contributors. They have to rely on their contributors taking time out of their day. Let’s do the same at work.

    View Slide

  23. "Better to have people
    think you're a fool than to
    open your mouth and
    remove all doubt."
    Maybe have been lincoln that said this. May have been Mark Twain

    View Slide

  24. I know it was said by wilson. w. wilson though on Home Improvement though.

    Idea is, assume the developers are uncomfortable/intimidated when trying to contribute code.

    View Slide

  25. The #1 friction is fear
    Fear of looking foolish

    Fear of breaking something

    Those who are fearless suffer from the Dunning–Kruger effect

    View Slide

  26. Use automation to
    prevent dumb mistakes

    View Slide

  27. It’s nicer to hear from a bot that you’re wrong than a human.

    As a human, you’re less strict than bots, which hurts in the long run.

    View Slide

  28. "pre-commit": [
    "grunt lint"
    ],
    "pre-push": [
    "grunt build",
    "grunt test"
    ]
    makes PRs much easier b/c you’re not having to be nitpick

    View Slide

  29. Automate testing
    Provides newcomers with a set of requirements to test their code changes against

    Allows much more confidence for newcomers that their changes haven't broken anything

    Unit, Midway, Visual

    Makes your job easier, as you can have a better trust that changes won’t break stuff.

    View Slide

  30. Let developers
    mess up
    Spend time trying to prevent it, but also spend time planning for recovery from mistakes. Expect failure. Plan for it.

    View Slide

  31. Write docs,
    but don't expect them
    to be read
    It’s tough to read all the documentation out there and not everyone has time.

    Docs should be more useful to the main dev team, as a quick way of pointing people in the right direction.

    Saves you the trouble of writing the same email explaining the same concept again and again. Be friendly with your RTFM.

    View Slide

  32. An example is worth
    1000 technical words

    View Slide

  33. $ ack .tag
    trying to find a class

    View Slide


  34. staging

    returned invalid results

    View Slide


  35. staging

    because it was treating it as a regex

    View Slide

  36. ack [options]
    PATTERN [FILE...]
    WHY ARE YOU YELLING AT ME? What is pattern? File…? What about searching folders?

    View Slide

  37. Options
    -a, --all
    Operate on all files, regardless of type (but still skip directories like blib, CVS , etc.)
    -A NUM , --after-context= NUM
    Print NUM lines of trailing context after matching lines.
    -B NUM , --before-context= NUM
    Print NUM lines of leading context before matching lines.
    -C [ NUM ], --context[= NUM ]
    Print NUM lines (default 2) of context around matching lines.
    -c, --count
    Suppress normal output; instead print a count of matching lines for each input file. If -l is in effect, it will only show the number of lines for each file that has lines matching. Without -l, some line counts may be zeroes.
    If combined with -h (--no-filename) ack outputs only one total count.
    --color, --nocolor
    --color highlights the matching text. --nocolor supresses the color. This is on by default unless the output is redirected.
    On Windows, this option is off by default unless the Win32::Console::ANSI module is installed or the "ACK_PAGER_COLOR" environment variable is used.
    --color-filename=color
    Sets the color to be used for filenames.
    --color-match=color
    Sets the color to be used for matches.
    --color-lineno=color
    Sets the color to be used for line numbers.
    --column
    Show the column number of the first match. This is helpful for editors that can place your cursor at a given position.
    --env, --noenv
    --noenv disables all environment processing. No .ackrc is read and all environment variables are ignored. By default, ack considers .ackrc and settings in the environment.
    --flush
    --flush flushes output immediately. This is off by default unless ack is running interactively (when output goes to a pipe or file).
    -f
    Only print the files that would be searched, without actually doing any searching. PATTERN must not be specified, or it will be taken as a path to search.
    --follow, --nofollow
    Follow or don't follow symlinks, other than whatever starting files or directories were specified on the command line.
    This is off by default.
    -G REGEX
    Only paths matching REGEX are included in the search. The entire path and filename are matched against REGEX , and REGEX is a Perl regular expression, not a shell glob.
    The options -i, -w, -v, and -Q do not apply to this REGEX .
    -g REGEX
    Print files where the relative path + filename matches REGEX . This option is a convenience shortcut for -f -G REGEX .
    The options -i, -w, -v, and -Q do not apply to this REGEX .
    --group, --nogroup
    --group groups matches by file name with. This is the default when used interactively.
    --nogroup prints one result per line, like grep. This is the default when output is redirected.
    -H, --with-filename
    Print the filename for each match.
    -h, --no-filename
    Suppress the prefixing of filenames on output when multiple files are searched.
    --help
    Print a short help statement.
    -i, --ignore-case
    Ignore case in the search strings.
    This applies only to the PATTERN , not to the regexes given for the -g and -G options.
    --[no]ignore-dir= DIRNAME
    Ignore directory (as CVS , .svn, etc are ignored). May be used multiple times to ignore multiple directories. For example, mason users may wish to include --ignore-dir=data. The --noignore-dir option allows users to search directories which would normally be ignored (perhaps to research the contents of .svn/props directories).
    The DIRNAME must always be a simple directory name. Nested directories like foo/bar are NOT supported. You would need to specify --ignore-dir=foo and then no files from any foo directory are taken into account by ack unless given explicitly on the command line.
    --line= NUM
    Only print line NUM of each file. Multiple lines can be given with multiple --line options or as a comma separated list (--line=3,5,7). --line=4-7 also works. The lines are always output in ascending order, no matter the order given on the command line.
    -l, --files-with-matches
    Only print the filenames of matching files, instead of the matching text.
    -L, --files-without-matches
    Only print the filenames of files that do NOT match. This is equivalent to specifying -l and -v.
    --match REGEX
    Specify the REGEX explicitly. This is helpful if you don't want to put the regex as your first argument, e.g. when executing multiple searches over the same set of files.
    # search for foo and bar in given files
    ack file1 t/file* --match foo
    ack file1 t/file* --match bar
    -m= NUM , --max-count= NUM
    Stop reading a file after NUM matches.
    --man
    Print this manual page.
    -n, --no-recurse
    No descending into subdirectories.
    -o
    Show only the part of each line matching PATTERN (turns off text highlighting)
    --output=expr
    Output the evaluation of expr for each line (turns off text highlighting)
    --pager=program
    Direct ack's output through program. This can also be specified via the "ACK_PAGER" and "ACK_PAGER_COLOR" environment variables.
    Using --pager does not suppress grouping and coloring like piping output on the command-line does.
    --passthru
    Prints all lines, whether or not they match the expression. Highlighting will still work, though, so it can be used to highlight matches while still seeing the entire file, as in:
    # Watch a log file, and highlight a certain IP address
    $ tail -f ~/access.log | ack --passthru 123.45.67.89
    --print0
    Only works in conjunction with -f, -g, -l or -c (filename output). The filenames are output separated with a null byte instead of the usual newline. This is helpful when dealing with filenames that contain whitespace, e.g.
    # remove all files of type html
    ack -f --html --print0 | xargs -0 rm -f
    -Q, --literal
    Quote all metacharacters in PATTERN , it is treated as a literal.
    This applies only to the PATTERN , not to the regexes given for the -g and -G options.
    -r, -R, --recurse
    Recurse into sub-directories. This is the default and just here for compatibility with grep. You can also use it for turning --no-recurse off.
    --smart-case, --no-smart-case
    Ignore case in the search strings if PATTERN contains no uppercase characters. This is similar to "smartcase" in vim. This option is off by default.
    -i always overrides this option.
    This applies only to the PATTERN , not to the regexes given for the -g and -G options.
    --sort-files
    Sorts the found files lexically. Use this if you want your file listings to be deterministic between runs of ack.
    --show-types
    Outputs the filetypes that ack associates with each file.
    Works with -f and -g options.
    --thpppt
    Display the all-important Bill The Cat logo. Note that the exact spelling of --thpppppt is not important. It's checked against a regular expression.
    --type=TYPE, --type=noTYPE
    Specify the types of files to include or exclude from a search. TYPE is a filetype, like perl or xml. --type=perl can also be specified as --perl, and --type=noperl can be done as --noperl.
    If a file is of both type "foo" and "bar", specifying --foo and --nobar will exclude the file, because an exclusion takes precedence over an inclusion.
    Type specifications can be repeated and are ORed together.
    See ack --help=types for a list of valid types.
    --type-add TYPE =.EXTENSION[,.EXT2[,...]]
    Files with the given EXTENSION (s) are recognized as being of (the existing) type TYPE . See also "Defining your own types".
    --type-set TYPE =.EXTENSION[,.EXT2[,...]]
    Files with the given EXTENSION (s) are recognized as being of type TYPE . This replaces an existing definition for type TYPE . See also "Defining your own types".
    -u, --unrestricted
    All files and directories (including blib/, core.*, ...) are searched, nothing is skipped. When both -u and --ignore-dir are used, the --ignore-dir option has no effect.
    -v, --invert-match
    Invert match: select non-matching lines
    This applies only to the PATTERN , not to the regexes given for the -g and -G options.
    --version
    Display version and copyright information.
    -w, --word-regexp
    Force PATTERN to match only whole words. The PATTERN is wrapped with "\b" metacharacters.
    This applies only to the PATTERN , not to the regexes given for the -g and -G options.
    -1
    Stops after reporting first match of any kind. This is different from --max-count=1 or -m1, where only one match per file is shown. Also, -1 works with -f and -g, where -m does not.
    Sure, I could have read through the man page and viewed all the options.

    View Slide

  38. $ ack simpleTerm
    $ ack “\.escape”
    $ ack -w onlywords
    show me examples instead. Lots and lots of examples

    View Slide

  39. Build a demo site.

    Twitter bootstrap, etc

    http://git-man-page-generator.lokaltog.net/

    View Slide

  40. Good variable naming &
    organization is a TL;DR
    for your code
    Focus on readable code over clever/small code

    View Slide

  41. Code
    comments
    https://twitter.com/nzkoz/status/538892801941848064
    Credit goes to Michael Koziarski (@nzkoz)

    Keep them up-front and center, so as to make it obvious when they're wrong.

    Don't hide docs in a wiki where people have to search to find them

    Apply DRY rules to docs as well, otherwise. JSDoc has this problem

    View Slide

  42. Know that many people
    want to help, but won't
    Time is limited, making a contribution to a big framework takes a lot of time that people with project deadlines don't have

    Don't fret too much about it. Be opening and encouraging, but not pushy

    Don't try gimmicks to get contributions. Contests only serve to undermine efforts

    -> Pumpkin contest, 4 entries out of ~200 people despite amazing prizes. People didn’t trust they could win, so they didn’t
    risk the effort.

    View Slide

  43. Empathetic Code
    Management

    View Slide

  44. better != together
    Contrary to what Jack Johnson sings, it’s not always better when we’re together

    may work when small, but won’t scale up.

    Longer release cycles regression testing everything

    Responsibility becomes vague. Who fixes the bugs?

    What if team A breaks team B’s code with update?

    View Slide

  45. App includes set of common features

    - site template,

    - re-usable components,

    - common styles

    "infrastructure code”

    Common to live together with the functional codebase

    View Slide

  46. View Slide

  47. View Slide

  48. View Slide

  49. View Slide

  50. Starks have been working 3 weeks on this new feature. A feature that requires some common code to behave a specific
    way. They’re a day away from releasing…

    View Slide

  51. Unfortunately, yesterday, lannisters released update w/modification to code that Team stark was expecting to work a
    specific way. Except it no longer does

    View Slide

  52. Now team in scramble mode

    They hadn’t told the lannisters about their work (why would they need to know)?

    View Slide

  53. View Slide

  54. Lannister already merged and shipped the code.

    View Slide

  55. Team is relying on those changes for the code that they just shipped. Sooo….

    View Slide

  56. At this point, people start bracing for a long weekend of refactoring code.

    View Slide

  57. The king is just “work the weekend m’kay” and we’ll do a “tiger team” to discuss this on 8am monday morning

    View Slide

  58. The Case for a Separate
    Infrastructure
    This could be avoided. Issue was two-fold

    No one responsible for infrastructure code and communication.

    infrastructure code being “pushed” out to teams

    View Slide

  59. Separate team
    Separate codebase
    Need separate infrastructure team, even if one person, to manage changes.

    Need a separate codebase to manage upgrades

    View Slide

  60. Most of the time, the common codebase gets pushed out to all the apps on update. You want to get features out there,
    right?

    View Slide

  61. View Slide

  62. But it’s a bad idea, and the open source world has figured that out. You will break prod with this model. Again and again and
    again.

    View Slide

  63. Pull your updates
    Allow the apps the room to "pull" in the code when their ready.

    This isn't an uncommon model. Almost all open-source projects out there follow it.

    View Slide

  64. Teams go to the common repo when they’re ready to upgrade.

    So app a can be upgraded separate from app b

    View Slide

  65. Then when app b has finished their dev cycle, they can upgrade on their own time.

    Package managers like NPM follow this model (unless you're using the latest flag, which again, you probably shouldn't be)

    View Slide

  66. Lower risk of breaking
    production with a release
    Teams able to fully regression test code, at their own pace (no weekends)

    Test changes with small pilot groups

    If something breaks due to the release, each team has option to delay upgrade until later

    Ability to rollback an update because infrastructure is versioned

    View Slide

  67. Faster iteration due to
    smaller testing efforts
    Able to release code quicker.

    Shortened release cycles, each version change will be more bite-sized, allowing teams an easier upgrade.

    Doesn't mean code isn't tested.

    infrastructure team should have regression tests they run.

    Also doesn't mean app teams on their own, or that the infrastructure team is free to make any change they want.

    Teams should be there for each other (empathy)

    View Slide

  68. Slower propagation
    of features
    Drawbacks:

    takes effort for a team to pull down code, compared to it automatically being pushed to them

    trade-off of the model and needs to be understood and accepted as part of the deal.

    Easy to blame pull model, therefore important to look deeper at the problem before abandonment

    View Slide

  69. Visual Inconsistencies
    hardest pill to swallow, effects app end-user

    If your infrastructure code includes visual design, splintering versions means having an inconsistent experience across apps
    which are supposed to look similar.

    View Slide

  70. Plan Ahead by Bundling
    Visual Changes
    If you know you have a series of visual updates that need consistency across the site, go through the effort to plan a
    combined release of these changes. Along with this, get in touch with the app teams and see if you can do a coordinated
    release for all of the apps. It does mean letting go of the benefits described above, but hopefully this isn't a common
    occurance.

    View Slide

  71. Passively communicate
    Give information for passively, to avoid spamming inboxes and to save time

    View Slide

  72. https://twitter.com/sosowski/status/517005067334205441
    Many ways to do versioning, including the microsoft way. Other’s include dates. But what if we can send information for free
    with every update?

    View Slide

  73. Semantic Versioning

    View Slide

  74. Major.Minor.Patch
    MAJOR version when you make incompatible API changes,

    MINOR version when you add functionality in a backwards-compatible manner, and

    PATCH version when you make backwards-compatible bug fixes.

    View Slide

  75. 0.10.35 -> 0.10.36
    Trust, but verify

    View Slide

  76. 0.10.36 -> 0.12.0
    Expect a little bit of work

    View Slide

  77. http://semver.org/
    In-depth thoughts on how to manage versioning

    View Slide

  78. Node Stability Index
    Allows you to introduce new components with a disclaimer. Add index with component documentation.

    View Slide

  79. 0 - Deprecated
    1 - Experimental
    2 - Unstable
    3 - Stable
    4 - API Frozen
    5 - Locked
    Can have an experimental component live next to a stable one

    Allows more flexibility

    View Slide

  80. Update them regularly
    Update regularly, otherwise it’s meaningless and not worth it.

    Mgmt might also not like it b/c it reveals a certain amount of hesitation around code :/

    View Slide

  81. keepachangelog.com
    Change log is a file which contains a curated, chronologically ordered list of notable changes for each version of an open
    source project.

    Make it easier for users and contributors to see precisely what notable changes have been made between each release (or
    version) of the project.

    Highlight breaking changes

    Auto-generate changelogs from commit messages.

    Side benefit is encouraging useful commit messages.

    Angular.js has an interesting approach to formatting their messages. They follow this model:

    View Slide

  82. typeOfChange(component):
    Description of change
    Angular follows as commit message format.

    Where changeType can be ‘feat’, ’style’, ‘tests’, 'fix',

    View Slide

  83. feat(ruler): add inches
    as well as centimeters

    View Slide

  84. fix(pen): use blue ink instead of red ink
    BREAKING CHANGE: Pen now uses blue ink instead
    of red.
    To migrate, change your code from the
    following:
    `pen.draw('blue')`
    To:
    `pen.draw('red')`
    Can include breaking change details

    View Slide

  85. Generates a markdown file like this

    View Slide

  86. lmgtfy.com/?q=
    changelog+generator
    github.com/skywinder/github-changelog-generator

    github.com/ajoslin/conventional-changelog

    View Slide

  87. Embrace the human
    community at work

    View Slide

  88. Don't get good at
    answering e-mails
    & attending meetings
    These behaviors build silos by keeping communication hidden. Unless publicly recorded, meetings and emails are “closed-
    source”. Meetings & emails don’t scale.

    View Slide

  89. Silos encourage politics &
    discourage outside voices
    How is someone supposed to help when they don’t know the details? Communicate in the open, via github, chat rooms,
    message boards

    View Slide

  90. Have an open, public
    chat room
    Slack/Hipchat/gitter.im

    Chat history that’s searchable

    Easier for quick discussion. Team can help out, instead of an individual. People can lurk.

    View Slide

  91. View Slide

  92. Publicize project status
    Work as if you’ll never meet your users face-to-face in real life

    View Slide

  93. workingon.co, waffle.io, trello.com

    View Slide

  94. Keep an ever-growing
    backlog
    Don't fear an endless backlog. It's okay to have too much work to do. It’s better to track the work than forget it.

    View Slide

  95. Fit in time to write
    Blog posts. Weekly ‘write ups’. A-ha’s. Keep a history of why decisions were made, you'll forget why a month later otherwise
    when someone asks.

    View Slide

  96. View Slide

  97. Set timelines for Code
    Reviews
    People are impatient and that’s okay. If they take the time to reach out to you, set expectations.

    If you know you don't have time to review a PR right away, make a quick comment saying so. Everyone understands people
    get busy

    Going 24 hours before reviewing code is not unreasonable to ask for small code reviews.

    Bigger PRs require more time, and likely more cycles of feedback. Make sure both sides understand this.

    Same for emails

    View Slide

  98. Reduce variability,
    yet embrace it
    Reducing is good for things like code syntax/style, release cycles, code standards.

    Embracing is good for things like experimental work, new libraries/techniques/languages

    View Slide

  99. Define community
    guidelines
    sass-lang.com/community-guidelines
    The best OSS projects are driven by their community. Go to a conference for Drupal or Angular or React or Sass and people
    will be most excited by the community around them.

    Foster that with guidelines that clearly define acceptable behavior.

    View Slide

  100. You want thinkers,
    not doers
    contributors are volunteers when it comes to their thoughts. You can't force good behavior. You have to foster it.

    View Slide

  101. 1) Reduce the friction to
    contribute
    2) Practice empathetic
    code management
    3) Embrace the Human
    Community at Work
    The problems that OSS projects face mirror the same problems workplaces face and revolve more around human
    interaction than the code itself. By following the lead of what the open-source world has done, you can ensure a more open,
    maintainable and friendly project, even if the work you do is closed-source.

    View Slide

  102. Thanks!
    open.klamp.in
    @klamping @gmail. com

    View Slide

  103. Thanks!
    open.klamp.in
    @klamping @gmail. com

    View Slide