Evolving Open Source - Yehuda Katz

Slide 1

Slide 1 text

Or Learning to Live With It ERADICATING CRUFT

Slide 2

Slide 2 text

SOLVING HARD PROBLEMS I'm going to focus this talk on open source projects trying to solve ambitious problems. I've worked on a few.

Slide 3

Slide 3 text

A WILD REPLACEMENT CHARACTER APPEARS Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work http://bit.ly/harder-than-it-looks I've given a talk on the topic of "It's Harder than it Looks" a few times that might be a good companion for this talk.

Slide 4

Slide 4 text

DOM READINESS. Typically, "solving hard problems" means providing solutions that handle common edge-cases. Edge-cases doesn't mean cases only a few people hit. For me, the important edge- cases are parts of standard-looking solutions that aren't obvious.

Slide 5

Slide 5 text

SECURITY, PLUGIN ECOSYSTEM, ENCODINGS.

Slide 6

Slide 6 text

DATA BINDINGS, APPLICATION STRUCTURE.

Slide 7

Slide 7 text

USUALLY MANY, MANY MORE.

Slide 8

Slide 8 text

GOOD VS. GREAT SOLUTIONS. For me, what separates good from great solutions is how completely they tackle the problem spaces they claim to solve.

Slide 9

Slide 9 text

✓Routing ✓Data Bindings ✓Model Layer ✓View Objects ✓MVWTF FEATURE CHECKBOXES. A lot of times people don't notice the difference between partial and complete solutions until the solution starts to break down.

Slide 10

Slide 10 text

✓Routing ✓Data Bindings ✓Model Layer ✓View Objects ✓MVWTF FEATURE CHECKBOXES. Feature-checkbox-based choices undervalue the *completeness* of a solution. They're implicitly used in the marketing of less complete solutions.

Slide 11

Slide 11 text

✓Routing ✓Data Bindings ✓Model Layer ✓View Objects ✓MVWTF FEATURE CHECKBOXES. Solutions that handle edge-cases well do it because of a belief in the value of complete solutions, despite the fact that it's impossible to market.

Slide 12

Slide 12 text

DETOUR I want to take a brief detour to address the most serious objection to the thrust of my talk.

Slide 13

Slide 13 text

Write programs that do one thing and do it well. “ "THE UNIX PHILOSOPHY" This amounts to an objection even to modular solutions where each module is built by the same project.

Slide 14

Slide 14 text

"Write programs that do one thing and do it well" This simply moves the complexity to rapidly evolving *public* APIs.

Slide 15

Slide 15 text

"Write programs that do one thing and do it well" This simply moves the complexity to rapidly evolving *public* APIs.

Slide 16

Slide 16 text

edge-cases "Write programs that do one thing and do it well" This simply moves the complexity to rapidly evolving *public* APIs.

Slide 17

Slide 17 text

"Write programs that do one thing and do it well" In many cases, it pushes the integration tax onto the *user*.

Slide 18

Slide 18 text

"Write programs that do one thing and do it well" In many cases, it pushes the integration tax onto the *user*.

Slide 19

Slide 19 text

edge-cases "Write programs that do one thing and do it well" In many cases, it pushes the integration tax onto the *user*.

Slide 20

Slide 20 text

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a “ THE UNIX PHILOSOPHY

Slide 21

Slide 21 text

$ git add $(git status | awk 'p==1{print $2}; /git add /{p=1}') COMPLEXITY. The Unix philosophy works best when working with simple text formats. In this case, it would be better to have a solution in the git porcelain.

Slide 22

Slide 22 text

SUPPLY CHAINS

Slide 23

Slide 23 text

Raw Materials Design Parts Assembly Marketing Distribution & Sales

Slide 24

Slide 24 text

INTEGRATION GIVES WAY TO DISTRIBUTED SYSTEMS.

Slide 25

Slide 25 text

WHEN?

Slide 26

Slide 26 text

WHEN? STANDARDIZATION. RELIABILITY.

Slide 27

Slide 27 text

SOFTWARE

Slide 28

Slide 28 text

integration costs edge cases Components connect to each other very promiscuously. Each link introduces a transaction cost. Typically the burden of this cost is on the end-developer.

Slide 29

Slide 29 text

integration costs edge cases Components connect to each other very promiscuously. Each link introduces a transaction cost. Typically the burden of this cost is on the end-developer.

Slide 30

Slide 30 text

edge cases integration gets cheaper Components connect to each other very promiscuously. Each link introduces a transaction cost. Typically the burden of this cost is on the end-developer.

Slide 31

Slide 31 text

integration gets cheaper some of the time Components connect to each other very promiscuously. Each link introduces a transaction cost. Typically the burden of this cost is on the end-developer.

Slide 32

Slide 32 text

RACK.

Slide 33

Slide 33 text

mongrel rails sessions and cookies parameter parsing HTTP security Request and Response objects HTTP mongrel-rails coupling

Slide 34

Slide 34 text

WHY? For a possibly long time, the costs of integrating with third parties is high. You basically have to build the marketplace yourself. It's cheaper to vertically integrate.

Slide 35

Slide 35 text

STANDARDS THAT AREN'T REALLY STANDARDS. Early on, people pimp "de facto standards", because they're a good bludgeon against an integrated solution, even though they don't soften the integration tax much. (Rack for the first year or two)

Slide 36

Slide 36 text

STANDARDIZATION.

Slide 37

Slide 37 text

mongrel / unicorn / thin HTTP rails entry point standards session middleware HTTP security standards Controller Action standards standards

Slide 38

Slide 38 text

OTHER INTEGRATION COSTS. Even when interfaces are standardized, there are still integration costs. This is why Rails uses standard interfaces, but still uses its own components except for a few areas.

Slide 39

Slide 39 text

WHO PAYS THE INTEGRATION COSTS? (the answer is never 'nobody')

Slide 40

Slide 40 text

IN RAPIDLY EVOLVING SPACES, TIGHTER INTEGRATION IMPROVES ROBUSTNESS.

Slide 41

Slide 41 text

write who wc unlink tty rmdir rm pr od mv mkdir mesg mailx ls ln link find ed du df date cp cmp chown chmod cat uniq time split kill paste tr sleep ps nice file comm cal newgrp tsort touch sed awk at sh AT&T UNIX Unix itself developed many of its tools (and therefore interfaces) in a single place, reduces coordination and integration costs and increasing the speed of evolution.

Slide 42

Slide 42 text

WHAT DOES THIS MEAN FOR SOFTWARE?

Slide 43

Slide 43 text

CRUFT IS INEVITABLE.

Slide 44

Slide 44 text

THE REAL WORLD IS MESSY. (IE6-8, Ruby 1.8, Rails 2.3, etc.)

Slide 45

Slide 45 text

SOLUTIONS ARE OFTEN UGLY.

Slide 46

Slide 46 text

SOLUTIONS ARE OFTEN CRUFTY. The cruft just reflects the ugliness of the underlying problem.

Slide 47

Slide 47 text

CRITICISM YIELDS DEFENSIVENESS. Early anti-cruft criticism, because it targets cruft that is aligned with real problems, creates an internal defensiveness to criticisms of cruft.

Slide 48

Slide 48 text

THIS BECOMES RECEIVED WISDOM. "Removing cruft for the sake of it will probably break something. Don't touch anything unless you're adding features! We knew what we were doing!"

Slide 49

Slide 49 text

REMOVING CRUFT. reinforces received wisdom

Slide 50

Slide 50 text

REMOVING CRUFT. good case reinforces received wisdom

Slide 51

Slide 51 text

REMOVING CRUFT. good case remove cruft reinforces received wisdom

Slide 52

Slide 52 text

REMOVING CRUFT. good case remove cruft test fails reinforces received wisdom

Slide 53

Slide 53 text

REMOVING CRUFT. good case remove cruft test fails aha! revert reinforces received wisdom

Slide 54

Slide 54 text

REMOVING CRUFT. good case remove cruft test fails aha! revert worse case reinforces received wisdom

Slide 55

Slide 55 text

REMOVING CRUFT. good case remove cruft test fails aha! revert worse case reinforces received wisdom remove cruft

Slide 56

Slide 56 text

REMOVING CRUFT. good case remove cruft test fails aha! revert worse case reinforces received wisdom remove cruft ship release

Slide 57

Slide 57 text

REMOVING CRUFT. good case remove cruft test fails aha! revert worse case reinforces received wisdom remove cruft ship release break users

Slide 58

Slide 58 text

REMOVING CRUFT. good case remove cruft test fails aha! revert worse case reinforces received wisdom remove cruft ship release break users oh shit! revert

Slide 59

Slide 59 text

PROJECTS DEVELOP IMMUNE SYSTEMS. If someone wants to remove cruft for the sake of it, people reject the changes out of hand.

Slide 60

Slide 60 text

BUT!

Slide 61

Slide 61 text

ASSUMPTIONS CHANGE.

Slide 62

Slide 62 text

JQUERY. DOM Node data a memory leak in IE blocks storing data on DOM nodes.

Slide 63

Slide 63 text

JQUERY. data a memory leak in IE blocks storing data on DOM nodes.

Slide 64

Slide 64 text

JQUERY. data leak a memory leak in IE blocks storing data on DOM nodes.

Slide 65

Slide 65 text

JQUERY. DOM Node jQuery1345727403427: 573 Cache data 573 this clearing needs to happen recursively.

Slide 66

Slide 66 text

JQUERY. Cache data 573 this clearing needs to happen recursively.

Slide 67

Slide 67 text

JQUERY. Cache this clearing needs to happen recursively.

Slide 68

Slide 68 text

IF WE SUPPORT OLDIE, WE NEED THIS CODE.

Slide 69

Slide 69 text

COMPETITORS WHO DECRY "CRUFT" ARE MISSING THE POINT.

Slide 70

Slide 70 text

BUT WHAT HAPPENS WHEN OLDIE BECOMES LESS RELEVANT?

Slide 71

Slide 71 text

FORK IN THE ROAD. By keeping track of the reasons for our cruft (and analyzing after the fact), we could determine the value to the codebase of dropping oldIE support.

Slide 72

Slide 72 text

LIFECYCLE. cruft defensiveness cruft = solution a gap appears received wisdom consolidation splash of cold water the most destructive part of the cycle is once the assumptions lose validity, the fact that an alternate solution does not solve every edge-case invalidates the choices they are making that use ecosystem changes to remove cruft.

Slide 73

Slide 73 text

A BETTER LIFECYCLE.

Slide 74

Slide 74 text

TRACK YOUR CRUFT! (CRUFT.txt?)

Slide 75

Slide 75 text

RE-EVALUATE ASSUMPTIONS PERIODICALLY.

Slide 76

Slide 76 text

IT'S OK TO BE "THE DOM LIBRARY FOR IE6 USERS" Projects built five years ago are in a good position to provide solutions for edge-cases that were prominent five years ago.

Slide 77

Slide 77 text

NOT JUST ABOUT SUPPORT MATRIX. Today, many people are moving towards making the browser "just another client". Tools built around server-side HTML rendering face similar choices.

Slide 78

Slide 78 text

WHAT DO DO?

Slide 79

Slide 79 text

KEEP TRACK OF EDGE CASES.

Slide 80

Slide 80 text

AND THEIR IMPACT ON ARCHITECTURE.

Slide 81

Slide 81 text

PERIODICALLY REVIEW DECISIONS.

Slide 82

Slide 82 text

MAKE CHOICES MORE INTENTIONAL. Help contributors make decisions without fear that they are walking on a grave.