Evolving Open Source - Yehuda Katz

Transcript

1. Or Learning to Live With It ERADICATING CRUFT
2. SOLVING HARD PROBLEMS I'm going to focus this talk on open source projects trying to solve ambitious problems. I've worked on a few.
3. A WILD REPLACEMENT CHARACTER APPEARS Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work http://bit.ly/harder-than-it-looks I've given a talk on the topic of "It's Harder than it Looks" a few times that might be a good companion for this talk.
4. DOM READINESS. Typically, "solving hard problems" means providing solutions that handle common edge-cases. Edge-cases doesn't mean cases only a few people hit. For me, the important edge- cases are parts of standard-looking solutions that aren't obvious.
5. SECURITY, PLUGIN ECOSYSTEM, ENCODINGS.
6. DATA BINDINGS, APPLICATION STRUCTURE.
7. USUALLY MANY, MANY MORE.
8. GOOD VS. GREAT SOLUTIONS. For me, what separates good from great solutions is how completely they tackle the problem spaces they claim to solve.
9. ✓Routing ✓Data Bindings ✓Model Layer ✓View Objects ✓MVWTF FEATURE CHECKBOXES. A lot of times people don't notice the difference between partial and complete solutions until the solution starts to break down.
10. ✓Routing ✓Data Bindings ✓Model Layer ✓View Objects ✓MVWTF FEATURE CHECKBOXES. Feature-checkbox-based choices undervalue the *completeness* of a solution. They're implicitly used in the marketing of less complete solutions.
11. ✓Routing ✓Data Bindings ✓Model Layer ✓View Objects ✓MVWTF FEATURE CHECKBOXES. Solutions that handle edge-cases well do it because of a belief in the value of complete solutions, despite the fact that it's impossible to market.
12. DETOUR I want to take a brief detour to address the most serious objection to the thrust of my talk.
13. Write programs that do one thing and do it well. “ "THE UNIX PHILOSOPHY" This amounts to an objection even to modular solutions where each module is built by the same project.
14. "Write programs that do one thing and do it well" This simply moves the complexity to rapidly evolving *public* APIs.
15. "Write programs that do one thing and do it well" This simply moves the complexity to rapidly evolving *public* APIs.
16. edge-cases "Write programs that do one thing and do it well" This simply moves the complexity to rapidly evolving *public* APIs.
17. "Write programs that do one thing and do it well" In many cases, it pushes the integration tax onto the *user*.
18. "Write programs that do one thing and do it well" In many cases, it pushes the integration tax onto the *user*.
19. edge-cases "Write programs that do one thing and do it well" In many cases, it pushes the integration tax onto the *user*.
20. This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a “ THE UNIX PHILOSOPHY
21. $ git add $(git status | awk 'p==1{print $2}; /git add /{p=1}') COMPLEXITY. The Unix philosophy works best when working with simple text formats. In this case, it would be better to have a solution in the git porcelain.
22. SUPPLY CHAINS
23. Raw Materials Design Parts Assembly Marketing Distribution & Sales
24. INTEGRATION GIVES WAY TO DISTRIBUTED SYSTEMS.
25. WHEN?
26. WHEN? STANDARDIZATION. RELIABILITY.
27. SOFTWARE
28. integration costs edge cases Components connect to each other very promiscuously. Each link introduces a transaction cost. Typically the burden of this cost is on the end-developer.
29. integration costs edge cases Components connect to each other very promiscuously. Each link introduces a transaction cost. Typically the burden of this cost is on the end-developer.
30. edge cases integration gets cheaper Components connect to each other very promiscuously. Each link introduces a transaction cost. Typically the burden of this cost is on the end-developer.
31. integration gets cheaper some of the time Components connect to each other very promiscuously. Each link introduces a transaction cost. Typically the burden of this cost is on the end-developer.
32. RACK.
33. mongrel rails sessions and cookies parameter parsing HTTP security Request and Response objects HTTP mongrel-rails coupling
34. WHY? For a possibly long time, the costs of integrating with third parties is high. You basically have to build the marketplace yourself. It's cheaper to vertically integrate.
35. STANDARDS THAT AREN'T REALLY STANDARDS. Early on, people pimp "de facto standards", because they're a good bludgeon against an integrated solution, even though they don't soften the integration tax much. (Rack for the first year or two)
36. STANDARDIZATION.
37. mongrel / unicorn / thin HTTP rails entry point standards session middleware HTTP security standards Controller Action standards standards
38. OTHER INTEGRATION COSTS. Even when interfaces are standardized, there are still integration costs. This is why Rails uses standard interfaces, but still uses its own components except for a few areas.
39. WHO PAYS THE INTEGRATION COSTS? (the answer is never 'nobody')
40. IN RAPIDLY EVOLVING SPACES, TIGHTER INTEGRATION IMPROVES ROBUSTNESS.
41. write who wc unlink tty rmdir rm pr od mv mkdir mesg mailx ls ln link find ed du df date cp cmp chown chmod cat uniq time split kill paste tr sleep ps nice file comm cal newgrp tsort touch sed awk at sh AT&T UNIX Unix itself developed many of its tools (and therefore interfaces) in a single place, reduces coordination and integration costs and increasing the speed of evolution.
42. WHAT DOES THIS MEAN FOR SOFTWARE?
43. CRUFT IS INEVITABLE.
44. THE REAL WORLD IS MESSY. (IE6-8, Ruby 1.8, Rails 2.3, etc.)
45. SOLUTIONS ARE OFTEN UGLY.
46. SOLUTIONS ARE OFTEN CRUFTY. The cruft just reflects the ugliness of the underlying problem.
47. CRITICISM YIELDS DEFENSIVENESS. Early anti-cruft criticism, because it targets cruft that is aligned with real problems, creates an internal defensiveness to criticisms of cruft.
48. THIS BECOMES RECEIVED WISDOM. "Removing cruft for the sake of it will probably break something. Don't touch anything unless you're adding features! We knew what we were doing!"
49. REMOVING CRUFT. reinforces received wisdom
50. REMOVING CRUFT. good case reinforces received wisdom
51. REMOVING CRUFT. good case remove cruft reinforces received wisdom
52. REMOVING CRUFT. good case remove cruft test fails reinforces received wisdom
53. REMOVING CRUFT. good case remove cruft test fails aha! revert reinforces received wisdom
54. REMOVING CRUFT. good case remove cruft test fails aha! revert worse case reinforces received wisdom
55. REMOVING CRUFT. good case remove cruft test fails aha! revert worse case reinforces received wisdom remove cruft
56. REMOVING CRUFT. good case remove cruft test fails aha! revert worse case reinforces received wisdom remove cruft ship release
57. REMOVING CRUFT. good case remove cruft test fails aha! revert worse case reinforces received wisdom remove cruft ship release break users
58. REMOVING CRUFT. good case remove cruft test fails aha! revert worse case reinforces received wisdom remove cruft ship release break users oh shit! revert
59. PROJECTS DEVELOP IMMUNE SYSTEMS. If someone wants to remove cruft for the sake of it, people reject the changes out of hand.
60. BUT!
61. ASSUMPTIONS CHANGE.
62. JQUERY. DOM Node data a memory leak in IE blocks storing data on DOM nodes.
63. JQUERY. data a memory leak in IE blocks storing data on DOM nodes.
64. JQUERY. data leak a memory leak in IE blocks storing data on DOM nodes.
65. JQUERY. DOM Node jQuery1345727403427: 573 Cache data 573 this clearing needs to happen recursively.
66. JQUERY. Cache data 573 this clearing needs to happen recursively.
67. JQUERY. Cache this clearing needs to happen recursively.
68. IF WE SUPPORT OLDIE, WE NEED THIS CODE.
69. COMPETITORS WHO DECRY "CRUFT" ARE MISSING THE POINT.
70. BUT WHAT HAPPENS WHEN OLDIE BECOMES LESS RELEVANT?
71. FORK IN THE ROAD. By keeping track of the reasons for our cruft (and analyzing after the fact), we could determine the value to the codebase of dropping oldIE support.
72. LIFECYCLE. cruft defensiveness cruft = solution a gap appears received wisdom consolidation splash of cold water the most destructive part of the cycle is once the assumptions lose validity, the fact that an alternate solution does not solve every edge-case invalidates the choices they are making that use ecosystem changes to remove cruft.
73. A BETTER LIFECYCLE.
74. TRACK YOUR CRUFT! (CRUFT.txt?)
75. RE-EVALUATE ASSUMPTIONS PERIODICALLY.
76. IT'S OK TO BE "THE DOM LIBRARY FOR IE6 USERS" Projects built five years ago are in a good position to provide solutions for edge-cases that were prominent five years ago.
77. NOT JUST ABOUT SUPPORT MATRIX. Today, many people are moving towards making the browser "just another client". Tools built around server-side HTML rendering face similar choices.
78. WHAT DO DO?
79. KEEP TRACK OF EDGE CASES.
80. AND THEIR IMPACT ON ARCHITECTURE.
81. PERIODICALLY REVIEW DECISIONS.
82. MAKE CHOICES MORE INTENTIONAL. Help contributors make decisions without fear that they are walking on a grave.
83. THANKS!
84. QUESTIONS ? @wycats wycats@gmail.com