It's Harder than it Looks

Last month, I started a kickstarter to make it easier to install Rails on OSX and get started developing Rails applications. The response was largely positive, and I met my funding goal rather quickly. On the flip side, a number of folks publicly asked why this project is necessary. Surely, they reasoned, it would not be very difficult to script a Rails installation.

This sort of reasoning pervades the open source community, allowing apparent simplicity to drive inappropriately simplistic solutions. Worse, it implies that non-trivial solutions are "overengineering" the problem. Because open source solutions have historically leaked implementation complexity into the public API, some end developers have become wary of large solutions, often assuming that the simpler the solution, the better.

As examples of this phenomenon, I will talk about some aspects of Tokaido (the Rails project) that are unexpectedly difficult, and also how the bundler project faced a similar reaction in some circles. I will also talk about how certain seemingly complex solutions lead to win-wins by eliminating sources of errors that are not limited to new developers. These kinds of solutions always require more work and more code than the simple solutions, but they are worth it.

Transcript

1. Building software that actually works HARDER THAN IT LOOKS
2. ▪ Encodings in Rails ▪ Tokaido (Rails.app) ▪ Bundler ▪ Ember.js HARD THINGS.
3. WHAT IS HARD?
4. IDENTIFY THE SYMPTOMS.
5. UNDERSTAND THE DOMAIN.
6. IDENTIFY THE ROOT CAUSE.
7. CONSIDER POSSIBLE SOLUTIONS.
8. SOLVE THE ROOT CAUSE.
9. PERFORM ANY COMMUNITY WORK.
10. PROFIT.
11. ENCODINGS IN RAILS
12. A WILD REPLACEMENT CHARACTER APPEARS Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
13. A WILD CORRUPTED CHARACTER APPEARS ümlaut Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
14. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work ENCODINGS.
15. "Yehüda".encode("ISO-8859-1").dump => "Yeh\xFCda" "Yehüda".encode("UTF-8").dump => "Yeh\xC3xBCda" "Yehüda".encode("UTF-16").dump => "\xFE\xFF\x00Y\x00e\x00h\x00\xFC\x00d \x00a" STRINGS.
16. "Yehüda".encode("ISO-8859-1") => "\x59\x65\x68\xFC\x64\x61" "Yehüda".encode("UTF-8") => "\x59\x65\x68\xC3xBC\x64\x61" "Yehüda".encode("UTF-16") => "\xFE\xFF\x00Y\x00e\x00h\x00\xFC\x00d \x00a" BYTES. A String is a sequence of bytes. It doesn't have any marker indicating how those bytes should be interpreted.
17. Y e h u d a 0x59 0x65 0x68 0x75 0x64 0x61 UTF-8 Y e h u d a 0x59 0x65 0x68 0x75 0x64 0x61 Latin-1 ASCII CHARS ARE REPRESENTED THE SAME.
18. Y e h u d a 0x59 0x65 0x68 0x75 0x64 0x61 UTF-8 Y e h u d a 0x59 0x65 0x68 0x75 0x64 0x61 Latin-1 Y e h ü d a 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 UTF-8 Y e h ü d a 0x59 0x65 0x68 0xfc 0x64 0x61 Latin-1 ASCII CHARS ARE REPRESENTED THE SAME. OTHER CHARS ARE REPRESENTED DIFFERENTLY.
20. Y e h ü d a 0x59 0x65 0x68 0xfc 0x64 0x61 Latin-1
22. Y e h ü d a 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8 ACCIDENTALLY INTERPRET INCORRECTLY.
23. ? ? ? ? ? ? 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8
24. Y ? ? ? ? ? 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8
25. Y e ? ? ? ? 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8
26. Y e h ? ? ? 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8
27. Y e h ? ? 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8
28. Y e h d ? 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8
29. Y e h d a 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8
31. Y e h ü d a 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 UTF-8
32. Y e h ü d a 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1
33. Y e h ü d a 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1 ACCIDENTALLY INTERPRET INCORRECTLY.
34. ? ? ? ? ? ? ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1
35. Y ? ? ? ? ? ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1
36. Y e ? ? ? ? ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1
37. Y e h ? ? ? ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1
38. Y e h à ? ? ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1
39. Y e h à ¼ ? ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1
40. Y e h à ¼ d ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1
41. Y e h à ¼ d a 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1
42. BOUNDARIES. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
43. rails HTTP MySQL GET params Nokogiri templates POST params
44. rails UTF-8 HTTP MySQL GET params Nokogiri templates POST params We would like to be able to work inside Rails as if everything is UTF-8.
45. rails UTF-8 POST params
46. rails UTF-8 POST params browser name=yehuda
47. rails UTF-8 POST params browser name=yehüda
48. rails UTF-8 POST params browser name=yehüda
49. rails UTF-8 POST params browser name=\x59\x65\x68 \xfc\x64\x61
50. WHAT ENCODING?
51. THE ENCODING OF THE PAGE! Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
52. cattr_accessor(:default_charset) { "utf-8" } CONTENT_TYPE = "Content-Type".freeze def assign_default_content_type_and_charset! return if headers[CONTENT_TYPE].present? @content_type ||= Mime::HTML @charset ||= self.class.default_charset type = @content_type.to_s.dup unless @sending_file type << "; charset=#{@charset}" end headers[CONTENT_TYPE] = type end PAGE'S ENCODING.
53. Content-Type: text/html; charset=utf-8
57. form attribute: accept-charset This attribute speci es the list of character encodings for input data that is accepted by the server processing this form “ HTML4 SPEC Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
58. form attribute: accept-charset This attribute speci es the list of character encodings for input data that is accepted by the server processing this form “ HTML4 SPEC Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
59. form attribute: accept-charset Possible values: UTF-8 If the user enters characters that are not in the character set of the document containing the form, the UTF-8 character set is used. “ MSDN
60. form attribute: accept-charset Possible values: UTF-8 If the user enters characters that are not in the character set of the document containing the form, the UTF-8 character set is used. “ MSDN
61. form attribute: accept-charset Possible values: UTF-8 If the user enters characters that are not in the character set of the document containing the form, the UTF-8 character set is used. “ MSDN
62. form attribute: accept-charset Possible values: UTF-8 If the user enters characters that are not in the character set of the document containing the form, the UTF-8 character set is used. “ MSDN
63. <form accept-charset="utf-8"> <textarea> Latin-1 chárâctërs pasted from another application </textarea> <button type="submit">Do it!</button> </form> PROBLEM.
64. FAILURE. User changes encoding to Latin-1 User pastes chárâctërs from Word IE sees that chárâctërs are in Latin-1 IE ignores accept-charset :( :( :(
65. <form accept-charset="utf-8"> <input type="hidden" name="utf8" value="✓"> <textarea> Latin-1 chárâctërs pasted from another application </textarea> <button type="submit">Do it!</button> </form> SNOWMAN HACK!
66. SUCCESS. User changes encoding to Latin-1 User pastes chárâctërs from Word IE sees that ✓ is not in Latin-1 IE honors accept-charset <3<3<3
67. IN CONTROL. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work rails UTF-8 POST params
68. rails UTF-8 HTTP MySQL GET params Nokogiri templates POST params Now rinse and repeat for the rest of these. REPEAT. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
69. FORCE_ENCODING? Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
70. DEFAULT_INTERNAL. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
71. TOKAIDO
72. I was keenly interested in an open source open government project (Sunlight Foundation). I tried to get it running on my Mac, on my personal time. I had direct expert help (friends who dev using Ruby, Rails, etc.). I understand the tool stack was churning, something about mismatched versions, runtimes, whatever. I gave up after three weeks. “ HACKER NEWS.
73. I was keenly interested in an open source open government project (Sunlight Foundation). I tried to get it running on my Mac, on my personal time. I had direct expert help (friends who dev using Ruby, Rails, etc.). I understand the tool stack was churning, something about mismatched versions, runtimes, whatever. I gave up after three weeks. “ HACKER NEWS. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
74. STATIC VS. DYNAMIC LINKING.
75. STATIC VS. DYNAMIC LINKING. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
76. > require "psych" => true
77. $ DYLD_PRINT_BINDINGS=1 ruby -rpsych -e"" ... snip ... dyld: loaded: <snip>psych.bundle dyld: loaded: <snip>libyaml-0.2.dylib DYNAMIC LOAD.
78. // ext/psych/parser.c static VALUE allocate(VALUE klass) { yaml_parser_t * parser; parser = xmalloc(sizeof(yaml_parser_t)); yaml_parser_initialize(parser); return Data_Wrap_Struct(klass, 0, dealloc, parser); } WHY?
79. $ otool -L psych.bundle /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby. 1.9.1.dylib (compatibility version 1.9.1, current version 1.9.1) " /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0) " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) HOW IT WORKS.
80. $ otool -L psych.bundle /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby. 1.9.1.dylib (compatibility version 1.9.1, current version 1.9.1) " /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0) " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) HOW IT WORKS.
81. $ otool -L psych.bundle /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby. 1.9.1.dylib (compatibility version 1.9.1, current version 1.9.1) " /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0) " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) HOW IT WORKS.
82. ruby libyaml psych.bundle libruby
83. ruby libyaml psych.bundle libruby Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
84. $ otool -L psych.bundle /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby. 1.9.1.dylib (compatibility version 1.9.1, current version 1.9.1) " /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0) " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) HIDDEN PROBLEMS.
85. $ otool -L psych.bundle /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby. 1.9.1.dylib (compatibility version 1.9.1, current version 1.9.1) " /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0) " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) HIDDEN PROBLEMS.
86. JUST FOR PSYCH.
87. ALSO ZLIB, OPENSSL, READLINE, ETC.
88. MISSING SQLITE3.H!?
89. AUTOMATE COMPILATION? Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
90. MOST OF THE PROBLEMS HAPPEN IN COMPILATION. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
91. PRECOMPILE RUBY?
92. $ otool -L psych.bundle /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby. 1.9.1.dylib (compatibility version 1.9.1, current version 1.9.1) " /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0) " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) HIDDEN PROBLEMS. Does the precompiled Ruby actually solve these problems? Is it actually portable?
93. STATICALLY COMPILED RUBY AND DEPENDENCIES. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
94. ./configure --static (libyaml) ./configure --enable-static (pkg-config) ./configure --disable-shared (others) pkg-config (needed for openssl) STATICALLY COMPILE.
95. ./configure --static (libyaml) ./configure --enable-static (pkg-config) ./configure --disable-shared (others) pkg-config (needed for openssl) STATICALLY COMPILE.
96. $ man ld -search_paths_first This is now the default (in Xcode4 tools). When processing -lx the linker now searches each directory in its library search paths for `libx.dylib' then `libx.a' before the moving on to the next path in the library search path. LINKER. Make sure that the linker finds the .a files before it finds system- installed .dylibs
97. $ otool -L psych.bundle /Users/wycats/.sm/pkg/versions/tokaidoapp/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) PROBLEM SOLVED? Does the precompiled Ruby actually solve these problems? Is it actually portable?
98. PRECOMPILED STATIC GEMS. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
99. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work A simple solution to a complex problem by definition means that much of the original complexity is hidden from you. That means, perversely, that the better a tool is at solving a complex problem, the less it seems necessary. Often complex problems seem simple to the naïve observer. If you are comparing two solutions and one is 10,000 lines of code and the other 1,000 lines of code, consider that the implementor of the second solution may not yet have a deep enough understanding of the problem to shield you the user from the complexity of the problem. Consider that the extra 9,000 lines that may seem like bloat to you may in fact be the full understanding of a complex problem, written down in code.
100. THANKS! @WYCATS