It's Harder than it Looks

It's Harder than it Looks

Building software that actually works

428167a3ec72235ba971162924492609?s=128

Yehuda Katz

May 03, 2012
Tweet

Transcript

  1. 10.
  2. 12.

    A WILD REPLACEMENT CHARACTER APPEARS Identify symptoms Understand domain Identify

    root cause Consider solutions Solve root cause Community work
  3. 13.

    A WILD CORRUPTED CHARACTER APPEARS ümlaut Identify symptoms Understand domain

    Identify root cause Consider solutions Solve root cause Community work
  4. 17.

    Y e h u d a 0x59 0x65 0x68 0x75

    0x64 0x61 UTF-8 Y e h u d a 0x59 0x65 0x68 0x75 0x64 0x61 Latin-1 ASCII CHARS ARE REPRESENTED THE SAME.
  5. 18.

    Y e h u d a 0x59 0x65 0x68 0x75

    0x64 0x61 UTF-8 Y e h u d a 0x59 0x65 0x68 0x75 0x64 0x61 Latin-1 Y e h ü d a 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 UTF-8 Y e h ü d a 0x59 0x65 0x68 0xfc 0x64 0x61 Latin-1 ASCII CHARS ARE REPRESENTED THE SAME. OTHER CHARS ARE REPRESENTED DIFFERENTLY.
  6. 19.
  7. 20.
  8. 21.
  9. 22.

    Y e h ü d a 0x59 0x65 0x68 0xfc

    0x64 0x61 UTF-8 ACCIDENTALLY INTERPRET INCORRECTLY.
  10. 23.
  11. 24.
  12. 25.
  13. 26.
  14. 30.
  15. 31.

    Y e h ü d a 0x59 0x65 0x68 0xc3

    0xbc 0x64 0x61 UTF-8
  16. 32.

    Y e h ü d a 0x59 0x65 0x68 0xc3

    0xbc 0x64 0x61 Latin-1
  17. 33.

    Y e h ü d a 0x59 0x65 0x68 0xc3

    0xbc 0x64 0x61 Latin-1 ACCIDENTALLY INTERPRET INCORRECTLY.
  18. 34.

    ? ? ? ? ? ? ? 0x59 0x65 0x68

    0xc3 0xbc 0x64 0x61 Latin-1
  19. 35.

    Y ? ? ? ? ? ? 0x59 0x65 0x68

    0xc3 0xbc 0x64 0x61 Latin-1
  20. 36.

    Y e ? ? ? ? ? 0x59 0x65 0x68

    0xc3 0xbc 0x64 0x61 Latin-1
  21. 37.

    Y e h ? ? ? ? 0x59 0x65 0x68

    0xc3 0xbc 0x64 0x61 Latin-1
  22. 38.

    Y e h à ? ? ? 0x59 0x65 0x68

    0xc3 0xbc 0x64 0x61 Latin-1
  23. 39.

    Y e h à ¼ ? ? 0x59 0x65 0x68

    0xc3 0xbc 0x64 0x61 Latin-1
  24. 40.

    Y e h à ¼ d ? 0x59 0x65 0x68

    0xc3 0xbc 0x64 0x61 Latin-1
  25. 41.

    Y e h à ¼ d a 0x59 0x65 0x68

    0xc3 0xbc 0x64 0x61 Latin-1
  26. 44.

    rails UTF-8 HTTP MySQL GET params Nokogiri templates POST params

    We would like to be able to work inside Rails as if everything is UTF-8.
  27. 51.

    THE ENCODING OF THE PAGE! Identify symptoms Understand domain Identify

    root cause Consider solutions Solve root cause Community work
  28. 52.

    cattr_accessor(:default_charset) { "utf-8" } CONTENT_TYPE = "Content-Type".freeze def assign_default_content_type_and_charset! return

    if headers[CONTENT_TYPE].present? @content_type ||= Mime::HTML @charset ||= self.class.default_charset type = @content_type.to_s.dup unless @sending_file type << "; charset=#{@charset}" end headers[CONTENT_TYPE] = type end PAGE'S ENCODING.
  29. 54.
  30. 55.
  31. 56.
  32. 57.

    form attribute: accept-charset This attribute speci es the list of

    character encodings for input data that is accepted by the server processing this form “ HTML4 SPEC Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
  33. 58.

    form attribute: accept-charset This attribute speci es the list of

    character encodings for input data that is accepted by the server processing this form “ HTML4 SPEC Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
  34. 59.

    form attribute: accept-charset Possible values: UTF-8 If the user enters

    characters that are not in the character set of the document containing the form, the UTF-8 character set is used. “ MSDN
  35. 60.

    form attribute: accept-charset Possible values: UTF-8 If the user enters

    characters that are not in the character set of the document containing the form, the UTF-8 character set is used. “ MSDN
  36. 61.

    form attribute: accept-charset Possible values: UTF-8 If the user enters

    characters that are not in the character set of the document containing the form, the UTF-8 character set is used. “ MSDN
  37. 62.

    form attribute: accept-charset Possible values: UTF-8 If the user enters

    characters that are not in the character set of the document containing the form, the UTF-8 character set is used. “ MSDN
  38. 64.

    FAILURE. User changes encoding to Latin-1 User pastes chárâctërs from

    Word IE sees that chárâctërs are in Latin-1 IE ignores accept-charset :( :( :(
  39. 65.

    <form accept-charset="utf-8"> <input type="hidden" name="utf8" value="✓"> <textarea> Latin-1 chárâctërs pasted

    from another application </textarea> <button type="submit">Do it!</button> </form> SNOWMAN HACK!
  40. 66.

    SUCCESS. User changes encoding to Latin-1 User pastes chárâctërs from

    Word IE sees that ✓ is not in Latin-1 IE honors accept-charset <3<3<3
  41. 67.

    IN CONTROL. Identify symptoms Understand domain Identify root cause Consider

    solutions Solve root cause Community work rails UTF-8 POST params
  42. 68.

    rails UTF-8 HTTP MySQL GET params Nokogiri templates POST params

    Now rinse and repeat for the rest of these. REPEAT. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
  43. 71.
  44. 72.

    I was keenly interested in an open source open government

    project (Sunlight Foundation). I tried to get it running on my Mac, on my personal time. I had direct expert help (friends who dev using Ruby, Rails, etc.). I understand the tool stack was churning, something about mismatched versions, runtimes, whatever. I gave up after three weeks. “ HACKER NEWS.
  45. 73.

    I was keenly interested in an open source open government

    project (Sunlight Foundation). I tried to get it running on my Mac, on my personal time. I had direct expert help (friends who dev using Ruby, Rails, etc.). I understand the tool stack was churning, something about mismatched versions, runtimes, whatever. I gave up after three weeks. “ HACKER NEWS. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work
  46. 75.

    STATIC VS. DYNAMIC LINKING. Identify symptoms Understand domain Identify root

    cause Consider solutions Solve root cause Community work
  47. 77.

    $ DYLD_PRINT_BINDINGS=1 ruby -rpsych -e"" ... snip ... dyld: loaded:

    <snip>psych.bundle dyld: loaded: <snip>libyaml-0.2.dylib DYNAMIC LOAD.
  48. 78.

    // ext/psych/parser.c static VALUE allocate(VALUE klass) { yaml_parser_t * parser;

    parser = xmalloc(sizeof(yaml_parser_t)); yaml_parser_initialize(parser); return Data_Wrap_Struct(klass, 0, dealloc, parser); } WHY?
  49. 79.

    $ otool -L psych.bundle /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby. 1.9.1.dylib (compatibility

    version 1.9.1, current version 1.9.1) " /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0) " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) HOW IT WORKS.
  50. 80.

    $ otool -L psych.bundle /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby. 1.9.1.dylib (compatibility

    version 1.9.1, current version 1.9.1) " /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0) " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) HOW IT WORKS.
  51. 81.

    $ otool -L psych.bundle /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby. 1.9.1.dylib (compatibility

    version 1.9.1, current version 1.9.1) " /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0) " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) HOW IT WORKS.
  52. 83.

    ruby libyaml psych.bundle libruby Identify symptoms Understand domain Identify root

    cause Consider solutions Solve root cause Community work
  53. 84.

    $ otool -L psych.bundle /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby. 1.9.1.dylib (compatibility

    version 1.9.1, current version 1.9.1) " /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0) " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) HIDDEN PROBLEMS.
  54. 85.

    $ otool -L psych.bundle /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby. 1.9.1.dylib (compatibility

    version 1.9.1, current version 1.9.1) " /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0) " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) HIDDEN PROBLEMS.
  55. 90.

    MOST OF THE PROBLEMS HAPPEN IN COMPILATION. Identify symptoms Understand

    domain Identify root cause Consider solutions Solve root cause Community work
  56. 92.

    $ otool -L psych.bundle /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby. 1.9.1.dylib (compatibility

    version 1.9.1, current version 1.9.1) " /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0) " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) HIDDEN PROBLEMS. Does the precompiled Ruby actually solve these problems? Is it actually portable?
  57. 93.

    STATICALLY COMPILED RUBY AND DEPENDENCIES. Identify symptoms Understand domain Identify

    root cause Consider solutions Solve root cause Community work
  58. 96.

    $ man ld -search_paths_first This is now the default (in

    Xcode4 tools). When processing -lx the linker now searches each directory in its library search paths for `libx.dylib' then `libx.a' before the moving on to the next path in the library search path. LINKER. Make sure that the linker finds the .a files before it finds system- installed .dylibs
  59. 97.

    $ otool -L psych.bundle /Users/wycats/.sm/pkg/versions/tokaidoapp/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /usr/lib/libSystem.B.dylib (compatibility version

    1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) PROBLEM SOLVED? Does the precompiled Ruby actually solve these problems? Is it actually portable?
  60. 98.
  61. 99.

    Identify symptoms Understand domain Identify root cause Consider solutions Solve

    root cause Community work A simple solution to a complex problem by definition means that much of the original complexity is hidden from you. That means, perversely, that the better a tool is at solving a complex problem, the less it seems necessary. Often complex problems seem simple to the naïve observer. If you are comparing two solutions and one is 10,000 lines of code and the other 1,000 lines of code, consider that the implementor of the second solution may not yet have a deep enough understanding of the problem to shield you the user from the complexity of the problem. Consider that the extra 9,000 lines that may seem like bloat to you may in fact be the full understanding of a complex problem, written down in code.