It's Harder than it Looks

Slide 1

Slide 1 text

Building software that actually works HARDER THAN IT LOOKS

Slide 2

Slide 2 text

■ Encodings in Rails ■ Tokaido (Rails.app) ■ Bundler ■ Ember.js HARD THINGS.

Slide 3

Slide 3 text

WHAT IS HARD?

Slide 4

Slide 4 text

IDENTIFY THE SYMPTOMS.

Slide 5

Slide 5 text

UNDERSTAND THE DOMAIN.

Slide 6

Slide 6 text

IDENTIFY THE ROOT CAUSE.

Slide 7

Slide 7 text

CONSIDER POSSIBLE SOLUTIONS.

Slide 8

Slide 8 text

SOLVE THE ROOT CAUSE.

Slide 9

Slide 9 text

PERFORM ANY COMMUNITY WORK.

Slide 10

Slide 10 text

PROFIT.

Slide 11

Slide 11 text

ENCODINGS IN RAILS

Slide 12

Slide 12 text

A WILD REPLACEMENT CHARACTER APPEARS Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 13

Slide 13 text

A WILD CORRUPTED CHARACTER APPEARS Ã¼mlaut Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 14

Slide 14 text

Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work ENCODINGS.

Slide 15

Slide 15 text

"Yehüda".encode("ISO-8859-1").dump => "Yeh\xFCda" "Yehüda".encode("UTF-8").dump => "Yeh\xC3xBCda" "Yehüda".encode("UTF-16").dump => "\xFE\xFF\x00Y\x00e\x00h\x00\xFC\x00d \x00a" STRINGS.

Slide 16

Slide 16 text

"Yehüda".encode("ISO-8859-1") => "\x59\x65\x68\xFC\x64\x61" "Yehüda".encode("UTF-8") => "\x59\x65\x68\xC3xBC\x64\x61" "Yehüda".encode("UTF-16") => "\xFE\xFF\x00Y\x00e\x00h\x00\xFC\x00d \x00a" BYTES. A String is a sequence of bytes. It doesn't have any marker indicating how those bytes should be interpreted.

Slide 17

Slide 17 text

Y e h u d a 0x59 0x65 0x68 0x75 0x64 0x61 UTF-8 Y e h u d a 0x59 0x65 0x68 0x75 0x64 0x61 Latin-1 ASCII CHARS ARE REPRESENTED THE SAME.

Slide 18

Slide 18 text

Y e h u d a 0x59 0x65 0x68 0x75 0x64 0x61 UTF-8 Y e h u d a 0x59 0x65 0x68 0x75 0x64 0x61 Latin-1 Y e h ü d a 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 UTF-8 Y e h ü d a 0x59 0x65 0x68 0xfc 0x64 0x61 Latin-1 ASCII CHARS ARE REPRESENTED THE SAME. OTHER CHARS ARE REPRESENTED DIFFERENTLY.

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Y e h ü d a 0x59 0x65 0x68 0xfc 0x64 0x61 Latin-1

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

Y e h ü d a 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8 ACCIDENTALLY INTERPRET INCORRECTLY.

Slide 23

Slide 23 text

? ? ? ? ? ? 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8

Slide 24

Slide 24 text

Y ? ? ? ? ? 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8

Slide 25

Slide 25 text

Y e ? ? ? ? 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8

Slide 26

Slide 26 text

Y e h ? ? ? 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8

Slide 27

Slide 27 text

Y e h ? ? 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8

Slide 28

Slide 28 text

Y e h d ? 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8

Slide 29

Slide 29 text

Y e h d a 0x59 0x65 0x68 0xfc 0x64 0x61 UTF-8

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

Y e h ü d a 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 UTF-8

Slide 32

Slide 32 text

Y e h ü d a 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1

Slide 33

Slide 33 text

Y e h ü d a 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1 ACCIDENTALLY INTERPRET INCORRECTLY.

Slide 34

Slide 34 text

? ? ? ? ? ? ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1

Slide 35

Slide 35 text

Y ? ? ? ? ? ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1

Slide 36

Slide 36 text

Y e ? ? ? ? ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1

Slide 37

Slide 37 text

Y e h ? ? ? ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1

Slide 38

Slide 38 text

Y e h Ã ? ? ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1

Slide 39

Slide 39 text

Y e h Ã ¼ ? ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1

Slide 40

Slide 40 text

Y e h Ã ¼ d ? 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1

Slide 41

Slide 41 text

Y e h Ã ¼ d a 0x59 0x65 0x68 0xc3 0xbc 0x64 0x61 Latin-1

Slide 42

Slide 42 text

BOUNDARIES. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 43

Slide 43 text

rails HTTP MySQL GET params Nokogiri templates POST params

Slide 44

Slide 44 text

rails UTF-8 HTTP MySQL GET params Nokogiri templates POST params We would like to be able to work inside Rails as if everything is UTF-8.

Slide 45

Slide 45 text

rails UTF-8 POST params

Slide 46

Slide 46 text

rails UTF-8 POST params browser name=yehuda

Slide 47

Slide 47 text

rails UTF-8 POST params browser name=yehüda

Slide 48

Slide 48 text

rails UTF-8 POST params browser name=yehüda

Slide 49

Slide 49 text

rails UTF-8 POST params browser name=\x59\x65\x68 \xfc\x64\x61

Slide 50

Slide 50 text

WHAT ENCODING?

Slide 51

Slide 51 text

THE ENCODING OF THE PAGE! Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 52

Slide 52 text

cattr_accessor(:default_charset) { "utf-8" } CONTENT_TYPE = "Content-Type".freeze def assign_default_content_type_and_charset! return if headers[CONTENT_TYPE].present? @content_type ||= Mime::HTML @charset ||= self.class.default_charset type = @content_type.to_s.dup unless @sending_file type << "; charset=#{@charset}" end headers[CONTENT_TYPE] = type end PAGE'S ENCODING.

Slide 53

Slide 53 text

Content-Type: text/html; charset=utf-8

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

form attribute: accept-charset This attribute speci es the list of character encodings for input data that is accepted by the server processing this form “ HTML4 SPEC Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 58

Slide 58 text

Slide 59

Slide 59 text

form attribute: accept-charset Possible values: UTF-8 If the user enters characters that are not in the character set of the document containing the form, the UTF-8 character set is used. “ MSDN

Slide 60

Slide 60 text

form attribute: accept-charset Possible values: UTF-8 If the user enters characters that are not in the character set of the document containing the form, the UTF-8 character set is used. “ MSDN

Slide 61

Slide 61 text

form attribute: accept-charset Possible values: UTF-8 If the user enters characters that are not in the character set of the document containing the form, the UTF-8 character set is used. “ MSDN

Slide 62

Slide 62 text

form attribute: accept-charset Possible values: UTF-8 If the user enters characters that are not in the character set of the document containing the form, the UTF-8 character set is used. “ MSDN

Slide 63

Slide 63 text

Latin-1 chárâctërs pasted from another application Do it! PROBLEM.

Slide 64

Slide 64 text

FAILURE. User changes encoding to Latin-1 User pastes chárâctërs from Word IE sees that chárâctërs are in Latin-1 IE ignores accept-charset :( :( :(

Slide 65

Slide 65 text

Latin-1 chárâctërs pasted from another application Do it! SNOWMAN HACK!

Slide 66

Slide 66 text

SUCCESS. User changes encoding to Latin-1 User pastes chárâctërs from Word IE sees that ✓ is not in Latin-1 IE honors accept-charset <3<3<3

Slide 67

Slide 67 text

IN CONTROL. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work rails UTF-8 POST params

Slide 68

Slide 68 text

rails UTF-8 HTTP MySQL GET params Nokogiri templates POST params Now rinse and repeat for the rest of these. REPEAT. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 69

Slide 69 text

FORCE_ENCODING? Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 70

Slide 70 text

DEFAULT_INTERNAL. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 71

Slide 71 text

TOKAIDO

Slide 72

Slide 72 text

I was keenly interested in an open source open government project (Sunlight Foundation). I tried to get it running on my Mac, on my personal time. I had direct expert help (friends who dev using Ruby, Rails, etc.). I understand the tool stack was churning, something about mismatched versions, runtimes, whatever. I gave up after three weeks. “ HACKER NEWS.

Slide 73

Slide 73 text

Slide 74

Slide 74 text

STATIC VS. DYNAMIC LINKING.

Slide 75

Slide 75 text

STATIC VS. DYNAMIC LINKING. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 76

Slide 76 text

> require "psych" => true

Slide 77

Slide 77 text

$ DYLD_PRINT_BINDINGS=1 ruby -rpsych -e"" ... snip ... dyld: loaded: psych.bundle dyld: loaded: libyaml-0.2.dylib DYNAMIC LOAD.

Slide 78

Slide 78 text

// ext/psych/parser.c static VALUE allocate(VALUE klass) { yaml_parser_t * parser; parser = xmalloc(sizeof(yaml_parser_t)); yaml_parser_initialize(parser); return Data_Wrap_Struct(klass, 0, dealloc, parser); } WHY?

Slide 79

Slide 79 text

$ otool -L psych.bundle /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby. 1.9.1.dylib (compatibility version 1.9.1, current version 1.9.1) " /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0) " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) HOW IT WORKS.

Slide 80

Slide 80 text

Slide 81

Slide 81 text

Slide 82

Slide 82 text

ruby libyaml psych.bundle libruby

Slide 83

Slide 83 text

ruby libyaml psych.bundle libruby Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 84

Slide 84 text

Slide 85

Slide 85 text

Slide 86

Slide 86 text

JUST FOR PSYCH.

Slide 87

Slide 87 text

ALSO ZLIB, OPENSSL, READLINE, ETC.

Slide 88

Slide 88 text

MISSING SQLITE3.H!?

Slide 89

Slide 89 text

AUTOMATE COMPILATION? Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 90

Slide 90 text

MOST OF THE PROBLEMS HAPPEN IN COMPILATION. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 91

Slide 91 text

PRECOMPILE RUBY?

Slide 92

Slide 92 text

Slide 93

Slide 93 text

STATICALLY COMPILED RUBY AND DEPENDENCIES. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 94

Slide 94 text

./configure --static (libyaml) ./configure --enable-static (pkg-config) ./configure --disable-shared (others) pkg-config (needed for openssl) STATICALLY COMPILE.

Slide 95

Slide 95 text

./configure --static (libyaml) ./configure --enable-static (pkg-config) ./configure --disable-shared (others) pkg-config (needed for openssl) STATICALLY COMPILE.

Slide 96

Slide 96 text

$ man ld -search_paths_first This is now the default (in Xcode4 tools). When processing -lx the linker now searches each directory in its library search paths for `libx.dylib' then `libx.a' before the moving on to the next path in the library search path. LINKER. Make sure that the linker finds the .a files before it finds system- installed .dylibs

Slide 97

Slide 97 text

$ otool -L psych.bundle /Users/wycats/.sm/pkg/versions/tokaidoapp/lib/ruby/1.9.1/ x86_64-darwin11.3.0/psych.bundle: " /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) " /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) PROBLEM SOLVED? Does the precompiled Ruby actually solve these problems? Is it actually portable?

Slide 98

Slide 98 text

PRECOMPILED STATIC GEMS. Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work

Slide 99

Slide 99 text

Identify symptoms Understand domain Identify root cause Consider solutions Solve root cause Community work A simple solution to a complex problem by definition means that much of the original complexity is hidden from you. That means, perversely, that the better a tool is at solving a complex problem, the less it seems necessary. Often complex problems seem simple to the naïve observer. If you are comparing two solutions and one is 10,000 lines of code and the other 1,000 lines of code, consider that the implementor of the second solution may not yet have a deep enough understanding of the problem to shield you the user from the complexity of the problem. Consider that the extra 9,000 lines that may seem like bloat to you may in fact be the full understanding of a complex problem, written down in code.

Slide 100

Slide 100 text

THANKS! @WYCATS