WTF-8

Recently, I had to fix a pernicious encoding bug in a Rails app I work on. Along the way, I learned a lot about string encodings, how they work, *why* they work, and how Ruby handles them. Join me as I share the fruits of my journey of discovery with you!

Transcript

1. WTF-8 Leigh Caplan texel1@gmail.com github.com/texel
4. class FakeController def show Event.add( "The user <subject>$0</subject> viewed the page.", current_user.name ) end end
5. class Event def self.add(format_string, user_name) doc = Nokogiri::XML::DocumentFragment.parse(format_string) subject = doc.css('subject').first subject.inner_html = subject.inner_html.gsub( "$0", user_name) create(:body => doc.to_s) end end
6. current_user.name = "Leigh Caplan" "The user <subject>Leigh Caplan</subject> viewed the page." current_user.name = "José Valim" "The user <subject></subject> viewed the page."
7. Repro steps? ‣ Test? Nope. ‣ Console? Nope. ‣ Dev environment? Yep.
8. describe Event do describe ".add" do it "should preserve special characters in names" do event = Event.add( "The user <subject>$0</subject> viewed the page.", "José Valim" ) event.body.should =~ /José/ end end end
9. So, what’s going on here?
10. joelonsoftware.com/articles/Unicode.html
11. What is an encoding?
12. Way to represent characters as binary data "hello" # => 104 101 108 108 111 # => 1101000 1100101 1101100 1101100 1101111
13. ASCII ‣ Invented in 1960s, prominent from the 1970s onward ‣ 7 bit encoding - 128 characters ‣ 0-31 control characters ‣ 32-127 characters (0-9, a-z, A-Z, etc)
14. 128-255: ¯\_(π)_/¯
15. ANSI Code Pages
16. ISO-8859-1 (Latin-1)
17. Asian Languages
18. “Good enough!”
19. “Good enough!” Because, the Internet.
20. Unicode
21. Unicode ‣ Character mapping, not an encoding ‣ Attempt to establish universal character set ‣ Covers all writing systems that ever existed ‣ Disambiguation of character definitions ‣ System of mapping characters to intermediate list of numbers (code points)
22. U+<hex number>
23. U+0041 => A
24. U+2603 => ‚
25. "hello" # => U+0068 U+0065 U+006C U+006C U+006F
26. Unicode Encodings
27. UTF-16
28. UTF-16 ‣ 2 bytes per character (16 bit) ‣ 65,536 total characters ‣ Inefficient storage for primarily-european strings ‣ Endinanness matters, byte order mark optional
30. "hello" # => FEFF 0068 0065 006C 006C 006F # => FFFE 6800 6500 6C00 6C00 6F00
31. ಠ_ಠ
32. UTF-8
33. UTF-8 ‣ Single byte for code points 0-127 ‣ Variable length (up to 6 bytes) ‣ Most common encoding found on the Web as of 2008 (reported by Google) ‣ Backwards-compatible with ASCII
34. ASCII => UTF-8 Hello Hello UTF-8 => ASCII Touché Touch☐
35. “There ain’t no such thing as plain text.” ~ Joel Spolsky
37. WTF yehudakatz.com/2010/05/17/encodings-unabridged/
38. In Ruby <=1.8 ‣ Strings were treated as raw bytes ‣ Concatenating strings of different encodings would result in corrupt strings ‣ Counting chars in multi-byte encoded strings was inaccurate
39. "Touché".length # => 7 In Ruby <=1.8
40. In Ruby 1.9+ ‣ Strings keep their native encoding ‣ Strings are “tagged” with an encoding ‣ String methods are encoding-aware
41. "Touché".length # => 6 In Ruby 1.9+
42. Why not just convert everything to UTF-*?
43. ShiftJIS ➠ Unicode ➠ ShiftJIS = ☹ Round trip sadness
44. Ruby encoding grab bag
45. Source file encodings ‣ Assumed to be US-ASCII unless explicitly specified ‣ Any source file w/ non-ASCII characters needs an encoding comment ‣ Determines encoding of all string literals in file
46. Get a string’s encoding string = "Hello there!" string.encoding # => #<Encoding:UTF-8>
47. Force encoding string.force_encoding('iso-8859-1') string.encoding # => #<Encoding:ISO-8859-1>
48. Convert encoding string = "omg ❤" string.encode('ISO-8859-1') # Encoding::UndefinedConversionError: U+2764 from UTF-8 to ISO-8859-1 string.encode('ISO-8859-1', :undef => :replace) # "omg ?"
49. Bad: DEFAULT_JOB_TITLE = "Señor Software Engineer"
50. Good: # encoding: utf-8 DEFAULT_JOB_TITLE = "Señor Software Engineer"
51. Default external ‣ Encoding used by default when reading from files and streams ‣ Global to entire Ruby process ‣ Doesn't affect string literals! ‣ Set automatically using computer's locale, but overrideable using assignment
52. Default external ENV['LANG'] # "en_US.UTF-8" Encoding.default_external = "UTF-8"
53. Default internal ‣ Normally Ruby leaves data from files and streams in its native encoding. ‣ Encoding.default_internal causes transcoding to specified encoding (rather than just tagging)
54. Default internal Encoding.default_internal = "UTF-8"
55. __ENCODING__ ‣ Evaluates to the encoding of the file you're currently working in
56. Meanwhile, back at the ranch...
57. class Event def self.add(format_string, user_name) binding.pry # Debugging! doc = Nokogiri::XML::DocumentFragment.parse(format_string) subject = doc.css('subject').first subject.inner_html = subject.inner_html.gsub( "$0", user_name) create(:body => doc.to_s) end end
58. >> format_string.encoding # => #<Encoding:US-ASCII>
59. # encoding: utf-8 describe Event do describe ".add" do it "should preserve special characters in names" do event = Event.add( "The user <subject>$0</subject> viewed the page.", "José Valim" ) event.body.should =~ /José/ end end end
60. Source file encodings ‣ Assumed to be US-ASCII unless explicitly specified ‣ Any source file w/ non-ASCII characters needs an encoding comment ‣ Determines encoding of all string literals in file
62. The Fix(es)
63. format_string.force_encoding('UTF-8') Nokogiri::XML::DocumentFragment.parse( format_string, 'UTF-8')
64. describe Event do describe ".add" do it "should preserve special characters in names" do format_string = "The user <subject>$0</subject> viewed the page." event = Event.add( format_string.force_encoding('US-ASCII'), "José Valim" ) event.body.should =~ /José/ end end end
65. Yak successfully shaved!
66. Questions