Pythons in the tower of babel

Or how to decode those unicode errors

Transcript

1. Pythons within the tower of Babel Or how to decode those Unicode errors Mario Corchero News Automation @Bloomberg
2. History of text - ASCII • American Standard Code for Information Interchange • 0-127
3. ASCII – Like EST for text • Accents? • Chineese? • Arabic?
4. Unicode • 0 to 0x10ffff • A table that maps all possible characters into a “code point” • An Unicode string is a sequence of code points A U+0041 Glyph Code point Latin capital letter A Character
5. Unicode - Save that to disk • A rule to translate an Unicode string to bytes is called encoding. The same way we encode/encode letters into sound the Unicode encodings allow us to encode/decode into bytes
6. Some encodings • UTF-16 • UTF-32 • UTF-8 (variable length) • ASCII (Python 2 defaults)
7. Python Python 2 Python 3 Default encoding: ascii Implicit conversions Default encoding: utf8
8. Conversions • Unicode to Bytes: encode • Bytes to Unicode: decode
9. Encodings
10. Encodings
12. Take home Bytes vs Unicode – There is no “string”
13. Take home Encodings – Know your stuff
14. Take home Python2 vs Python3
15. Take home Unicode Sandwich
16. Take home • Bytes vs Unicode – There is no “string” • Encodings – Know your stuff • Python2 vs Python3 • Unicode Sandwich • Test with unicode
17. Common errors
18. Interpreter “SyntaxError: Non-ASCII character '\xeb' in file test.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details”
19. Terminal setup “UnicodeEncodeError: 'charmap' codec can't encode character u'\u1234' in position 0: character maps to <undefined>” This means your console is not setup to display Unicode See https://wiki.python.org/moin/PrintFails
20. Operations on str objects
21. JSON JSON module is broken (yes and no) When parsing in javascript I get: "æ± è¯"
22. JSON Input Ensure_ascii output u” 汉语” True '"\\u6c49\\u8bed"' u” 汉语” False u'"\u6c49\u8bed"' u”汉语".encode("utf-8") True '"\\u6c49\\u8bed"’ u”汉语".encode("utf-8") False '"\xe6\xb1\x89\xe8\xaf\xad"' [u"汉语", u"汉语".encode("utf-8")] True '["\\u6c49\\u8bed", "\\u6c49\\u8bed"]' [u"汉语", u"汉语".encode("utf-8")] False UnicodeDecodeError There is no binary in the JSON format!
23. Formatting with str from __future__ import __unicode_literals__ Change the default encoding? No thanks
24. Opening files • Python2 open works with bytes
25. Take home
26. Take home I ♥ Unicode And utf-8 is my encoding Move to Py3
27. Questions?