Making Acquaintances with Exotic Encodings

Text encoding is a relatively straightforward feature of Python at first glance, but in the wild there are a lot of strange cases we can encounter.

Transcript

1. Making Acquaintances with Exotic Encodings Trapsilo Bumi - HENNGE PyCon Australia 2019
2. About Me Trapsilo Pramudya Bumi From Bandung, Indonesia Software Engineer at HENNGE K.K. Shibuya, Tokyo, Japan B2B cloud security solutions hennge.com/global/about
3. Awesome internship program! - Also looking for mid-careers too hennge.com/global/gip
4. What is Encoding? Let’s Revisit
8. US-ASCII was one of the first successful standardized “mappings” Different standards were created by different people around the world Many were extending ASCII in different ways This created “bilingual encodings”, but not “multilingual” The Unicode Standard was created to unify them
9. Unicode concept: code-points Assigns a number, not a glyph, to a “character” or “grapheme” Leaves the visual rendering to software Independent of the code unit (bit sequence)
10. Character Glyph Unicode Code Point Code Units in UTF-8 Code Units in UTF-32 LATIN SMALL LETTER A a a U+0061 0x61 0x00000061 LATIN SMALL LETTER SHARP S ß U+00DF 0xDF 0x000000DF HAN FOR EAST 東 U+6771 0x67 0x71 0x00006771 DESERET CAPITAL LETTER LONG I U+10400 0xF0 0x90 0x90 0x80 0x00010400
11. Encoding in Python Text Sequence a.k.a “Unicode strings” Binary Sequence a.k.a “Byte strings” Sequence of Unicode code points Sequence of single bytes
12. Encoding in Python (2) Converting one to another: decode() and encode() bytes.decode(encoding, errors) -> str str.encode(encoding, errors) -> bytes First parameter: encoding (how to translate from code point to bytes/reverse) Second parameter: errors (how to handle unrecognized code points)
13. Unicode in the Wild
14. Who in the world doesn’t use Unicode?
15. Source: https://googleblog.blogspot.com/2012/02/unicode-over-60-percent-of-web.html
16. UnicodeEncodeError: 'latin-1' codec can't encode character '\u1234' in position 3: ordinal not in range(256)
17. Non-Unicode text as the standard: Emails RFC 821 (SMTP protocol) dictates the use of ASCII Using non-ASCII characters requires SMTP extensions/MIME (RFC 5321) Many email clients don’t provide Unicode as the default
18. Latin-1 a.k.a ISO-8859-1 a.k.a CP819 The basis for most popular 8-bit character sets The default encoding for documents that are delivered via HTTP MIME type “text/*”
19. Windows-1252 a.k.a CP-1252 Used by default in the legacy components for Windows Windows → Microsoft Office → popular!
20. GB18030 The official character set for the People’s Republic of China “Unicode Transformation Format” Superseding GB2312
21. Big5 A popular encoding in Taiwan, Hong Kong, and Macau 1980s standard, before Unicode China used this before creating the GB* standard Extended by lots of organizations E.g. Microsoft Code Page 950
22. Shift JIS Developed by “ASCII Corporation” and Microsoft One of the prominent Japanese character encodings
23. Extended Unix Code (EUC) CJK characters unite! Three variants: EUC-JP EUC-CN EUC-TW
24. Handling (non-Unicode) Text in Python
25. The codecs package Simply, using encode() and decode() to switch between bytes ⇔ str Defaults to UTF-8, can specify alternative encoding errors parameter specifies how to handle unrecognized code points: ignore - skip the code point strict - raise an Exception replace - replace the unknown code point with something, e.g. �
26. But what if... We don’t know the encoding for this text? There is a specified encoding, but it doesn’t match/work/spits out mojibake? There is just no way to specify the encoding in that document/file/whatever? It happens a lot: Scraping websites Reading/parsing email
27. Enter… chardet Character encoding auto-detection library How it works: heuristics to map common patterns to a specific language Based on universalchardet by Mozilla
28. So, is chardet the answer to everything? No, use it as a last resort when: It’s HTTP and there is no charset parameter in the Content-Type header It’s HTML and there is no <meta http-equiv="content-type"> It’s XML and there is no encoding attribute in the prolog It’s something else and there is no explicit character encoding
29. chardet is not perfect It can spit out EUC-TW: not in Python’s standard library! It has a confidence rating that is sometimes very low Set a threshold according to your tolerance
30. The last, last resort Using a static table that is based on experience: PYCODECNAMES = { 'cp943' : 'cp932', 'cp943c' : 'cp932', 'cswindows31j': 'cp932', 'ibm-943' : 'cp932', 'ibm-943c' : 'cp932', 'ja16sjis' : 'cp932', 'macjapan' : 'cp932', }
31. Especially for MIME messages, encodings aren’t always written one way. { 'cp65001' : 'utf_8', 'en_gb' : 'utf_8', 'utf-8-hfs' : 'utf_8', 'utf-8-mac' : 'utf_8', 'utf8-docomo' : 'utf_8', 'utf8-kddi' : 'utf_8', 'utf8-mac' : 'utf_8', 'utf8-softbank' : 'utf_8', }
32. Summary
33. Encoding is important to understand Useful for text processing Keep some tricks up the sleeve for handling exotic character mappings chardet package Static list of aliases
34. Thank you!