Opening The Floodgates: Unicode Identifiers in Python

Python 3 introduced unicode identifiers into the language. This talk discusses what this means, what restrictions are included and shows what can happen when you drop those restrictions

Transcript

1. Opening The Floodgates Unicode Identifiers In Python
2. “Hey Matt, look what Swift can do”
3. Python can’t do do it Well… Python won’t do it.
4. Text Representation The Treachery Of Images by René Magritte
5. Encoding failure
6. Encode throws DecodeError?! ლ(ಠ益ಠლ)
8. Unicode • Standard to define how characters should be represented in computers • Other standards (ascii, latin-1, euc-jp) are culturally specific ◦ Inappropriate for a global internet • Over one million characters allowing all known characters to be represented • There is even room for Emoji!
9. Unicode • A unicode string is a collection of code points from 0 to 1,114,112 • These codepoints correspond to characters ◦ U+0045 -> E ◦ U+1F60A -> • To represent these in the machine an encoding is used.
10. Encoding 101 • UTF-8 is one way of encoding unicode • Each code point is represented with one or more bytes ◦ E -> U+0045 -> 01000101 ◦ -> U+1F60A -> 11110000 10011111 10011000 10001010
11. Why Should you care • When you leave the pretty world that is your program you are in the real world. The real world is in bytes ◦ Databases ◦ Console ◦ HTTP • '(\xe2\x95\xaf\xc2\xb0\xe2\x96\xa1\xc2\xb0\xef\xbc\x89\xe2\x95\xa f\xef\xb8\xb5 \xe2\x94\xbb\xe2\x94\x81\xe2\x94\xbb' ◦ Understanding that these bytes are encoded in UTF-8 is the difference between '(╯°□°）╯︵ ┻━┻' and a UnicodeDecodeError
12. So about Encode throwing DecodeError • “¥” is a python 2 string • u”¥” is a unicode object • encode must be called on unicode objects ◦ Encode into bytes ◦ Decode into unicode • Call encode on a string Python tries to “help” by decoding it into unicode first • And it explodes.
13. Identifiers in Python • Python Identifiers pre Python 3 only supported basic latin letters, digits, and underscore • Pep 3131 expands this definition to include Unicode
14. A Mixed Blessing • Inclusiveness ◦ Not everyone uses latin alphabets • Complexity ◦ Imagine using a library with a unicode heavy API ▪ γάμμα.ɸ() ◦ Harder to read source code ◦ Security concerns
15. So why can’t I program with poop already?! • Python3 identifiers only allow a subset of Unicode • Specifically the follow the suggestions of The Unicode Consortium ◦ Identifier: XID_START XID_CONTINUE* • XID_START ◦ Basically “This is a letter” • XID_CONTINUE ◦ Everything in XID_START and modifier characters
16. Continuation mark demonstrated Combining Grave Accent
18. Long Story Short Emoji is not considered the start or continuation of a word
19. Swift Follows These Suggestions (mostly) • Exception made for emoji (for some reason) • Gives better error ◦ Python says the accent is invalid, but technically it just is not a valid start ◦ I wrote a patch (http://bugs.python.org/issue23263)
20. OK! WE KNOW EVERYTHING WE NEED TO KNOW!
21. Obviously, I need to fork the interpreter
22. Ok… fine
23. Victory! https://github.com/Bachmann1234/python-emoji
24. That was a lot of… stuff Ned Batchelder: Pragmatic Unicode Travis Fischer, Esther Nam: Character encoding and Unicode in Python
25. Matt Bachmann @MattBachmann https://github.com/Bachmann1234