The str/bytes nightmare before python2 EOL

Still haunted by Python 2's str vs. bytes nightmare? This talk dissects the root cause and shows the way to handle them in Python 2 and 3.

Transcript

1. The str/bytes nightmare before python2 EOL Kir Chou A9 (Amazon Product Search) 1
2. ! Confusing Terms ! • Str: str() of python • Bytes: bytes() of python • Text: unicode() in Python2 or str() in Python3 • String: Text ∪ Bytes 2 You just need to know that those terms are different in this talk.
4. Google Trend 4 REF The frequency of encoding ~= iterator or variable scope.
5. Objective 5
6. For any level pythonist, nothing new in this talk 6 What is it? Why is the design? Treatment? D-K effect curve
7. Outline 1. (Covered) Talk Objective 2. Motivation 3. Python String 101 – What is string? 4. Python String 101 – Why is this design? 5. 5 Treatments 7
8. Motivation 8
9. Python2 End-Of-Life https://pythonclock.org/ 9
10. Optimistic Numbers 10 By Jetbrains 2018 survey ∵ The number of python2 user ↓ ∴ The number of new package will be written in python3
11. 11 Pessimistic Facts legacy dependencies
13. 13 Migration – Pain Points painful easy
14. Python String 101 text vs bytes 14 (unicode)
15. 15 Text: How to present info in memory? ℙ ƴ ℌ ø ἤ e2 84 99 c6 b4 e2 98 82 e2 84 8c c3 b8 e1 bc a4
16. 16 ℙ ƴ ℌ ø ἤ e2 84 99 c6 b4 e2 98 82 e2 84 8c c3 b8 e1 bc a4 Bytes: How to store info into memory?
17. Python3 >>> 'python' 'python' >>> 'ℙƴ ℌøἤ' 'ℙƴ ℌøἤ' 17 Default: text
18. Python3 >>> b'python' b'python’ >>> 'python'.encode(encoding='ascii') b'python ' >>> b'ℙƴ ℌøἤ' SyntaxError: bytes can only contain ASCII literal characters. >>> 'ℙƴ ℌøἤ'.encode(encoding='utf-8') b'\xe2\x84\x99\xc6\xb4\xe2\x98\x82\xe2\x84\x8c\xc3\ xb8\xe1\xbc\xa4' 18 b' ' : ascii encode (0~127) text.encode(encoding)
19. Python2 >>> 'python' 'python' >>> 'ℙƴ ℌøἤ' '\xe2\x84\x99\xc6\xb4\xe2\x98\x82\xe2\x84\x8c\x c3\xb8\xe1\xbc\xa4' 19 Default: auto-encoded bytes
20. Python2 >>> u'ℙƴ ℌøἤ' u'\u2119\u01b4\u2602\u210c\xf8\u1f24' 20 u' ': text
21. Python String 101 encode vs decode 21 Text -> Bytes encode(encoding) Bytes -> Text decode(encoding)
22. Python3 >>> dir(str()) […, 'encode', …] >>> dir(bytes()) […, 'decode', …] 22 Text -> Bytes encode(encoding) Bytes -> Text decode(encoding)
23. Python2 >>> dir(str()) […, 'decode', 'encode’, …] >>> dir(bytes()) […, 'decode', 'encode’, …] 23
24. Python2 >>> dir(str()) […, 'decode', 'encode', …] >>> dir(bytes()) […, 'decode', 'encode', …] 24
25. Python String 101 Python2 Design Philosophy 25
27. Python2 cares ascii encoding at first, BUT… • Most of encoding supports one way: utf-8, latin-1 • But there are few exceptions: base64, rot13 >>> 'python'.encode('rot13') 'clguba' >>> 'python'.decode('rot13') u'clguba' 27
28. Python String 101 Fundamental Difference - Encoding 28
29. ASCII encoding behave similar, BUT… >>> b'python' == u'python' False 29 >>> b'python' == u'python' True 3 2
30. You must be told, or you have to guess By Ned Batchelder 30
31. Treatment 1 Knowing the Fact of String 31
32. The Fact of String By Ned Batchelder • Python3: every string are unicode • Encoding needs to be handled manually • One-way encode/decode behind str/bytes (Good) • Python2: every string are auto encoded to bytes • Ascii is consistently handled • Two-way encode/decode behind str/bytes (Broken after python is broadly used in many different human languages.) 32
33. Treatment 2 Explicit Declaration (From community approach to supporting python3) 33
34. Consistent IO • Standard I/O (bytes) • Python2: sys.stdin / sys.stdout • Python3: sys.buffer.stdin / sys.buffer.stdout • File IO import io # consistent api in both versions io.open('path/to/file', 'wt') # text, bytes 34
35. Bytes or Text (with Encoding) • u'' or b''? • No more raw string '' • Which encoding is used for the text? • No more guess, always provide encoding: latin-1, utf-8… 35 def my_encrypt(text, encoding='utf-8'): … 3 2
36. 36 >>> # -*- coding: utf-8 -*- sys.getdefaultencoding() Encoding of your Operating System 3 2
37. Treatments 3 Unicode Sandwich (From community approach to supporting python3) 37
38. Unicode Sandwich Decode ASAP, Encode ALAP 38 Don’t care about encoding!
39. Treatments 4 (minor) Python2/3 compatibility (From community approach to supporting python3) 39
40. 40 >>> from __future__ import unicode_literals Add unicode_literals after porting to python3 2
41. Treatments 5 Typing Hints (Learn from dropbox migration notes) 41
42. Typing Hints [mypy] [pyre] def encrypt(data, key): return cipher 42 def encrypt(data: bytes, key: bytes) -> bytes: return cipher def encrypt(data, key): # (bytes, bytes) -> bytes return cipher 3 3 2 2
43. Conclusion 43
44. 44 Q: Why supporting python3 is hard? A: ✋"#
45. Take-Home Messages Write text/bytes explicitly with typing hints Always provide encoding for bytes Apply Unicode Sandwich if possible Copy encode/decode from StackOverflow 45 Python String is no longer a nightmare! !
46. Homework (Verify your understanding) 1. Search “UnicodeEncodeError” in StackOverflow 2. Randomly pick one solved question 3. Read the content 4. Explain to yourself what happened and why is it 46
47. Q&A • String • SupportingPython3 47
48. Reference: Docs • Python2 unicode • Python3 unicode • pyporting • python-future.org/compatible_idioms.pdf • 2018 Jetbrains Survey • Dropbox Migration Notes 48
49. Reference: Talks • Ned Batchelder: Pragmatic Unicode, or, How do I stop the pain? • Guido van Rossum: BDFL Python 3 retrospective • Brett Cannon - How to make your code Python 2/3 compatible - PyCon 2015 • Writing Python 2/3 compatible code by Edward Schofield • Brandon Rhodes - Oh, Come On Who Needs Bytearrays 49