PyConZA 2015: "Supporting Python 3" by Neil Muller

The tools for supporting Python 2 & 3 in a single code base have improved significantly from the early Python 3 days of just 2to3. Porting Python 2 code to work with Python 2 & 3 without a constant 2to3 translation step is not hard anymore, and leads to much easier-to-maintain ports.

In this talk, I will discuss some of the suggested best practices for supporting both Python 2 and Python 3. We will cover some of the tools for detecting potential issues, such as pylint, the available options, such as six, futurize and modernize, to simplify the process of updating the code and spend some time discussing what parts of porting still require care and attention, such as the text / binary data division.

Transcript

1. SUPPORTING PYTHON SUPPORTING PYTHON 3 3 HOW TO MAKE YOUR CODE HOW TO MAKE YOUR CODE COMPATIBLE WITH PYTHON 2 & 3 COMPATIBLE WITH PYTHON 2 & 3 PYCON ZA 2015 — 2ND OCTOBER 2015 PYCON ZA 2015 — 2ND OCTOBER 2015 by Neil Muller Supporting Python 3 1 of 23
2. THE OLD ADVICE THE OLD ADVICE Heavily stressed the approach of using 2to3 to covert the code Recommended running 2to3 at install time to handle the conversion When a user installs your project for Python 3, you can have either distutils or Distribute run 2to3 on your behalf. Supporting Python 3 2 of 23
3. This approach has some significant downsides Need to add conversion step and testing on Python 3 to the development process Conversion introduces a disconnect between the code written and the code actually being run, which makes debugging harder Works OK if you're completely abandoning Python 2 support, though. Supporting Python 3 3 of 23
4. CURRENT WISDOM CURRENT WISDOM The tools for supporting both Python 2 & 3 are now mature — Use them six is awesome Use 2to3 and related tools to identify and fix the bits aren't Python 3 compatible modernize or futurize depending on taste Supporting Python 3 4 of 23
5. WHAT TO DO? WHAT TO DO? Ensure you have good test coverage Drop support for Python 2.5 six can help if you really need to support this But the last Python 2.5 release was December 2008 – Do you really still need to support it? Somewhat easier if you can only support Python 2.7 onwards, but Python 2.6 onwards is very doable Supporting Python 3 5 of 23
6. Cleanup python -3 warnings Makes life simpler for other tools Make liberal use of __future__ imports print_function, absolute_imports, unicode_literals, division all good ideas Good test coverage really is the most useful bit of advice. Supporting Python 3 6 of 23
7. Use transpilers to do the mechanical work Things like module renames, import fixes and attribute changes can largely be handled automatically May need to fiddle with options to get good results, but it is a once off process Use other tools and tests to ensure you don't break things in the future Supporting Python 3 7 of 23
8. MODERNIZE MODERNIZE https://pypi.python.org/pypi/modernize Uses lib2to3 and six to identify and fix code that's not Python 2/3 compatible By default aims to produce code that supports Python 2.6, Python 2.7 and Python 3.3 onwards Can produce code without a dependancy on six if asked, but generally give best results when using six Supporting Python 3 8 of 23
9. FUTURIZE FUTURIZE http://python-future.org/ Think modernize with a Python 3 feel Part of the future project This provides extra backports of Python 3 features to Python 2 Introduces a heavier dependancy than modernize, but result closer to Python 3 code Supporting Python 3 9 of 23
10. EXAMPLE EXAMPLE Conversion isn't perfect But the number of manual fixes needed is greatly reduced Supporting Python 3 10 of 23
11. OTHER TOOLS OTHER TOOLS python2 -3 to can help identify possible problems Running 2to3 can also be useful for identifying issues Pylint supports a bunch of python 3 checks pylint --py3k runs only these checks (supported in pylint 1.4) Supporting Python 3 11 of 23
12. THE HARD STUFF THE HARD STUFF Some bits still require care and thought The text / bytes transition shouldn't be handled automatically Remember the advice about test coverage? Need to make API decisions about text versus bytes This actually is a really useful exercise to go through. Explicit is better than implicit. Supporting Python 3 12 of 23
13. KNOW YOUR DATA KNOW YOUR DATA Use str / unicode in Python 2, bytes / str in Python 3 Marking string literals is a good idea b'..' prefix and unicode_literals get you most of the way there u'..' and b'..' prefixes also work Ultimately, should be obvious whether for every string whether its either text or binary data Supporting Python 3 13 of 23
14. UPDATE YOUR APIS UPDATE YOUR APIS Is it text data? It should work with unicode Is it binary data? It should work with bytes Be strict about the distinction Avoid using str everywhere in Python 2 When talking to the outside world, convert as close to the boundary as possible Supporting Python 3 14 of 23
15. COMMON GOTCHAS COMMON GOTCHAS Indexing a bytes object return integer, not bytes >>> b'xyz'[0] 120 Use slicing instead >>> b'xyz'[0:1] b'x' Supporting Python 3 15 of 23
16. Comparison between bytes and strings has one surprise Python 2 will coerce types happily, python3 will generally raise TypeError >>> b'' < '' ... TypeError: unorderable types: bytes() < str() Except when it doesn't >>> b'' == '' False Language requires that == always returns a result Supporting Python 3 16 of 23
17. Converting from __unicode__ / __str__ methods to just __str__ shouldn't be done blindly class Thing(object): def __init__(self): self.data = u'abc' def __unicode__(self): return self.data def __str__(self): return unicode(self).encode('utf8') Supporting Python 3 17 of 23
18. Simplistic port's can do the wrong thing Running 2to3 from Python 3.4 gives the following class Thing(object): def __init__(self): self.data = 'abc' def __unicode__(self): return self.data def __str__(self): return str(self).encode('utf8') What happens when you call str(Thing())? Leads to wierd bugs when carelessly converted. Supporting Python 3 18 of 23
19. USE THE AVAILABLE TOOLS TO HELP USE THE AVAILABLE TOOLS TO HELP YOU YOU six has some very nice helpers, especially for things like the __str__ / __unicode__ handling python3 -b will warn about some common bytes errors Warns about comparisons and indexing errors and some other things python3 -bb turns warnings into errors python2 -b is a no-op, so you can blindly use the flag Supporting Python 3 19 of 23
20. PORTING C EXTENSIONS PORTING C EXTENSIONS Bytes / Unicode distinction must be addressed Aren't that many other significant differences in the C API Porting is mainly just tedious tweaking of code to handle both Python 2 & 3 APIs Unfortunately, there aren't great tools to help with this Good documentation available Porting Extension Modules to Python 3 Supporting Python 3 20 of 23
21. DROP HAND-ROLLED EXTENSION DROP HAND-ROLLED EXTENSION MODULES MODULES Tools like cffi, ctypes or Cython are a lot easier to manage Boost.Python supports Python 3 since Boost 1.40 Swig supports Python 3 since Swig 1.3, although 2.0 or later is recommended Supporting Python 3 21 of 23
22. Cython produces code that compiles for both Python 2 and 3 ctypes and cffi work on both python 2 & 3 Still need to be aware of the bytes / str coding issues - char* is handled differently between Python 2 and 3 Have a look at cffi, for it is really neat Also gives you support for PyPy Supporting Python 3 22 of 23
23. IN CONCLUSION IN CONCLUSION Supporting Python 2 & 3 in a single code-base not hard anymore Lots of good documentation available docs.python.org Python 3 Porting Howto Supporting Python 3: An in-depth guide Numerous others Python 3 is the future, so go forth and port code. It's eductional & fun Supporting Python 3 23 of 23