By Your Bootstraps: Porting your App to Python3 by Tres Seaver

Transcript

1. By Your Bootstraps: Porting your Application to Python3 PyCon 2014, Montréal Tres Seaver @tresseaver tseaver@agendaless.com
2. Objectives “Straddling” Python 2 / 3 in single codebase
3. Objectives “Straddling” Python 2 / 3 in single codebase Choosing target Python versions
4. Objectives “Straddling” Python 2 / 3 in single codebase Choosing target Python versions Porting as an iterative process Ordering components by dependencies
5. Objectives “Straddling” Python 2 / 3 in single codebase Choosing target Python versions Porting as an iterative process Ordering components by dependencies Adding test coverage to reduce risk
6. Objectives “Straddling” Python 2 / 3 in single codebase Choosing target Python versions Porting as an iterative process Ordering components by dependencies Adding test coverage to reduce risk Covering C extensions
7. Objectives “Straddling” Python 2 / 3 in single codebase Choosing target Python versions Porting as an iterative process Ordering components by dependencies Adding test coverage to reduce risk Covering C extensions Porting Hygeine
8. Background Experience report / best practices summary Based on porting ~180 kLOC Python, ~25 kLOC c codebase Zope3 component architecture ZODB WebOb Pyramid Other dependencies
9. Porting Strategies Port once, abandon Python2 Not the subject of this talk Customers / users still need Python2 More feasible for applications than libraries 2to3 may be useful starting point
10. Porting Strategies Port once, abandon Python2 Not the subject of this talk Customers / users still need Python2 More feasible for applications than libraries 2to3 may be useful starting point
11. Porting Strategies “Fix up” at installation using 2to3 Python2 users unaffected Python3 source “drifts” from canonical version Bug reports don't match 2to3 painfully slow on large codebases
12. Porting Strategies “Fix up” at installation using 2to3 Python2 users unaffected Python3 source “drifts” from canonical version Bug reports don't match 2to3 painfully slow on large codebases
13. Porting Strategies “Straddling” in a single codebase Use “compatible subset” of Python syntax Conditional imports mask stdlib changes six module can help (but you might not need it)
14. Porting Strategies “Straddling” in a single codebase WIN! Use “compatible subset” of Python syntax Conditional imports mask stdlib changes six module can help (but you might not need it)
15. Targeting Python Versions Syntax changes make Python2 < 2.6 hard No b'' literals No except Exception as e:
16. Targeting Python Versions Syntax changes make Python2 < 2.6 hard No b'' literals No except Exception as e: Much more cruft / pain
17. Targeting Python Versions Syntax changes make Python2 < 2.6 hard No b'' literals No except Exception as e: Much more cruft / pain 2.4 / 2.5 are long past EOL But some folks need system Python in “enterprisey” systems
18. Targeting Python Versions Incompatibilities make Python3 < 3.2 hard PEP 3333 fixes WSGI in Py3k callable() restored in 3.2 3.3 restores u'' literals 3.2 is “system Python3” on some LTS systems
19. Targeting Python Versions Summary: 2.6, 2.7, 3.2+
20. Targeting Python Versions Summary: 2.6, 2.7, 3.2+ Maybe 2.7, 3.3+
21. Managing Porting Risks Ports are great opportunities for bug injection
22. Managing Porting Risks Ports are great opportunities for bug injection Fear of breaking working software is the barrier Even more than effort
23. Managing Porting Risks Some mitigations also improve your software Improved testing Modernized idioms in Python2 Clarity in text vs. bytes
24. Bottom-Up Porting Port packages with no dependencies first
25. Bottom-Up Porting Port packages with no dependencies first Then port packages with already-ported dependencies Note Python versions supported by dependencies
26. Bottom-Up Porting Port packages with no dependencies first Then port packages with already-ported dependencies Note Python versions supported by dependencies Lather, rinse, repeat...
27. Bottom-Up Porting Port packages with no dependencies first Then port packages with already-ported dependencies Note Python versions supported by dependencies Lather, rinse, repeat... Finally, port the application
28. “Common subset” idioms Lennart Regebro's book, http://python3porting.com/noconv.html
29. “Common subset” idioms Lennart Regebro's book, http://python3porting.com/noconv.html python2.7 -3 can point out problem areas
30. “Common subset” idioms Lennart Regebro's book, http://python3porting.com/noconv.html python2.7 -3 can point out problem areas “Modernize” idioms in Python2 code
31. “Common subset” idioms Lennart Regebro's book, http://python3porting.com/noconv.html python2.7 -3 can point out problem areas “Modernize” idioms in Python2 code Distinguish bytes vs. text Use b'' / u'' for all literals
32. “Common subset” idioms Adopt new syntax E.g., except … as … print()(function-like)
33. “Common subset” idioms Adopt new syntax E.g., except … as … print()(function-like) Use new stdlib facilities E.g., io.BytesIO vs StringIO.StringIO
34. Testing Avoids Hair-Loss Untested code is where the bugs go to hide
35. Testing Avoids Hair-Loss Untested code is where the bugs go to hide 100% coverage is ideal before porting Unit testing preferable for libraries Functional testing best for applications
36. Testing Avoids Hair-Loss Untested code is where the bugs go to hide 100% coverage is ideal before porting Unit testing preferable for libraries Functional testing best for applications Measure coverage: https://pypi.python.org/pypi/coverage
37. Testing Avoids Hair-Loss Work to improve assertions as well as coverage Assert contracts, not implementation details
38. Testing Avoids Hair-Loss Doctests make straddling really hard Replace doctests with unit / functional tests If at all feasible, convert doctests to Sphinx examples
39. Testing Avoids Hair-Loss
40. Testing Avoids Hair-Loss Automate running tests tox helps ensure that tests pass under all supported versions Testing with pypy! Example
41. Considerations for C Extensions http://python3porting.com/cextensions.html Testing C is harder!
42. Considerations for C Extensions Python reference implementation Easier to test Supports PyPy
43. Considerations for C Extensions Python reference implementation Easier to test Supports PyPy Design for same API as C
44. Considerations for C Extensions Python reference implementation Easier to test Supports PyPy Design for same API as C 100% coverage for Python
45. Considerations for C Extensions Python reference implementation Easier to test Supports PyPy Design for same API as C 100% coverage for Python Ensure C version passes same tests
46. Considerations for C Extensions Example: persistent
47. Hygeine Issues Signal supported versions using Trove classifiers
48. Hygeine Issues Signal supported versions using Trove classifiers Consider bumping major version Allow users to stick with “safe” versions as you iterates
49. Hygeine Issues Signal supported versions using Trove classifiers Consider bumping major version Allow users to stick with “safe” versions as you iterate Apply continuous integration Travis-CI Jenkins Shining Panda for Windows
50. Resources http://python3porting.com/ https://testrun.org/tox/latest/ http://pypi.python.org/pypi/six
51. Q & A @tresseaver