Brett Cannon - How to make your code Python 2/3 compatible

Slide 1

Slide 1 text

How to make your code Python 2/3 compatible Dr. Brett Cannon [email protected] 2015-04-10 @ PyCon

Slide 2

Slide 2 text

This talk is NOT about convincing you to use Python 3 See my PyCon 2013 talk if you need convincing

Slide 3

Slide 3 text

You can start TODAY!!! If you only get one thing out of this talk, let it be this

Slide 4

Slide 4 text

References ● http://python3porting.com ● "What's New" documents for each Python release ● Porting HOWTO: docs.python. org/3/howto/pyporting.html

Slide 5

Slide 5 text

Learn to love six ● Compatibility library to smooth out edges ● Supports Python 2.5 - Python 3 ● Single module for easy vendoring ● https://pypi.python.org/pypi/six

Slide 6

Slide 6 text

Only support Python 2.7 RHEL users can get Python 2.7 through Red Hat Collections

Slide 7

Slide 7 text

Good test coverage is critical ● So you don't accidentally break anything when porting ● coverage.py is handy ○ https://pypi.python.org/pypi/coverage

Slide 8

Slide 8 text

(Basic) new file template # coding: utf-8 from __future__ import (absolute_import, division, print_function, unicode_literals)

Slide 9

Slide 9 text

Transpilers do all the easy stuff Other tools help you to not undo your hard work

Slide 10

Slide 10 text

Modernize ● Harnesses 2to3 to update Python 2 code to work with Python 2.6 - 3 as much as possible ● https://pypi.python.org/pypi/modernize

Slide 11

Slide 11 text

Futurize ● Think Modernize but with more of a Python 3 feel ● Provides backports of things from Python 3 such as the bytes type ● Part of future project: https://pypi.python. org/pypi/future

Slide 12

Slide 12 text

Some fixes require thinking Sorry.

Slide 13

Slide 13 text

Need to care about text vs. binary data Can't conflate the two anymore

Slide 14

Slide 14 text

Need to make API decisions about text vs. binary data unicode/str in Python 2, str/bytes in Python 3

Slide 15

Slide 15 text

Mark all your string literals ● I recommend b prefix + unicode_literals future statement when possible ● u and b prefixes also work ● In the end you should know exactly what type of data a string literal represents ○ Tooling will help enforce this

Slide 16

Slide 16 text

Updating your APIs ● If it's to work with text … ○ Make it work with Unicode ● If it's to work with binary data … ○ Watch out for indexing on bytes ● Be strict with whether you pass in text or binary data, not just str in Python 2 ● Let six help you

Slide 17

Slide 17 text

Text/bytes method uniqueness str ● __mod__ ● encode ● format ● isdecimal ● isnumeric bytes ● decode

Slide 18

Slide 18 text

Python 3.5 improvements ● Bytes interpolation ○ b'I %s bytes' % (b'love',) ● -b will warn when comparing bytes to int ○ Helps with the bytes-indexing issue

Slide 19

Slide 19 text

Division This shouldn't be a surprise; been coming since Python 2.2

Slide 20

Slide 20 text

What to watch out for ● 5 / 2 ○ 2 in Python 2 ○ 2.5 in Python 3 ● Python 3 semantics in Python 2 ○ from __future__ import division ○ -Q flag to interpreter ● Not automatic in case you're using something other than built-in types

Slide 21

Slide 21 text

Pylint ● Can warn against some things not allowed or changed in Python 3 ● use the --py3k flag to run only checks related to Python 3 compatibility

Slide 22

Slide 22 text

Python flags ● -3 ○ Triggers various warnings for things not available in Python 3 ○ Can use -W to control how severe to make the warnings ● -b ○ To help with common bytes-related issues ○ Is a no-op in Python 2, so can blindly use

Slide 23

Slide 23 text

Your code now works in Python 3! Don't forget python2 -3, python3 -b, and Pylint in your testing/CI

Slide 24

Slide 24 text

Dealing with those pesky dependencies Relying on others can be so trying sometimes

Slide 25

Slide 25 text

caniusepython3 ● Checks your (in)direct dependencies to see who is blocking your move to Python 3 ● API for test integration ● Has extra checkers to work with Pylint ● https://caniusepython3.com/ ● https://pypi.python.org/pypi/caniusepython3

Slide 26

Slide 26 text

Getting dependencies ported ● Ask ● Do it yourself ● Hire someone to do it for you

Slide 27

Slide 27 text

Use cffi, Cython, or ctypes for extensions There is also an official HOWTO on porting hand-written extension code

Slide 28

Slide 28 text

Now you can use Python 3! Welcome to the latest version of Python

Slide 29

Slide 29 text

python3 -bb Warns about common mistakes from mixing str and bytes

Slide 30

Slide 30 text

Continuous integration ● Use pylint --py3k to prevent regressions ● Use Tox to run tests under various Python versions ○ https://pypi.python.org/pypi/tox

Slide 31

Slide 31 text

Q&A

Slide 32

Slide 32 text

Bonus slides from my Thumbtack talk; search for [thumbtack brett cannon] for YouTube video

Slide 33

Slide 33 text

Change is good for you! Stuff in Python 2.7 that's different in Python 3.4

Slide 34

Slide 34 text

Fewer built-ins ● apply() ● buffer()* ● coerce() ● cmp() ● execfile() ● file() ● raw_input()* ● xrange()* ● StandardError

Slide 35

Slide 35 text

More iterators ● filter() ● map() ● zip() ● dict.items() et. al.

Slide 36

Slide 36 text

Advancing iterators it.next() next(it)

Slide 37

Slide 37 text

Less syntax, more functions exec 'print `"Hello!"`' exec('print(repr("Hello!"))')

Slide 38

Slide 38 text

New-style classes everywhere class Foo(object): pass class Foo(): pass class Foo: pass

Slide 39

Slide 39 text

Declaring metaclasses class Foo(object): __metaclass__ = type class Foo(object, metaclass=type): pass

Slide 40

Slide 40 text

Parameter unpacking is gone def func(a, (b, c), d): pass

Slide 41

Slide 41 text

Catching exceptions except Exception, exc: ... except Exception as exc: ...

Slide 42

Slide 42 text

Raising exceptions raise Exception, 'uh-oh' raise Exception('uh-oh')

Slide 43

Slide 43 text

Imports from __future__ import absolute_import from ..spam import eggs

Slide 44

Slide 44 text

Octal and binary literals 0720 0o720 0b10101

Slide 45

Slide 45 text

Integer unification ● int went away ● long became int ● L suffix is no more

Slide 46

Slide 46 text

Standard library renamings ● Fixed some bad names ○ ConfigParser -> configparser ● Turned some things into packages ○ httplib -> http.client ○ BaseHTTPServer et. al. -> http.server

Slide 47

Slide 47 text

All of that works in Python 2.6! And you can have it in an automated fashion!

Slide 48

Slide 48 text

Decorate/sort/undecorate sorted(x, cmp=...) sorted(x, key=...)

Slide 49

Slide 49 text

Integer division ● int / int returns a float ● int // int does what Python 2 does ● Get the semantics in Python 2 ○ from __future__ import division ○ -Q new ○ Been around since Python 2.2

Slide 50

Slide 50 text

Text and binary data ● Python 2 ○ Text is basestring: (str, unicode), essentially ○ Binary data is str (bytes is an alias in Python 2.6) ● Python 3 ○ Text is str (similar to unicode in Python 2) ○ Binary data is bytes (sort of similar to str in Python 2) ○ To see differences, try set(dir(str)). difference(dir(bytes))

Slide 51

Slide 51 text

All of that is still available in Python 2.6! It just takes some effort to have

Slide 52

Slide 52 text

New features! In Python 3.4 that you can't have in Python 2.6

Slide 53

Slide 53 text

Set literals x = {1, 2, 3, 4}

Slide 54

Slide 54 text

Set & dict comprehensions {x**x for x in range(10)} {x: x**x for x in range(10)}

Slide 55

Slide 55 text

All of that is in Python 2.7! Everything from now on is exclusive to Python 3, I promise

Slide 56

Slide 56 text

Unicode everywhere ● Source code is UTF-8 encoded by default ● Based on the Unicode standard annex UAX- 31 with some tweaks

Slide 57

Slide 57 text

__pycache__ ● All .pyc and .pyo files are put in a __pycache__ subdirectory ● All bytecode files are tagged per interpreter to prevent overwriting when using a different Python version

Slide 58

Slide 58 text

Extended iterable unpacking a, *b, c = range(10) a == 0 b == list(range(1, 9)) c == 9

Slide 59

Slide 59 text

Enhanced exceptions ● Chaining connects causal chain of exceptions ○ Implicit from simply raising another exception while another is active ○ Explicit with raise exc2 from exc1 ● Traceback now embedded in exception

Slide 60

Slide 60 text

Keyword-only arguments def func(a,*, are_you_sure): pass

Slide 61

Slide 61 text

Function annotations def func(a:int) -> float: pass

Slide 62

Slide 62 text

nonlocal def outer(): x = 0 def inner(): nonlocal x x += 1 return x, inner

Slide 63

Slide 63 text

super() class Foo(bar): def __init__(self): super().__init__()

Slide 64

Slide 64 text

Stable ABI ● Hides interpreter details ● Guaranteed not to change ● Define Py_LIMITED_API and your extension module won't require recompilation per Python version

Slide 65

Slide 65 text

yield from for x in range(10): yield x yield from range(10)

Slide 66

Slide 66 text

Significant stdlib additions ● ssl.SSLContext ● asyncio ● tracemalloc

Slide 67

Slide 67 text

pip & venv ● pip is now installed by default ● Virtual environments created by venv install pip by default ● Plans to have platform installers install pip in a future Python 2.7 release

Slide 68

Slide 68 text

Performance ● decimal implemented in C ● Integer math faster ● More efficient string memory use ● Key-sharing dictionaries ● Custom memory allocators ● Interchangeable hash algorithm

Slide 69

Slide 69 text

Looking to the future Preview of Python 3.5

Slide 70

Slide 70 text

Matrix multiplication x @ y x @= y

Slide 71

Slide 71 text

% formatting for bytes ● Supported subset of what % does for strings ● Makes constructing ASCII-based binary data easier ● Will help binary-manipulating Python 2 code also work in Python 3