Preparing for the future - Python 3 is here to stay

A look at the history and current adoption of Python 2 and 3, major differences between the two version, hurdles to porting and finally a set of tips and recommendations for writing code to run on both Python 2 and 3.

This talk was presented at the Amsterdam Python meetup group on 2015-03-24 (http://www.meetup.com/Amsterdam-Python-Meetup-Group/events/220704819/).

Transcript

1. Preparing for the future
   Python 3 is here to stay
   #py020
   

   View Slide

2. Hi, I'm Nick
   https://nick.groenen.me/about/
   

   View Slide

3. Are you using Python 3 yet?
   

   View Slide

4. ➔ Python 3.0 final was released on December 3rd, 2008.
   ➔ The final 2.x version (2.7) release came out mid-2010
   ➔ Python 2.7 support extended by 5 years, to 2020
   A little bit of history
   

   View Slide

5. Today
   Django
   Flask
   NumPy
   SciPy
   Boto
   Pandas
   Tornado
   Celery
   SQLAlchemy
   PyPy
   

   View Slide

6. Tomorrow
   ➔ OpenStack
   ➔ Debian
   ➔ Ubuntu
   ➔ Fedora
   

   View Slide

7. Stragglers
   ➔ Twisted
   ➔ gevent
   ➔ Jython
   ➔ IronPython
   

   View Slide

8. Like it or not
   The future is Python 3000
   

   View Slide

9. What's so different about
   Python 3?
   

   View Slide

10. ➔ Cleans up some warts in the language
    ➔ More readable, consistent & explicit
    Intentionally backward-incompatible
    

    View Slide

11. The unicode "issue"
    

    View Slide

12. PEP 3100
    Miscellaneous Python 3.0 Plans
    

    View Slide

13. Python 2
    Unicode vs. 8-bit
    Python 3
    Text vs. data
    

    View Slide

14. Python 2
    Unicode vs. 8-bit
    ASCII str type
    Separate unicode type
    No byte type
    Python 3
    Text vs. data
    str is unicode text
    byte and bytearray for
    encoded text
    

    View Slide

15. Python 2 allows mixing of
    str and unicode objects
    >>> u'This is unicode.' + " This isn't"
    u"This is unicode. This isn't"
    

    View Slide

16. Python 3 does not
    >>> "This is unicode." + b' These are bytes.'
    TypeError: Can't convert 'bytes' object to str implicitly
    >>> "This is unicode." + b' These are bytes.'.decode('us-ascii')
    'This is unicode. These are bytes.'
    

    View Slide

17. Renames
    cookielib
    Cookie
    htmlentitydefs
    HTMLParser
    httplib
    BaseHTTPServer
    CGIHTTPServer
    SimpleHTTPServer
    pipes.quote
    SocketServer
    UserDict.UserDict
    UserList.UserList
    UserString.UserString
    xmlrpclib
    SimpleXMLRPCServer
    http.cookiejar
    http.cookies
    html.entities
    html.parser
    http.client
    http.server
    http.server
    http.server
    shlex.quote
    socketserver
    collections.UserDict
    collections.UserList
    collections.UserString
    xmlrpc.client
    xmlrpc.server
    

    View Slide

18. Print is a function
    (as are a few others)
    

    View Slide

19. Generators
    ➔ zip()
    ➔ map()
    ➔ filter()
    ➔ dictionary's .keys() method
    ➔ dictionary's .values() method
    ➔ dictionary's .items() method
    

    View Slide

20. Python 2
    3/2 = 1
    3.0/2.0 = 1.5
    Integer division
    

    View Slide

21. Python 2
    3/2 = 1
    3.0/2.0 = 1.5
    Integer division
    Python 3
    3/2 = 1.5
    3.0/2.0 = 1.5
    3//2 = 1
    

    View Slide

22. Comparing unorderable types
    [1, 2] > 'foo' = False
    (1, 2) > 'foo' = True
    [1, 2] > (1, 2) = False
    

    View Slide

23. Comparing unorderable types
    [1, 2] > 'foo' = False
    (1, 2) > 'foo' = True
    [1, 2] > (1, 2) = False
    TypeError: unorderable types: list() > str()
    

    View Slide

24. Improved variable unpacking
    a, b, c = range(3)
    a = 0
    b = 1
    c = 2
    

    View Slide

25. Improved variable unpacking
    a, b, c = range(3)
    a = 0
    b = 1
    c = 2
    a, *rest, b = range(10)
    a = 0
    rest = [2, 3, 4, 5, 6, 7, 8, 9]
    b = 9
    

    View Slide

26. Keyword-only arguments
    >>> def f(a, b, *args, option=True):
    ... print(repr(args))
    ... print(repr(option))
    ...
    

    View Slide

27. Keyword-only arguments
    >>> def f(a, b, *args, option=True):
    ... print(repr(args))
    ... print(repr(option))
    ...
    >>> f(1, 2, False)
    (False,)
    True
    

    View Slide

28. Keyword-only arguments
    >>> def f(a, b, *args, option=True):
    ... print(repr(args))
    ... print(repr(option))
    ...
    >>> f(1, 2, False)
    (False,)
    True
    >>> f(1, 2, option=False)
    ()
    False
    

    View Slide

29. Keyword-only arguments
    >>> def f(a, b, *, option=True):
    ... print(repr(option))
    ...
    

    View Slide

30. Keyword-only arguments
    >>> def f(a, b, *, option=True):
    ... print(repr(option))
    ...
    >>> f(1, 2, 3)
    Traceback (most recent call last):
    File "", line 1, in
    TypeError: f() takes 2 positional arguments but 3 were given
    

    View Slide

31. Keyword-only arguments
    >>> def f(a, b, *, option=True):
    ... print(repr(option))
    ...
    >>> f(1, 2, 3)
    Traceback (most recent call last):
    File "", line 1, in
    TypeError: f() takes 2 positional arguments but 3 were given
    >>> f(1, 2, option=False)
    False
    

    View Slide

32. PEP 3134
    Exception Chaining and Embedded Tracebacks
    

    View Slide

33. Chained exceptions
    >>> import socket
    >>>
    >>> class IPLookupError(Exception):
    ... """Unable to resolve hostname to IP address"""
    ...
    >>>
    >>> def hostname_to_ip(hostname):
    ... try:
    ... return socket.gethostbyname(hostname)
    ... except socket.gaierror:
    ... raise IPLookupError("Unable to resolve '%s'" % hostname)
    ...
    

    View Slide

34. Chained exceptions
    >>> print(hostname_to_ip("nick.groenen.me"))
    104.28.24.37
    

    View Slide

35. Chained exceptions
    >>> print(hostname_to_ip("nick.groenen.me"))
    104.28.24.37
    >>> print(hostname_to_ip("nosuchhost"))
    Traceback (most recent call last):
    File "", line 1, in
    File "", line 5, in hostname_to_ip
    __main__.IPLookupError: Unable to resolve 'nosuchhost'
    

    View Slide

36. Chained exceptions
    >>> print(hostname_to_ip("nick.groenen.me"))
    104.28.24.37
    >>> print(hostname_to_ip("nosuchhost"))
    Traceback (most recent call last):
    File "", line 1, in
    File "", line 5, in hostname_to_ip
    __main__.IPLookupError: Unable to resolve 'nosuchhost'
    Where did socket.gaierror go?!?
    

    View Slide

37. Chained exceptions
    >>> print(hostname_to_ip("nosuchhost"))
    Traceback (most recent call last):
    File "", line 3, in hostname_to_ip
    socket.gaierror: [Errno -2] Name or service not known
    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
    File "", line 1, in
    File "", line 5, in hostname_to_ip
    __main__.IPLookupError: Unable to resolve 'nosuchhost'
    

    View Slide

38. raise from ..
    

    View Slide

39. Chained exceptions
    >>> def hostname_to_ip(hostname):
    ... try:
    ... return socket.gethostbyname(hostname)
    ... except socket.gaierror:
    ... raise IPLookupError("Unable to resolve '%s'" % hostname)
    ...
    

    View Slide

40. Chained exceptions
    >>> def hostname_to_ip(hostname):
    ... try:
    ... return socket.gethostbyname(hostname)
    ... except socket.gaierror as e:
    ... raise IPLookupError("Unable to resolve '%s'" % hostname) from e
    ...
    

    View Slide

41. Chained exceptions
    >>> print(hostname_to_ip("nosuchhost"))
    Traceback (most recent call last):
    File "", line 3, in hostname_to_ip
    socket.gaierror: [Errno -2] Name or service not known
    The above exception was the direct cause of the following exception:
    Traceback (most recent call last):
    File "", line 1, in
    File "", line 5, in hostname_to_ip
    __main__.IPLookupError: Unable to resolve 'nosuchhost'
    

    View Slide

42. OSError subclasses
    try:
    shutil.copy2(source, dest)
    except OSError as e:
    if e.errno in [errno.EPERM, errno.EACCES]:
    sudo(shutil.copy2(source, dest))
    else:
    raise
    

    View Slide

43. OSError subclasses
    try:
    shutil.copy2(source, dest)
    except PermissionError:
    sudo(shutil.copy2(source, dest))
    

    View Slide

44. asyncio
    (formerly Tulip)
    

    View Slide

45. >>> def f(x: int) -> float:
    ... pass
    …
    >>> f.__annotations__
    {'return': , 'x': }
    Function annotations
    

    View Slide

46. Pyc files
    No more `find . -name '*.pyc' -delete`
    

    View Slide

47. venv
    

    View Slide

48. Python 3 looks awesome!
    Should I stop caring about
    Python 2 now?
    

    View Slide

49. Target both Python 2 and
    Python 3
    

    View Slide

50. Start with Python 3 only
    

    View Slide

51. Target Python 3.3 and up
    

    View Slide

52. Get your character {en,de}
    codings/unicode right
    

    View Slide

53. The Absolute Minimum Every Software
    Developer Absolutely, Positively Must Know
    About Unicode and Character Sets (No
    Excuses!)
    http://www.joelonsoftware.com/articles/Unicode.html
    

    View Slide

54. ➔ Encoded text comes in (socket, pipe, file, etc)
    ➔ .decode()
    ➔ Unicode internally
    ➔ .encode()
    ➔ Encoded text goes out
    Unicode sandwich
    

    View Slide

55. Start porting code
    

    View Slide

56. Have extensive test
    coverage
    You already have this, right? :-)
    

    View Slide

57. Target Python 2.7 and 3.3+
    from __future__ import absolute_import, division, print_function
    PY2 = sys.version_info[0] < 3
    PY3 = not PY2
    if PY3:
    from urllib.parse import urljoin
    from collections import MutableMapping as DictMixin
    else:
    from urlparse import urljoin
    from UserDict import DictMixin
    

    View Slide

58. Python-Future
    from __future__ import (
    absolute_import,
    division,
    print_function,
    )
    from builtins import (bytes, str, open, super, range,
    zip, round, input, int, pow, object)
    

    View Slide

59. Python-Future
    # Backported Py3 bytes object
    b = bytes(b'ABCD')
    assert repr(b) == "b'ABCD'"
    # These raise TypeErrors:
    # b + u'EFGH'
    # Backported Py3 str object
    s = str(u'ABCD')
    assert s != bytes(b'ABCD')
    # These raise TypeErrors:
    # bytes(b'B') in s
    # input() replaces Py2's raw_input() (with no eval()):
    name = input('What is your name? ')
    print('Hello ' + name)
    

    View Slide

60. Python-Future
    # Many Py3 module names are supported directly on both Py2.x and 3.x:
    from http.client import HttpConnection
    import html.parser
    import queue
    import xmlrpc.client
    # Refactored modules with clashing names on Py2 and Py3 are supported
    # as follows:
    from future import standard_library
    standard_library.install_aliases()
    # Then, for example:
    from itertools import filterfalse, zip_longest
    from urllib.request import urlopen
    from collections import Counter, OrderedDict # backported to Py2.6
    from collections import UserDict, UserList, UserString
    from subprocess import getoutput, getstatusoutput
    

    View Slide

61. Python-Future
    ➔ Py2-only -> Py2/Py3 with futurize
    ➔ Py3-only -> Py2/Py3 with pasturize
    

    View Slide

62. Six
    ➔ Uglier (but less magicky)
    ➔ Supports Python 2.5 and Python < 3.3
    

    View Slide

63. 2to3
    $ 2to3 -w my_python_2_script.py
    ➔ There's also a separate 3to2 to go the other
    way around
    

    View Slide

64. 2to3
    from setuptools import setup
    setup(
    name='your.module',
    version = '1.0',
    description='A description of your module',
    packages = ['your', 'you.module'],
    use_2to3 = True,
    convert_2to3_doctests = ['src/your/module/README.txt'],
    )
    

    View Slide

65. Thank you!
    

    View Slide

66. Resources
    ➔ Porting to Python 3 - The Book Site
    http://python3porting.com/bookindex.html
    ➔ Can I use Python 3?
    https://caniusepython3.com/
    ➔ Python 3 Readiness
    http://py3readiness.org/
    ➔ Porting to Python 3 (Django)
    https://docs.djangoproject.com/en/1.7/topics/python3/
    ➔ PortingToPy3k
    https://wiki.python.org/moin/PortingToPy3k/
    ➔ The Absolute Minimum Every Software Developer Absolutely, Positively Must Know
    About Unicode and Character Sets (No Excuses!)
    http://www.joelonsoftware.com/articles/Unicode.html
    

    View Slide