2017 - Building Bridges: Stopping Python 2 without damages

Description
Deciding to stop support for Python 2 is a hard choice. As library authors we try to minimize the breakage for users. IPython recently moved from single-source to Python 3 only, and worked with Core Python to make such a transition possible and easy. We'll show you the new features available in pip/setuptools to make that easy for you, and tell you how the transition went for IPython.

Abstract
Up until December 2016 it was hard if not impossible to mark a release of a Python package or library only compatible with some version of Python. Pip would happily download a Python 3 only version on Python 2 system, install it and break users systems.

After much work and patching many upstream project Python packagers can finally make user of the Requires-Python greater than or equal to 3.3 metadata, which will indicate to pip not to upgrade and thus shield most users from getting indecipherable error message from installing (or running) incompatible package versions.

While these changes should now be widely available, not all systems are fully up-to-date, and knowing how things can fails is necessary to packaging your libraries accordingly. We'll show you how to prepare for such a transition.

Finally we've release IPython 6.0 – using the above techniques – and have some numbers and our experience to share, for other to be ready if/when one of the library they know (or develop) is going to stop Python 2 support. Hopefully to do a tiny bit better than us.

Bio
Matthias Bussonnier is a Post Doctoral Scholar at the University of California Berkeley institute for data science and has been a core developer of IPython and Jupyter since 2012.

https://youtu.be/3i6n1RwqQCo

Transcript

1. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
   http://localhost:1948/pybay.md?print-pdf#/ 1/72
   Building Bridges
   Stopping Python 2 support in libraries without
   damages
   Matthias Bussonnier
   PyBay – Saturday August 13 2017 – Fisher West
   Slides available at http://bit.ly/pybay-2017
   1
   

   View Slide

2. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
   http://localhost:1948/pybay.md?print-pdf#/ 2/72
   About Me
   2 . 1
   

   View Slide

3. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
   http://localhost:1948/pybay.md?print-pdf#/ 3/72
   I'm French
   Appologies for Ze téribleuh Frènch akkeucent.
   2 . 2
   

   View Slide

4. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
   http://localhost:1948/pybay.md?print-pdf#/ 4/72
   Post Doc, IPython and Jupyter Core Dev
   GitHub : – Twitter :
   @Carreau @mbussonn
   2 . 3
   

   View Slide

5. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
   http://localhost:1948/pybay.md?print-pdf#/ 5/72
   jupytercon.com – August 22-25; NYC
   2 . 4
   

   View Slide

6. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
   http://localhost:1948/pybay.md?print-pdf#/ 6/72
   What this talk is not about
   Is Python 2 or 3 the right choice?
   Should I migrate to Python3-only?
   3 . 1
   

   View Slide

7. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
   http://localhost:1948/pybay.md?print-pdf#/ 7/72
   What this talk is about
   We migrated IPython to Python 3 only.
   We care about all of our users, Python 2 and 3 alike.
   Make the transition the least frustrating for users and developers.
   3 . 2
   

   View Slide

8. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
   http://localhost:1948/pybay.md?print-pdf#/ 8/72
   I'll be describing how we did this, how you can do it. How to do it
   right, potentially better than us.
   Things we did wrong – what you can learn from the mistake we
   did. And we'd like to learn from your experience when you do it.
   3 . 3
   

   View Slide

9. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
   http://localhost:1948/pybay.md?print-pdf#/ 9/72
   Python 2 vs 3 is an example
   The lessons we've learned are not specific a python2 to python3
   transition.
   This also applies to stopping support for any version (e.g., 2.6 or
   3.3).
   3 . 4
   

   View Slide

10. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 10/72
    Python 3 Statement
    List of who is stopping Python 2 support when. Resources on how
    to stop support with minimal frustration for users
    www.python3statement.org
    4 . 1
    

    View Slide

11. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 11/72
    4 . 2
    

    View Slide

12. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 12/72
    Timeline of Python 2 End of Life for various projects.
    4 . 3
    

    View Slide

13. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 13/72
    Why
    ”
    Numpy won't drop Python 2 while libraries
    depending on it are still python 2 compatible
    “
    ”
    We'll drop Python 2 once Numpy has dropped
    Python 2
    “
    4 . 4
    

    View Slide

14. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 14/72
    More informations
    www.python3statement.org
    4 . 5
    

    View Slide

15. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 15/72
    Application vs Library
    5 . 1
    

    View Slide

16. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 16/72
    Application
    You deploy it and have complete control over, often standalone.
    Dependencies should likely be pinned, in requirements.txt
    5 . 2
    

    View Slide

17. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 17/72
    Libraries
    Reusable components that are shared.
    Dependencies in setup.py, often "loose".
    5 . 3
    

    View Slide

18. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 18/72
    A couple of talk on upgrading an Application:
    Bringing Python 3 to LinkedIn (Zvezdan Petkovic) <- Not Online
    yet, that was 1h ago.
    PyCon 2017 Keynote – Python3@Instagram (Lisa Guo, Hui Ding)
    5 . 4
    

    View Slide

19. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 19/72
    Libraries
    Don't control the deploy environment.
    Still get issues from users
    We want to:
    Keep a wide compatibility range
    Keep users happy
    Minimise work for developers
    5 . 5
    

    View Slide

20. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 20/72
    Some users know where you live...
    5 . 6
    

    View Slide

21. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 21/72
    Scratch your own itch
    We wanted to release IPython 6, to be Python 3 only.
    We care about Python 2 users, so if a Python 2 user runs
    it should install the latest version of IPython 5, not IPython 6!
    $ pip install ipython --upgrade
    6 . 1
    

    View Slide

22. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 22/72
    Keep a LTS Branch
    6 . 2
    

    View Slide

23. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 23/72
    Core of the problem
    $ pip install ipython --upgrade
    Installing ipython... doing magic... success
    $ python
    >>> import IPython
    SyntaxWarningErrorError("I Just don't like you.")
    6 . 3
    

    View Slide

24. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 24/72
    What can we do about that ?
    6 . 4
    

    View Slide

25. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 25/72
    Solutions (Not really)
    Let's go back to 2016.
    7 . 1
    

    View Slide

26. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 26/72
    Just $ pip install "ipython<6" on Python 2
    Users do not always read documentation before installing.
    Scripts do not read documentation before installing.
    Users/scripts do not read error messages.
    dependencies – all packages need update to have conditional
    dependencies.
    7 . 2
    

    View Slide

27. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 27/72
    Rename ?
    That's going to be confusing and void most of the documentation
    on the Internet.
    Import names different from package name is also a bit tricky to
    explain sometime.
    (you meant pip install jinja2 not pip install jinja)
    7 . 3
    

    View Slide

28. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 28/72
    Wheel only ?
    Ok-ish for Pure-Python packages.
    Many downstream distributions requires (or really like) sdists.
    7 . 4
    

    View Slide

29. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 29/72
    Use a metapackage
    Use a package with virtually no-code that have conditional
    dependencies, and move the "real" code to a sub package.
    You kinda need to re-release old code (can't requires old-
    yourself)
    pip upgrade metapackage will not pull core unless pinned deps
    7 . 5
    

    View Slide

30. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 30/72
    As Raymond Hettinger would say if he is in the room
    ”
    There must be a better way !
    “
    7 . 6
    

    View Slide

31. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 31/72
    The new way: Python-Requires
    (Aka: make packaging great again)
    8 . 1
    

    View Slide

32. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 32/72
    Since December with pip 9.0.1+, and setuptools 24.3:
    Use pip install and it will adhere to python_requires.
    N.B.: Do not invoke setup.py directly!
    # setup.py
    setup(...,
    python_requires='>=3.4'
    )
    8 . 2
    

    View Slide

33. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 33/72
    In greater detail
    8 . 3
    

    View Slide

34. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 34/72
    python_requires metadata comes from , 2005.
    But for 11 years nothing implemented or understood it.
    pep 345
    8 . 4
    

    View Slide

35. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 35/72
    setuptools >= 24.3
    The python_requires keyword in known only by setuptools versions
    > 24.3.
    Required to build the sdist/wheel and publish the package
    Required to install from sdist.
    8 . 5
    

    View Slide

36. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 36/72
    pip >= 9.0.1
    Versions of pip < 9 ignore requires-python.
    This will result in installing incompatible versions.
    8 . 6
    

    View Slide

37. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 37/72
    Call to all package maintainers
    Use tools and services that respect Require-Python
    pip 9+
    setuptools >24.2
    Tag your package with requires_python
    (That's also valid if you do requires_python<3, or
    requires_python>2.2)
    9 . 1
    

    View Slide

38. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 38/72
    Tell users to update pip!
    Ask your users to upgrade
    Remind your users to upgrade
    Make sure your users upgrade.
    9 . 2
    

    View Slide

39. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 39/72
    Python 2 Python 3
    Old Pip ✗ ✓
    New Pip ✓* ✓
    9 . 3
    

    View Slide

40. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 40/72
    Defensive packaging
    Even if you adhere to that, problems will arise.
    Some helpful principles to keep your users as happy as possible
    10 . 1
    

    View Slide

41. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 41/72
    1. Update your documentation, CIs, and scripts to use pip.
    2. Communicate, Communicate, Communicate: Blog, twitter...
    3. Keep setup.py and __init__.py python 2 compatible,
    but catch errors early.
    4. For clear error messages in complicated situations,
    use multiple lines.
    10 . 2
    

    View Slide

42. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 42/72
    Tell everyone to pip install
    Update your documentation and scripts to use pip install [-e] .
    Reiteration: Do not use python setup.py <...>;
    it ignores requires_python for dependencies.
    10 . 3
    

    View Slide

43. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 43/72
    Communicate
    Let it be known to your community what your intentions are. Do not
    let people be surprised. Hopefully they will be prepared.
    10 . 4
    

    View Slide

44. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 44/72
    Keep setup.py python 2 compatible.
    If setup.py runs, most probable reason:
    pip < 9.
    Don't just say your package is incompatible with python2.
    Instead: Ask users to update pip.
    10 . 5
    

    View Slide

45. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 45/72
    E.g.,: in setup.py:
    if sys.version_info < (3, 3):
    error = """
    IPython 6.0+ does not support Python 2.6, 2.7, 3.0,
    3.1, or 3.2. Beginning with IPython 6.0, Python 3.3
    and above is required.
    This may be due to an out of date pip.
    Make sure you have pip >= 9.0.1.
    """
    sys.exit(error)
    10 . 6
    

    View Slide

46. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 46/72
    Keep __init__.py python 2 compatible
    Users will still find ways to avoid pip and setup.py.
    E.g.:
    $ pip install -e .
    $ ...
    $ git pull # update without install
    10 . 7
    

    View Slide

47. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 47/72
    E.g., in __init__.py:
    import sys
    if sys.version_info < (3,3):
    raise ImportError(
    """
    IPython 6.0+ does not support Python 2.6, 2.7, 3.0,
    3.1, or 3.2. Beginning with IPython 6.0, Python 3.3
    and above is required.
    """)
    10 . 8
    

    View Slide

48. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 48/72
    Summary
    Python 2 Python 3
    Old Pip ✓
    New Pip ✓* ✓
    Meaningfull warning: Pip out of date
    *
    Unless editable install, but protection in __init__.py
    11
    

    View Slide

49. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 49/72
    You know the difference between
    theory and practice ?
    In theory it's the same.
    12
    

    View Slide

50. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 50/72
    Result for IPython
    13 . 1
    

    View Slide

51. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 51/72
    Bug reports / complaints
    Two.
    During RC: python setup.py install got 6.0 on Python 2
    Remember: don't use python setup.py!
    13 . 2
    

    View Slide

52. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 52/72
    Remember: upgrade pip!
    (This also tell us not to trust what users are saying)
    ”
    My Bad. I thought the pip upgrade error
    message was irrelevant —
    when I upgrade pip it works.
    “
    13 . 3
    

    View Slide

53. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 53/72
    Ask the numbers !
    14 . 1
    

    View Slide

54. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 54/72
    PyPI download stats on BigQuery
    Month Before release ~65% using new pip.
    14 . 2
    

    View Slide

55. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 55/72
    14 . 3
    

    View Slide

56. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 56/72
    Python 3 adoption of new release
    14 . 4
    

    View Slide

57. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 57/72
    Python 2 adoption of new release
    14 . 5
    

    View Slide

58. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 58/72
    Pip 9+ does not download IPython 6.0 on Python 2 !
    Why the big download peek if the pip ratio was ~65% new pip ?
    14 . 6
    

    View Slide

59. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 59/72
    14 . 7
    

    View Slide

60. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 60/72
    Missing Data
    How many users on Python 2 have a new version of pip and are
    not upgrading?
    Fix by releasing a dummy version of the python 2 compatible
    version with no code change. Thus both Python 2 and 3 user
    should upgrade.
    14 . 8
    

    View Slide

61. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 61/72
    Releasing 2 versions at once
    14 . 9
    

    View Slide

62. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 62/72
    :-( Some people now have 5.3 Pinned.
    14 . 10
    

    View Slide

63. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 63/72
    Migration tips
    15 . 1
    

    View Slide

64. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 64/72
    Release a Patch version with Zero source change.
    Observe the change in number on PyPI
    15 . 2
    

    View Slide

65. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 65/72
    Have good error messages telling how to fix things, not why it
    failed.
    Upgrade pip, or
    pin with ipython<6
    (or compatible with: ipython~=5.1 )
    15 . 3
    

    View Slide

66. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 66/72
    Comunicate in advance
    15 . 4
    

    View Slide

67. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 67/72
    Practical tips for after the migration
    Resist the urge to delete code if you have an LTS.
    Backport merges will conflict otherwise
    15 . 5
    

    View Slide

68. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 68/72
    Conclusions
    15 . 6
    

    View Slide

69. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 69/72
    On IPython
    IPython 6+ is Python3 only
    We're still updating the IPython 5.x – Python 2 LTS branch
    Transition has gone relatively well for IPython!
    It will only get easier!
    15 . 7
    

    View Slide

70. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 70/72
    On converting packages to Python3 only
    use packaging tools that respect Requires_Python
    encourage everyone, everywhere to use pip 9+
    follow defensive packaging practices
    Read and contribute to python3statement practicalities section
    15 . 8
    

    View Slide

71. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 71/72
    Thanks!
    Python 3 \u2661 Python 2
    (that's a )
    16 . 1
    

    View Slide

72. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages
    http://localhost:1948/pybay.md?print-pdf#/ 72/72
    Slides at: http://bit.ly/pybay-2017
    16 . 2
    

    View Slide