2017 - Building Bridges: Stopping Python 2 without damages

Slide 1

Slide 1 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 1/72 Building Bridges Stopping Python 2 support in libraries without damages Matthias Bussonnier PyBay – Saturday August 13 2017 – Fisher West Slides available at http://bit.ly/pybay-2017 1

Slide 2

Slide 2 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 2/72 About Me 2 . 1

Slide 3

Slide 3 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 3/72 I'm French Appologies for Ze téribleuh Frènch akkeucent. 2 . 2

Slide 4

Slide 4 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 4/72 Post Doc, IPython and Jupyter Core Dev GitHub : – Twitter : @Carreau @mbussonn 2 . 3

Slide 5

Slide 5 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 5/72 jupytercon.com – August 22-25; NYC 2 . 4

Slide 6

Slide 6 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 6/72 What this talk is not about Is Python 2 or 3 the right choice? Should I migrate to Python3-only? 3 . 1

Slide 7

Slide 7 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 7/72 What this talk is about We migrated IPython to Python 3 only. We care about all of our users, Python 2 and 3 alike. Make the transition the least frustrating for users and developers. 3 . 2

Slide 8

Slide 8 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 8/72 I'll be describing how we did this, how you can do it. How to do it right, potentially better than us. Things we did wrong – what you can learn from the mistake we did. And we'd like to learn from your experience when you do it. 3 . 3

Slide 9

Slide 9 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 9/72 Python 2 vs 3 is an example The lessons we've learned are not speciﬁc a python2 to python3 transition. This also applies to stopping support for any version (e.g., 2.6 or 3.3). 3 . 4

Slide 10

Slide 10 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 10/72 Python 3 Statement List of who is stopping Python 2 support when. Resources on how to stop support with minimal frustration for users www.python3statement.org 4 . 1

Slide 11

Slide 11 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 11/72 4 . 2

Slide 12

Slide 12 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 12/72 Timeline of Python 2 End of Life for various projects. 4 . 3

Slide 13

Slide 13 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 13/72 Why ” Numpy won't drop Python 2 while libraries depending on it are still python 2 compatible “ ” We'll drop Python 2 once Numpy has dropped Python 2 “ 4 . 4

Slide 14

Slide 14 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 14/72 More informations www.python3statement.org 4 . 5

Slide 15

Slide 15 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 15/72 Application vs Library 5 . 1

Slide 16

Slide 16 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 16/72 Application You deploy it and have complete control over, often standalone. Dependencies should likely be pinned, in requirements.txt 5 . 2

Slide 17

Slide 17 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 17/72 Libraries Reusable components that are shared. Dependencies in setup.py, often "loose". 5 . 3

Slide 18

Slide 18 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 18/72 A couple of talk on upgrading an Application: Bringing Python 3 to LinkedIn (Zvezdan Petkovic) <- Not Online yet, that was 1h ago. PyCon 2017 Keynote – Python3@Instagram (Lisa Guo, Hui Ding) 5 . 4

Slide 19

Slide 19 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 19/72 Libraries Don't control the deploy environment. Still get issues from users We want to: Keep a wide compatibility range Keep users happy Minimise work for developers 5 . 5

Slide 20

Slide 20 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 20/72 Some users know where you live... 5 . 6

Slide 21

Slide 21 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 21/72 Scratch your own itch We wanted to release IPython 6, to be Python 3 only. We care about Python 2 users, so if a Python 2 user runs it should install the latest version of IPython 5, not IPython 6! $ pip install ipython --upgrade 6 . 1

Slide 22

Slide 22 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 22/72 Keep a LTS Branch 6 . 2

Slide 23

Slide 23 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 23/72 Core of the problem $ pip install ipython --upgrade Installing ipython... doing magic... success $ python >>> import IPython SyntaxWarningErrorError("I Just don't like you.") 6 . 3

Slide 24

Slide 24 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 24/72 What can we do about that ? 6 . 4

Slide 25

Slide 25 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 25/72 Solutions (Not really) Let's go back to 2016. 7 . 1

Slide 26

Slide 26 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 26/72 Just $ pip install "ipython<6" on Python 2 Users do not always read documentation before installing. Scripts do not read documentation before installing. Users/scripts do not read error messages. dependencies – all packages need update to have conditional dependencies. 7 . 2

Slide 27

Slide 27 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 27/72 Rename ? That's going to be confusing and void most of the documentation on the Internet. Import names different from package name is also a bit tricky to explain sometime. (you meant pip install jinja2 not pip install jinja) 7 . 3

Slide 28

Slide 28 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 28/72 Wheel only ? Ok-ish for Pure-Python packages. Many downstream distributions requires (or really like) sdists. 7 . 4

Slide 29

Slide 29 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 29/72 Use a metapackage Use a package with virtually no-code that have conditional dependencies, and move the "real" code to a sub package. You kinda need to re-release old code (can't requires old- yourself) pip upgrade metapackage will not pull core unless pinned deps 7 . 5

Slide 30

Slide 30 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 30/72 As Raymond Hettinger would say if he is in the room ” There must be a better way ! “ 7 . 6

Slide 31

Slide 31 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 31/72 The new way: Python-Requires (Aka: make packaging great again) 8 . 1

Slide 32

Slide 32 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 32/72 Since December with pip 9.0.1+, and setuptools 24.3: Use pip install and it will adhere to python_requires. N.B.: Do not invoke setup.py directly! # setup.py setup(..., python_requires='>=3.4' ) 8 . 2

Slide 33

Slide 33 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 33/72 In greater detail 8 . 3

Slide 34

Slide 34 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 34/72 python_requires metadata comes from , 2005. But for 11 years nothing implemented or understood it. pep 345 8 . 4

Slide 35

Slide 35 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 35/72 setuptools >= 24.3 The python_requires keyword in known only by setuptools versions > 24.3. Required to build the sdist/wheel and publish the package Required to install from sdist. 8 . 5

Slide 36

Slide 36 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 36/72 pip >= 9.0.1 Versions of pip < 9 ignore requires-python. This will result in installing incompatible versions. 8 . 6

Slide 37

Slide 37 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 37/72 Call to all package maintainers Use tools and services that respect Require-Python pip 9+ setuptools >24.2 Tag your package with requires_python (That's also valid if you do requires_python<3, or requires_python>2.2) 9 . 1

Slide 38

Slide 38 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 38/72 Tell users to update pip! Ask your users to upgrade Remind your users to upgrade Make sure your users upgrade. 9 . 2

Slide 39

Slide 39 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 39/72 Python 2 Python 3 Old Pip ✗ ✓ New Pip ✓* ✓ 9 . 3

Slide 40

Slide 40 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 40/72 Defensive packaging Even if you adhere to that, problems will arise. Some helpful principles to keep your users as happy as possible 10 . 1

Slide 41

Slide 41 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 41/72 1. Update your documentation, CIs, and scripts to use pip. 2. Communicate, Communicate, Communicate: Blog, twitter... 3. Keep setup.py and __init__.py python 2 compatible, but catch errors early. 4. For clear error messages in complicated situations, use multiple lines. 10 . 2

Slide 42

Slide 42 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 42/72 Tell everyone to pip install Update your documentation and scripts to use pip install [-e] . Reiteration: Do not use python setup.py <...>; it ignores requires_python for dependencies. 10 . 3

Slide 43

Slide 43 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 43/72 Communicate Let it be known to your community what your intentions are. Do not let people be surprised. Hopefully they will be prepared. 10 . 4

Slide 44

Slide 44 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 44/72 Keep setup.py python 2 compatible. If setup.py runs, most probable reason: pip < 9. Don't just say your package is incompatible with python2. Instead: Ask users to update pip. 10 . 5

Slide 45

Slide 45 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 45/72 E.g.,: in setup.py: if sys.version_info < (3, 3): error = """ IPython 6.0+ does not support Python 2.6, 2.7, 3.0, 3.1, or 3.2. Beginning with IPython 6.0, Python 3.3 and above is required. This may be due to an out of date pip. Make sure you have pip >= 9.0.1. """ sys.exit(error) 10 . 6

Slide 46

Slide 46 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 46/72 Keep __init__.py python 2 compatible Users will still ﬁnd ways to avoid pip and setup.py. E.g.: $ pip install -e . $ ... $ git pull # update without install 10 . 7

Slide 47

Slide 47 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 47/72 E.g., in __init__.py: import sys if sys.version_info < (3,3): raise ImportError( """ IPython 6.0+ does not support Python 2.6, 2.7, 3.0, 3.1, or 3.2. Beginning with IPython 6.0, Python 3.3 and above is required. """) 10 . 8

Slide 48

Slide 48 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 48/72 Summary Python 2 Python 3 Old Pip ✓ New Pip ✓* ✓ Meaningfull warning: Pip out of date * Unless editable install, but protection in __init__.py 11

Slide 49

Slide 49 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 49/72 You know the difference between theory and practice ? In theory it's the same. 12

Slide 50

Slide 50 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 50/72 Result for IPython 13 . 1

Slide 51

Slide 51 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 51/72 Bug reports / complaints Two. During RC: python setup.py install got 6.0 on Python 2 Remember: don't use python setup.py! 13 . 2

Slide 52

Slide 52 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 52/72 Remember: upgrade pip! (This also tell us not to trust what users are saying) ” My Bad. I thought the pip upgrade error message was irrelevant — when I upgrade pip it works. “ 13 . 3

Slide 53

Slide 53 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 53/72 Ask the numbers ! 14 . 1

Slide 54

Slide 54 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 54/72 PyPI download stats on BigQuery Month Before release ~65% using new pip. 14 . 2

Slide 55

Slide 55 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 55/72 14 . 3

Slide 56

Slide 56 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 56/72 Python 3 adoption of new release 14 . 4

Slide 57

Slide 57 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 57/72 Python 2 adoption of new release 14 . 5

Slide 58

Slide 58 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 58/72 Pip 9+ does not download IPython 6.0 on Python 2 ! Why the big download peek if the pip ratio was ~65% new pip ? 14 . 6

Slide 59

Slide 59 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 59/72 14 . 7

Slide 60

Slide 60 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 60/72 Missing Data How many users on Python 2 have a new version of pip and are not upgrading? Fix by releasing a dummy version of the python 2 compatible version with no code change. Thus both Python 2 and 3 user should upgrade. 14 . 8

Slide 61

Slide 61 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 61/72 Releasing 2 versions at once 14 . 9

Slide 62

Slide 62 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 62/72 :-( Some people now have 5.3 Pinned. 14 . 10

Slide 63

Slide 63 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 63/72 Migration tips 15 . 1

Slide 64

Slide 64 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 64/72 Release a Patch version with Zero source change. Observe the change in number on PyPI 15 . 2

Slide 65

Slide 65 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 65/72 Have good error messages telling how to ﬁx things, not why it failed. Upgrade pip, or pin with ipython<6 (or compatible with: ipython~=5.1 ) 15 . 3

Slide 66

Slide 66 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 66/72 Comunicate in advance 15 . 4

Slide 67

Slide 67 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 67/72 Practical tips for after the migration Resist the urge to delete code if you have an LTS. Backport merges will conﬂict otherwise 15 . 5

Slide 68

Slide 68 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 68/72 Conclusions 15 . 6

Slide 69

Slide 69 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 69/72 On IPython IPython 6+ is Python3 only We're still updating the IPython 5.x – Python 2 LTS branch Transition has gone relatively well for IPython! It will only get easier! 15 . 7

Slide 70

Slide 70 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 70/72 On converting packages to Python3 only use packaging tools that respect Requires_Python encourage everyone, everywhere to use pip 9+ follow defensive packaging practices Read and contribute to python3statement practicalities section 15 . 8

Slide 71

Slide 71 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 71/72 Thanks! Python 3 \u2661 Python 2 (that's a ) 16 . 1

Slide 72

Slide 72 text

8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 72/72 Slides at: http://bit.ly/pybay-2017 16 . 2