Twisted & Python 3 (Python Language Summit, 2016)

Transcript

1. Twisted & Python 3
2. Hello, I’m Amber Brown (HawkOwl)
4. My Python 3 Porting (in numbers)
5. Twisted’s Python 3 Port: 50% completion by LoC
6. 42+ Twisted patches (personally authored and merged)
7. 754943c Merge reactor-app-py3-8299: Port twisted.plugins.twisted_reactors to Python 3 9dc247e Merge telnet-py3-8228: Port twisted.conch.telnet to Python 3 f0cac8f Merge conch-checkers-py3-8225: Port twisted.conch.checkers to Python 3 23fb808 Merge strcred-dist3-8216: port twisted.cred.strcred to Python 3. 8fd84c3 Merge conch-ssh-keys-py3-7998-3: Port twisted.conch.ssh.keys to Python 3 0edae7d Merge conch-ssh-keys-py3-7998-3: Port twisted.conch.ssh.keys to Python 3 90c1ce1 Merge py35-tox-8106: Add Python 3.5 to the Tox matrix 153e705 Merge iosim-py3-8064: Port the rest of twisted.test.iosim to Python 3 ac9c537 Merge agent-py3-7407-3: Port twisted.web.client.Agent to Python 3 b5a49ca Merge multipart-py3-8052: Fix multipart handling on Python 3 1e2c670 Merge downloadpage-py3-6197-2: Port twisted.web.client.downloadPage to Python 3 29e07e7 Merge python35-8042-2: Support Python 3.5 on POSIX 75dfeff Merge zippath-py3-6917-2: Port twisted.python.zipfile to Python 3 dadd41d Merge twistd-web-py3-8008-3: Port twistd web to Python 3 4c49f5a Merge twistd-py3-8012: Port twistd to Python 3 a25a74d Merge strports-py3-8011: Port twisted.application.strports to Python 3 0215dd0 Merge processingfailed-py3-7996: Make twisted.web.server.Request.processingFailed work on Python 3 9d676cc Merge serverFromString-py-7982: Port twisted.internet.endpoints.serverFromString to Python 3 59daefa Merge positioning-py3-7987: Port twisted.positioning to Python 3 0413047 Merge threads-py3-7990: Install all _threads submodules on Python 3 f90909f Merge trialrunner-py3-5965-9: Port trial to Python 3 867e470 Merge kqueue-eintr-7887-3: Fix twisted.internet.kqueuereactor doesn'tinspect an exception's errno properly on Python 3 55f7231 Merge twwebtemplate-py3-7811-4: Port twisted.web.template to Python 3 3f5fc18 Merge twwebtemplate-py3-7811-4: Port twisted.web.template to Python 3 bfc9f54 Merge twapp-py3-6914-2: Port twisted.application.app to Python 3 e8be925 Merge twapp-internet-py3-7817: Port twisted.application.internet to Python 3 3aa5cde Merge tp-testutil-7940: twisted.python.test.test_util should be ported to Python 3 788821a Merge twstan-py3-7952: Port twisted.web._stan to Python 3 e0ee4f2 Merge tiunix-py3-7874-2: Port twisted.internet.unix to Python 3 ea33edb Merge sendmsg-py3-7884-2: Port twisted.python.sendmsg to Python 3 4a3d509 Merge twcred-py3-7834-3: Port twisted.cred.checkers to Python 3 79bb6c7 Merge stdio-py3-7899: Port twisted.internet.stdio to Python 3 a31eed7 Merge tccredentials-py3-6176-3: Port twisted.cred.credentials to Python 3 b7b1a8d Merge defaultgai-py34-7886: twisted.internet.endpoints.HostnameEndpoint now uses getaddrinfo properly on Python 3.4 and above. 6387bf9 Merge idnabytes-7287: Fix twisted.internet._sslverify._idnaBytes on Python 3 ca00b8f Merge processendpoint-py3-7824-5: Port twisted.internet.endpoints.ProcessEndpoint to Python 3 2b73df4 Merge tiprocess-py3-5987-5: Port twisted.internet.process to Python 3 989d007 Merge py3tests-venv-7838: Ensure that Twisted installs on Python 3 correctly ab110cc Merge twstatic-py3-6177-4: Port twisted.web.static to Python 3 ee2d6ed Merge kqueue-python3-7823: twisted.internet.kqreactor should be ported to Python 3 f7f4bb0 Merging trialtestlog-py3-7812: Port twisted.trial.test.test_log to Python 3 adf626e Merge port-systemd-7419: Port twisted.python.systemd to Python 3.
8. 40,000+ LoC ported 100,000+ LoC opened to port
9. 6472 insertions(+) 4437 deletions(-)
10. Why has it taken so long?
11. Bytes/Unicode divide A good change Cleaned up a lot of ambiguity
12. Twisted deals primarily with bytestrings
13. No b"" initially in Python 3 hamstrung efforts
14. No % formatting in the majority of Python 3 versions means rewriting that code
15. No .format means our new 2.7 only code has to be written
16. Extra time porting, more code to review
17. Python effectively has three string types bytes, unicode, str
18. cgi.parse_multipart returns strs on Python 3
19. Avalanche of changes caused problems porting
20. New-style classes by default broke a lot of things
21. No bound/unbound methods broke a lot of things
22. But I'm sure you all know the problems with Python 3 porting
23. The Cost
24. Python 3 is an expensive transition
25. ~2 hours a day for a year and a half == In excess of 1000 hours
26. In $ values: $10,000+ sponsored work (me) $1,600 PSF grant $50,000+ (estimate) unpaid time
27. Thousands of hours left to go
28. The unfortunate reality is that if I didn't do it, it would not happen.
29. Python 3 has been a massive drain on Twisted feature development
30. Review queue is 50%+ Python 3 porting work
31. Reviewers are scarce and precious, and Python 3 porting requires a lot of care
32. And who is using it?
33. Speed
34. The reality: Python is falling by the wayside for new performance- sensitive applications.
35. Twisted on Python 3 is a less attractive deployment target than Twisted on PyPy 5.1
36. Spend time porting, radically update your codebase for… slower code?
37. "From Python to Go and Back Again" by Ben Bangert Python 3 with threads, to go, to Python 2 + Twisted on PyPy
38. PyPy means Python is competitive with Go
39. 3,500+ C API functions in CPython
40. Extra burden on PyPy means we're not getting there yet
41. Twisted on Pyston would still be 3.5
42. How do we stop this from happening again?
43. asyncio
44. Questions? (pls no statements, save them for after)