Twisted & Python 3 (Python Language Summit, 2016)

Transcript

1. Twisted & Python 3

2. Hello, I’m Amber Brown (HawkOwl)

3. None

4. My Python 3 Porting (in numbers)

5. Twisted’s Python 3 Port: 50% completion by LoC

6. 42+ Twisted patches (personally authored and merged)

7. 754943c Merge reactor-app-py3-8299: Port twisted.plugins.twisted_reactors to Python 3 9dc247e Merge

   telnet-py3-8228: Port twisted.conch.telnet to Python 3 f0cac8f Merge conch-checkers-py3-8225: Port twisted.conch.checkers to Python 3 23fb808 Merge strcred-dist3-8216: port twisted.cred.strcred to Python 3. 8fd84c3 Merge conch-ssh-keys-py3-7998-3: Port twisted.conch.ssh.keys to Python 3 0edae7d Merge conch-ssh-keys-py3-7998-3: Port twisted.conch.ssh.keys to Python 3 90c1ce1 Merge py35-tox-8106: Add Python 3.5 to the Tox matrix 153e705 Merge iosim-py3-8064: Port the rest of twisted.test.iosim to Python 3 ac9c537 Merge agent-py3-7407-3: Port twisted.web.client.Agent to Python 3 b5a49ca Merge multipart-py3-8052: Fix multipart handling on Python 3 1e2c670 Merge downloadpage-py3-6197-2: Port twisted.web.client.downloadPage to Python 3 29e07e7 Merge python35-8042-2: Support Python 3.5 on POSIX 75dfeff Merge zippath-py3-6917-2: Port twisted.python.zipfile to Python 3 dadd41d Merge twistd-web-py3-8008-3: Port twistd web to Python 3 4c49f5a Merge twistd-py3-8012: Port twistd to Python 3 a25a74d Merge strports-py3-8011: Port twisted.application.strports to Python 3 0215dd0 Merge processingfailed-py3-7996: Make twisted.web.server.Request.processingFailed work on Python 3 9d676cc Merge serverFromString-py-7982: Port twisted.internet.endpoints.serverFromString to Python 3 59daefa Merge positioning-py3-7987: Port twisted.positioning to Python 3 0413047 Merge threads-py3-7990: Install all _threads submodules on Python 3 f90909f Merge trialrunner-py3-5965-9: Port trial to Python 3 867e470 Merge kqueue-eintr-7887-3: Fix twisted.internet.kqueuereactor doesn'tinspect an exception's errno properly on Python 3 55f7231 Merge twwebtemplate-py3-7811-4: Port twisted.web.template to Python 3 3f5fc18 Merge twwebtemplate-py3-7811-4: Port twisted.web.template to Python 3 bfc9f54 Merge twapp-py3-6914-2: Port twisted.application.app to Python 3 e8be925 Merge twapp-internet-py3-7817: Port twisted.application.internet to Python 3 3aa5cde Merge tp-testutil-7940: twisted.python.test.test_util should be ported to Python 3 788821a Merge twstan-py3-7952: Port twisted.web._stan to Python 3 e0ee4f2 Merge tiunix-py3-7874-2: Port twisted.internet.unix to Python 3 ea33edb Merge sendmsg-py3-7884-2: Port twisted.python.sendmsg to Python 3 4a3d509 Merge twcred-py3-7834-3: Port twisted.cred.checkers to Python 3 79bb6c7 Merge stdio-py3-7899: Port twisted.internet.stdio to Python 3 a31eed7 Merge tccredentials-py3-6176-3: Port twisted.cred.credentials to Python 3 b7b1a8d Merge defaultgai-py34-7886: twisted.internet.endpoints.HostnameEndpoint now uses getaddrinfo properly on Python 3.4 and above. 6387bf9 Merge idnabytes-7287: Fix twisted.internet._sslverify._idnaBytes on Python 3 ca00b8f Merge processendpoint-py3-7824-5: Port twisted.internet.endpoints.ProcessEndpoint to Python 3 2b73df4 Merge tiprocess-py3-5987-5: Port twisted.internet.process to Python 3 989d007 Merge py3tests-venv-7838: Ensure that Twisted installs on Python 3 correctly ab110cc Merge twstatic-py3-6177-4: Port twisted.web.static to Python 3 ee2d6ed Merge kqueue-python3-7823: twisted.internet.kqreactor should be ported to Python 3 f7f4bb0 Merging trialtestlog-py3-7812: Port twisted.trial.test.test_log to Python 3 adf626e Merge port-systemd-7419: Port twisted.python.systemd to Python 3.

8. 40,000+ LoC ported 100,000+ LoC opened to port

9. 6472 insertions(+) 4437 deletions(-)

10. Why has it taken so long?

11. Bytes/Unicode divide A good change Cleaned up a lot of

    ambiguity

12. Twisted deals primarily with bytestrings

13. No b"" initially in Python 3 hamstrung efforts

14. No % formatting in the majority of Python 3 versions

    means rewriting that code

15. No .format means our new 2.7 only code has to

    be written

16. Extra time porting, more code to review

17. Python effectively has three string types bytes, unicode, str

18. cgi.parse_multipart returns strs on Python 3

19. Avalanche of changes caused problems porting

20. New-style classes by default broke a lot of things

21. No bound/unbound methods broke a lot of things

22. But I'm sure you all know the problems with Python

    3 porting

23. The Cost

24. Python 3 is an expensive transition

25. ~2 hours a day for a year and a half

    == In excess of 1000 hours

26. In $ values: $10,000+ sponsored work (me) $1,600 PSF grant

    $50,000+ (estimate) unpaid time

27. Thousands of hours left to go

28. The unfortunate reality is that if I didn't do it,

    it would not happen.

29. Python 3 has been a massive drain on Twisted feature

    development

30. Review queue is 50%+ Python 3 porting work

31. Reviewers are scarce and precious, and Python 3 porting requires

    a lot of care

32. And who is using it?

33. Speed

34. The reality: Python is falling by the wayside for new

    performance- sensitive applications.

35. Twisted on Python 3 is a less attractive deployment target

    than Twisted on PyPy 5.1

36. Spend time porting, radically update your codebase for… slower code?

37. "From Python to Go and Back Again" by Ben Bangert

    Python 3 with threads, to go, to Python 2 + Twisted on PyPy

38. PyPy means Python is competitive with Go

39. 3,500+ C API functions in CPython

40. Extra burden on PyPy means we're not getting there yet

41. Twisted on Pyston would still be 3.5

42. How do we stop this from happening again?

43. asyncio

44. Questions? (pls no statements, save them for after)