Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Twisted & Python 3 (Python Language Summit, 2016)

Twisted & Python 3 (Python Language Summit, 2016)


Amber Brown (HawkOwl)

May 29, 2016

More Decks by Amber Brown (HawkOwl)

Other Decks in Programming


  1. Twisted & Python 3

  2. Hello, I’m Amber Brown (HawkOwl)

  3. None
  4. My Python 3 Porting (in numbers)

  5. Twisted’s Python 3 Port: 50% completion by LoC

  6. 42+ Twisted patches (personally authored and merged)

  7. 754943c Merge reactor-app-py3-8299: Port twisted.plugins.twisted_reactors to Python 3 9dc247e Merge

    telnet-py3-8228: Port twisted.conch.telnet to Python 3 f0cac8f Merge conch-checkers-py3-8225: Port twisted.conch.checkers to Python 3 23fb808 Merge strcred-dist3-8216: port twisted.cred.strcred to Python 3. 8fd84c3 Merge conch-ssh-keys-py3-7998-3: Port twisted.conch.ssh.keys to Python 3 0edae7d Merge conch-ssh-keys-py3-7998-3: Port twisted.conch.ssh.keys to Python 3 90c1ce1 Merge py35-tox-8106: Add Python 3.5 to the Tox matrix 153e705 Merge iosim-py3-8064: Port the rest of twisted.test.iosim to Python 3 ac9c537 Merge agent-py3-7407-3: Port twisted.web.client.Agent to Python 3 b5a49ca Merge multipart-py3-8052: Fix multipart handling on Python 3 1e2c670 Merge downloadpage-py3-6197-2: Port twisted.web.client.downloadPage to Python 3 29e07e7 Merge python35-8042-2: Support Python 3.5 on POSIX 75dfeff Merge zippath-py3-6917-2: Port twisted.python.zipfile to Python 3 dadd41d Merge twistd-web-py3-8008-3: Port twistd web to Python 3 4c49f5a Merge twistd-py3-8012: Port twistd to Python 3 a25a74d Merge strports-py3-8011: Port twisted.application.strports to Python 3 0215dd0 Merge processingfailed-py3-7996: Make twisted.web.server.Request.processingFailed work on Python 3 9d676cc Merge serverFromString-py-7982: Port twisted.internet.endpoints.serverFromString to Python 3 59daefa Merge positioning-py3-7987: Port twisted.positioning to Python 3 0413047 Merge threads-py3-7990: Install all _threads submodules on Python 3 f90909f Merge trialrunner-py3-5965-9: Port trial to Python 3 867e470 Merge kqueue-eintr-7887-3: Fix twisted.internet.kqueuereactor doesn'tinspect an exception's errno properly on Python 3 55f7231 Merge twwebtemplate-py3-7811-4: Port twisted.web.template to Python 3 3f5fc18 Merge twwebtemplate-py3-7811-4: Port twisted.web.template to Python 3 bfc9f54 Merge twapp-py3-6914-2: Port twisted.application.app to Python 3 e8be925 Merge twapp-internet-py3-7817: Port twisted.application.internet to Python 3 3aa5cde Merge tp-testutil-7940: twisted.python.test.test_util should be ported to Python 3 788821a Merge twstan-py3-7952: Port twisted.web._stan to Python 3 e0ee4f2 Merge tiunix-py3-7874-2: Port twisted.internet.unix to Python 3 ea33edb Merge sendmsg-py3-7884-2: Port twisted.python.sendmsg to Python 3 4a3d509 Merge twcred-py3-7834-3: Port twisted.cred.checkers to Python 3 79bb6c7 Merge stdio-py3-7899: Port twisted.internet.stdio to Python 3 a31eed7 Merge tccredentials-py3-6176-3: Port twisted.cred.credentials to Python 3 b7b1a8d Merge defaultgai-py34-7886: twisted.internet.endpoints.HostnameEndpoint now uses getaddrinfo properly on Python 3.4 and above. 6387bf9 Merge idnabytes-7287: Fix twisted.internet._sslverify._idnaBytes on Python 3 ca00b8f Merge processendpoint-py3-7824-5: Port twisted.internet.endpoints.ProcessEndpoint to Python 3 2b73df4 Merge tiprocess-py3-5987-5: Port twisted.internet.process to Python 3 989d007 Merge py3tests-venv-7838: Ensure that Twisted installs on Python 3 correctly ab110cc Merge twstatic-py3-6177-4: Port twisted.web.static to Python 3 ee2d6ed Merge kqueue-python3-7823: twisted.internet.kqreactor should be ported to Python 3 f7f4bb0 Merging trialtestlog-py3-7812: Port twisted.trial.test.test_log to Python 3 adf626e Merge port-systemd-7419: Port twisted.python.systemd to Python 3.
  8. 40,000+ LoC ported 100,000+ LoC opened to port

  9. 6472 insertions(+) 4437 deletions(-)

  10. Why has it taken so long?

  11. Bytes/Unicode divide A good change Cleaned up a lot of

  12. Twisted deals primarily with bytestrings

  13. No b"" initially in Python 3 hamstrung efforts

  14. No % formatting in the majority of Python 3 versions

    means rewriting that code
  15. No .format means our new 2.7 only code has to

    be written
  16. Extra time porting, more code to review

  17. Python effectively has three string types bytes, unicode, str

  18. cgi.parse_multipart returns strs on Python 3

  19. Avalanche of changes caused problems porting

  20. New-style classes by default broke a lot of things

  21. No bound/unbound methods broke a lot of things

  22. But I'm sure you all know the problems with Python

    3 porting
  23. The Cost

  24. Python 3 is an expensive transition

  25. ~2 hours a day for a year and a half

    == In excess of 1000 hours
  26. In $ values: $10,000+ sponsored work (me) $1,600 PSF grant

    $50,000+ (estimate) unpaid time
  27. Thousands of hours left to go

  28. The unfortunate reality is that if I didn't do it,

    it would not happen.
  29. Python 3 has been a massive drain on Twisted feature

  30. Review queue is 50%+ Python 3 porting work

  31. Reviewers are scarce and precious, and Python 3 porting requires

    a lot of care
  32. And who is using it?

  33. Speed

  34. The reality: Python is falling by the wayside for new

    performance- sensitive applications.
  35. Twisted on Python 3 is a less attractive deployment target

    than Twisted on PyPy 5.1
  36. Spend time porting, radically update your codebase for… slower code?

  37. "From Python to Go and Back Again" by Ben Bangert

    Python 3 with threads, to go, to Python 2 + Twisted on PyPy
  38. PyPy means Python is competitive with Go

  39. 3,500+ C API functions in CPython

  40. Extra burden on PyPy means we're not getting there yet

  41. Twisted on Pyston would still be 3.5

  42. How do we stop this from happening again?

  43. asyncio

  44. Questions? (pls no statements, save them for after)