$30 off During Our Annual Pro Sale. View Details »

Cory Benfield - Building Protocol Libraries The Right Way

Cory Benfield - Building Protocol Libraries The Right Way

One of the great strengths of the Python ecosystem is the enormous collection of powerful, flexible libraries. However, these libraries tend to suffer from one extremely common design flaw that mean that the work done is not easily re-usable or transferable. In this talk, we talk about how to build libraries that can be used as widely as possible, through the lens of the Python Hyper HTTP project.

https://us.pycon.org/2016/schedule/presentation/1743/

PyCon 2016

May 29, 2016
Tweet

More Decks by PyCon 2016

Other Decks in Programming

Transcript

  1. Building Protocol
    Libraries
    The Right Way

    View Slide

  2. Hi!

    View Slide

  3. @lukasaoz
    @lukasa

    View Slide

  4. View Slide

  5. Python HTTP
    • Hyper
    • h2, hpack, priority, etc.
    • Requests + urllib3
    • PyCA (PyOpenSSL)

    View Slide

  6. Why All Your Libraries
    Are Garbage, And All
    But One Of Mine Are
    Too

    View Slide

  7. Python HTTP
    Rocks

    View Slide

  8. • Requests/urllib3/HTTPie
    • gunicorn
    • aiohttp
    • Twisted
    • Tornado

    View Slide

  9. ~0% Shared
    Code

    View Slide

  10. Code Re-Use

    View Slide

  11. I/O

    View Slide

  12. • httplib: state machine and
    parser mixed with I/O
    • aiohttp/Tornado/Twisted:
    state machine and parser
    use concurrency primitives

    View Slide

  13. Choice of I/O limits
    choice of libraries

    View Slide

  14. So What?

    View Slide

  15. Wasted Effort

    View Slide

  16. Reduces
    Experimentation

    View Slide

  17. Duplicate
    Bugs

    View Slide

  18. Limits
    Optimisation

    View Slide

  19. Avoiding
    This Problem

    View Slide

  20. Don’t Do I/O

    View Slide

  21. Alternative API
    events = handle_data(in_bytes)
    out_bytes = perform_action()

    View Slide

  22. Not All-Purpose

    View Slide

  23. Build
    Wrappers

    View Slide

  24. Why Bother?

    View Slide

  25. ===================== test session starts ======================
    ------- coverage: platform darwin, python 2.7.11-final-0 -------
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------------
    h2/__init__.py 2 0 0 0 100%
    h2/connection.py 505 0 120 0 100%
    h2/errors.py 17 0 0 0 100%
    h2/events.py 107 0 4 0 100%
    h2/exceptions.py 41 0 0 0 100%
    h2/frame_buffer.py 55 0 18 0 100%
    h2/settings.py 79 0 26 0 100%
    h2/stream.py 356 0 76 0 100%
    h2/utilities.py 76 0 44 0 100%
    ----------------------------------------------------------------
    TOTAL 1238 0 288 0 100%
    ================== 894 passed in 6.87 seconds ==================

    View Slide

  26. I/O Is
    Application-
    Specific

    View Slide

  27. Build A
    Toolbox

    View Slide

  28. Examples
    • hyper-h2, a HTTP/2 protocol
    stack.
    • http://python-hyper.org/h2
    • h11, for HTTP/1.1
    • https://github.com/njsmith/h11

    View Slide

  29. Help Us

    View Slide

  30. View Slide