Cory Benfield - Building Protocol Libraries The Right Way

One of the great strengths of the Python ecosystem is the enormous collection of powerful, flexible libraries. However, these libraries tend to suffer from one extremely common design flaw that mean that the work done is not easily re-usable or transferable. In this talk, we talk about how to build libraries that can be used as widely as possible, through the lens of the Python Hyper HTTP project.

https://us.pycon.org/2016/schedule/presentation/1743/

Transcript

1. Building Protocol
   Libraries
   The Right Way
   

   View Slide

2. Hi!
   

   View Slide

3. @lukasaoz
   @lukasa
   

   View Slide

4. View Slide

5. Python HTTP
   • Hyper
   • h2, hpack, priority, etc.
   • Requests + urllib3
   • PyCA (PyOpenSSL)
   

   View Slide

6. Why All Your Libraries
   Are Garbage, And All
   But One Of Mine Are
   Too
   

   View Slide

7. Python HTTP
   Rocks
   

   View Slide

8. • Requests/urllib3/HTTPie
   • gunicorn
   • aiohttp
   • Twisted
   • Tornado
   

   View Slide

9. ~0% Shared
   Code
   

   View Slide

10. Code Re-Use
    
    

    View Slide

11. I/O
    

    View Slide

12. • httplib: state machine and
    parser mixed with I/O
    • aiohttp/Tornado/Twisted:
    state machine and parser
    use concurrency primitives
    

    View Slide

13. Choice of I/O limits
    choice of libraries
    

    View Slide

14. So What?
    

    View Slide

15. Wasted Effort
    

    View Slide

16. Reduces
    Experimentation
    

    View Slide

17. Duplicate
    Bugs
    

    View Slide

18. Limits
    Optimisation
    

    View Slide

19. Avoiding
    This Problem
    

    View Slide

20. Don’t Do I/O
    

    View Slide

21. Alternative API
    events = handle_data(in_bytes)
    out_bytes = perform_action()
    

    View Slide

22. Not All-Purpose
    

    View Slide

23. Build
    Wrappers
    

    View Slide

24. Why Bother?
    

    View Slide

25. ===================== test session starts ======================
    ------- coverage: platform darwin, python 2.7.11-final-0 -------
    Name Stmts Miss Branch BrPart Cover Missing
    ----------------------------------------------------------------
    h2/__init__.py 2 0 0 0 100%
    h2/connection.py 505 0 120 0 100%
    h2/errors.py 17 0 0 0 100%
    h2/events.py 107 0 4 0 100%
    h2/exceptions.py 41 0 0 0 100%
    h2/frame_buffer.py 55 0 18 0 100%
    h2/settings.py 79 0 26 0 100%
    h2/stream.py 356 0 76 0 100%
    h2/utilities.py 76 0 44 0 100%
    ----------------------------------------------------------------
    TOTAL 1238 0 288 0 100%
    ================== 894 passed in 6.87 seconds ==================
    

    View Slide

26. I/O Is
    Application-
    Specific
    

    View Slide

27. Build A
    Toolbox
    

    View Slide

28. Examples
    • hyper-h2, a HTTP/2 protocol
    stack.
    • http://python-hyper.org/h2
    • h11, for HTTP/1.1
    • https://github.com/njsmith/h11
    

    View Slide

29. Help Us
    

    View Slide

30. View Slide