hyperjson - the journey towards faster, safer JSON parsing in Python using Rust

During a recent Python Hackathon in Düsseldorf, Matthias somehow managed to sneak in some Rust code while writing a fast, safe Python module for encoding and decoding JSON. It passes (most of) the Python test suite for the json module and was unreasonably pleasant to write. Listen carefully as he tells the story of a little side-project that got out of hand and how Rust can help speed up even the most boring, highly-optimized tasks like parsing file formats in the future.

Transcript

1. View Slide

2. hydrojson
   

   View Slide

3. EXPECTATION...
   REALITY...
   

   View Slide

4. View Slide

5. "Hey, I wrote a JSON encoder!"
   

   View Slide

6. View Slide

7. https://www.cartoonstock.com/cartoonview.asp?catref=rdin355
   

   View Slide

8. View Slide

9. http://seriot.ch/parsing_json.php
   

   View Slide

10. https://bugs.python.org/issue21529
    

    View Slide

11. View Slide

12. View Slide

13. Parsing JSON - Hieronymus Bosch, 1501
    

    View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. View Slide

19. View Slide

20. -------------------------------------------------------------------------------
    Language files blank comment code
    -------------------------------------------------------------------------------
    JSON 3 0 0 1087568
    Python 15 404 148 1331
    Rust 2 63 65 468
    Markdown 1 46 0 107
    Bourne Shell 4 17 7 54
    make 1 9 0 32
    YAML 1 6 2 23
    TOML 1 5 0 20
    -------------------------------------------------------------------------------
    SUM: 28 550 222 1089603
    -------------------------------------------------------------------------------
    

    View Slide

21. >>> import lenrs
    >>> len("hello")
    5
    >>> len(["this", "is", "a", "test"])
    4
    >>> len(1)
    Traceback (most recent call last):
    File "", line 1, in
    TypeError: object of type 'int' has no len()
    

    View Slide

22. #![feature(proc_macro)]
    #![feature(proc_macro_path_invoc)]
    extern crate pyo3;
    use pyo3::prelude::*;
    #[py::modinit(_lenrs)]
    fn init(py: Python, m: &PyModule) -> PyResult<()> {
    #[pyfn(m, "len")]
    fn len(py: Python, obj: PyObject) -> PyResult {
    if let Ok(s) = obj.extract::(py) {
    return Ok(s.len().to_object(py));
    }
    if let Ok(v) = obj.extract::>(py) {
    return Ok(v.len().to_object(py));
    }
    Err(PyErr::new::("Not supported"))
    }
    Ok(())
    }
    Requires nightly Rust
    Module name in Python
    Function name in Python
    

    View Slide

23. Write your own Python Extension in Rust!

    

    https://www.youtube.com/watch?v=D9r__qxtRMQ
    

    View Slide

24. View Slide