Python module in Rust

Transcript

1. Python module in Rust Chiu-Hsiang Hsu PyCon TW 2017
2. Who am I • Software Engineer @ Deep Sentinel (My First Job !) • Last Year: First try for CAS, SymPy with codegen • GitHub: @wdv4758h
3. Talk's Assumption Version: Python 3.6 Platform: Linux Your Skills: Some Python, Some C
4. Fast overview of Rust ecosystem
5. Rust Survey 2016 Python
6. Rust Language: Why Rust Memory Safety without GC Compiled Language Strong, Static Type
7. Who is using Rust ? https://www.rust-lang.org/friends.html
8. Benchmarks Game http://benchmarksgame.alioth.debian.org/
9. Benchmarks Game - Where is Python different weight class ...
10. Benchmarks Game - Python http://benchmarksgame.alioth.debian.org/
11. Rust is young May 15, 2015
12. Rust 1.18.0 (2017-06-08) 6 week release cycle
13. Crates (like PyPI) https://crates.io/
14. Cargo (like pip + setuptools) http://doc.crates.io/
15. rustup (like pyenv) https://www.rustup.rs/
16. rustup (like pyenv) compiler version
17. rustup (like pyenv) target platform
18. RFCs (like PEP) https://github.com/rust-lang/rfcs
19. Official Tutorial https://doc.rust-lang.org/book/
20. Standard Library Documentation https://doc.rust-lang.org/std/
21. Documentation for Crates You can build your docs by cargo doc API doc with your docstring autogen for your crates: https://docs.rs/
22. Documentation for Crates https://docs.rs/
23. Documentation for Crates https://docs.rs/
24. Official Repos https://github.com/rust-lang/
25. Official Repos https://github.com/rust-lang-nursery/
26. Rust is a lower level language than Python
27. Lower Level https://doc.rust-lang.org/std/fmt/index.html format! (like str.format)
28. Lower Level . Rust iterator (like Python iterator) https://doc.rust-lang.org/std/iter/trait.Iterator.html
29. Lower Level . Rust iterator (like Python iterator) https://doc.rust-lang.org/std/iter/trait.Iterator.html
30. Lower Level . Rust iterator (like Python iterator) https://doc.rust-lang.org/std/iter/trait.Iterator.html
31. Lower Level . Rust iterator (like Python iterator) https://doc.rust-lang.org/std/iter/trait.Iterator.html
32. Lower Level .. Rust lambda (like Python lambda) https://doc.rust-lang.org/book/closures.html
33. Lower Level ... Wait ! It doesn't look like that low level !?
34. Lower Level ... Wait ! It doesn't look like that low level !? I pick what I want to show you :P
35. Compiler
36. Compiler
37. Compiler
38. Compiler
39. Python Code • Pure Python Code • Python Code + Binary built from (C/C++/…) • Python Code + Other Language's Code
40. Python + Binary Make Friends !
41. How can Python play with binary • ctypes • CFFI • CPython Extension • ...
42. ctypes • pip • IPython • PyInstaller • SymPy • prompt_toolkit • ... Cases CPython Extension • NumPy • Pillow • simplejson • TensorFlow • psutil • ... CFFI • Cryptography • psycopg2cffi • pyzmq • ...
43. ctypes - quick view import ctypes from ctypes import c_char_p , c_void_p lib = ctypes.cdll.LoadLibrary("pycontw.so") lib.hello_world.argtypes = (c_void_p,) lib.hello.argtypes = (c_char_p,) lib.hello.restypes = c_char_p
44. ctypes - flow C Code Shared Library Python Code ctypes libffi
45. CFFI - quick view from cffi import FFI ffi = FFI() lib = ffi.dlopen("pycontw.so") ffi.cdef(""" void hello_world(); char* hello(const char* name); """)
46. CFFI - flow (ABI level) C Code Shared Library Python Code CFFI libffi
47. CPython Extension - import PyInit_pycontw() import pycontw Find Pure Python Package or Module Find pycontw.so
48. CPython Extension - flow C Code Shared Library Python Code CPython C API
49. CPython Extension - simple call path import Find <modulename>.so POSIX dlsym PyModule_Create() PyInit_<modulename>()
50. CPython Extension - quick view static PyMethodDef pycontw_methods[] = { {"hello_world", hello_world, METH_VARARGS, NULL}, {"hello", hello, METH_VARARGS, NULL}, {NULL, NULL, 0, NULL} };
51. CPython Extension - quick view static PyModuleDef pycontw_module = { PyModuleDef_HEAD_INIT, "pycontw", "PyCon TW 2017", -1, pycontw_methods, NULL, NULL, NULL, NULL };
52. CPython Extension - quick view PyMODINIT_FUNC PyInit_pycontw() { return PyModule_Create(&pycontw_module); }
53. CPython Extension - quick view And you need to implement functions with CPython C API
54. CPython Extension - build from distutils.core import setup, Extension module1 = Extension('pycontw', sources = ['cpython-extension-sample.c']) setup(name = 'pycontw', version = '1.0', description = 'This is a demo package', ext_modules = [module1])
55. CPython Extension - build $ python setup.py build
56. Back to things related to Python
57. Rust with ctypes/CFFI ?
58. Sure
59. CPython + Rust: by ctypes or CFFI #[repr(C)] struct MyData { a: i32, // Other members... }
60. CPython + Rust: by ctypes or CFFI extern "C" fn myfunc() { println!("It's PyCon TW 2017 !!!"); }
61. CPython + Rust: by ctypes or CFFI • ctypes: use it like C • CFFI: provide a C header for Rust code
62. Rust for CPython Extension ?
63. not so easy, but possible
64. CPython + Rust: by rust-cpython Repo: github.com/wdv4758h/pycontw-rust-module
65. CPython Extension with rust-cpython Rust Code Shared Library Python Code rust-cpython CPython C API
66. rust-cpython - quick view // things like Python "import" #[macro_use] extern crate cpython; use cpython::{PyObject, PyResult, Python};
67. rust-cpython - quick view fn hello_world(py: Python) -> PyResult<PyObject> { println!("It's PyCon TW 2017"); Ok(py.None()) }
68. rust-cpython - quick view fn hello(_: Python, name: &str) -> PyResult<String> { Ok(format!("It's PyCon TW 2017, hello {} !!!", name)) }
69. rust-cpython - quick view py_module_initializer!(pycontw, initpycontw, PyInit_pycontw, |py, m| { m.add(py, "__doc__", "PyCon TW 2017")?; m.add(py, "hello", py_fn!(py, hello(name: &str)))?; m.add(py, "hello_world", py_fn!(py, hello_world()))?; Ok(()) });
70. rust-cpython - Cargo.toml [package] name = "pycontw" version = "0.1.0" [lib] name = "pycontw" crate-type = ["cdylib"] [dependencies] cpython = { version = "0.1.0", features = ["extension-module"] }
71. rust-cpython - build $ cargo build
72. rust-cpython - build $ cargo build --release
73. setuptools-rust https://github.com/PyO3/setuptools-rust
74. setuptools-rust - setup.py from setuptools import setup from setuptools_rust import RustExtension setup( name='pycontw', version='2017', rust_extensions=[RustExtension('pycontw', './Cargo.toml')], packages=['pycontw'], # Rust extensions are not zip safe, just like C extensions zip_safe=False, )
75. Build Your Rust CPython Extension $ python setup.py build
76. CPython Extension in Rust Simple Examples
77. Hello
78. Print
79. Hash
80. Hash with random sate
81. Rust Vec for Python
82. Dropbox Brotli in Rust
83. Dropbox Brotli in Rust Simple Wrapper
84. Dropbox Brotli in Rust - Cargo.toml [dependencies] brotli = "1.0.8"
85. Dropbox Brotli in Rust - src/lib.rs extern crate brotli as _brotli; use _brotli::enc::reader::CompressorReader; py_module_initializer!(pycontw, initpycontw, PyInit_pycontw, |py, m| { m.add(py, "brotli", py_fn!(py, brotli(data: Vec<u8>)))?; Ok(()) });
86. Dropbox Brotli in Rust - src/lib.rs fn brotli(_: Python, data: Vec<u8>) -> PyResult<Vec<u8>> { let quality = 5_u32; let lg_window_size = 20_u32; let mut enc_data = vec![]; let mut reader = CompressorReader::new(data.as_slice(), 4096, quality, lg_window_size); let _ = reader.read_to_end(&mut enc_data); Ok(enc_data) }
87. More Resource- Rust by Example http://rustbyexample.com/
88. More Resource - Awesome Rust https://github.com/rust-unofficial/awesome-rust
89. Future Work • Build module by CI for multiple platforms • Cross Compile module • More complete Python & Rust integration • ...
90. Question ?