Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python module in Rust

95963a44712b973e2fd7766baa419478?s=47 dv
June 08, 2017
50

Python module in Rust

95963a44712b973e2fd7766baa419478?s=128

dv

June 08, 2017
Tweet

Transcript

  1. Python module in Rust Chiu-Hsiang Hsu PyCon TW 2017

  2. Who am I • Software Engineer @ Deep Sentinel (My

    First Job !) • Last Year: First try for CAS, SymPy with codegen • GitHub: @wdv4758h
  3. Talk's Assumption Version: Python 3.6 Platform: Linux Your Skills: Some

    Python, Some C
  4. Fast overview of Rust ecosystem

  5. Rust Survey 2016 Python

  6. Rust Language: Why Rust Memory Safety without GC Compiled Language

    Strong, Static Type
  7. Who is using Rust ? https://www.rust-lang.org/friends.html

  8. Benchmarks Game http://benchmarksgame.alioth.debian.org/

  9. Benchmarks Game - Where is Python different weight class ...

  10. Benchmarks Game - Python http://benchmarksgame.alioth.debian.org/

  11. Rust is young May 15, 2015

  12. Rust 1.18.0 (2017-06-08) 6 week release cycle

  13. Crates (like PyPI) https://crates.io/

  14. Cargo (like pip + setuptools) http://doc.crates.io/

  15. rustup (like pyenv) https://www.rustup.rs/

  16. rustup (like pyenv) compiler version

  17. rustup (like pyenv) target platform

  18. RFCs (like PEP) https://github.com/rust-lang/rfcs

  19. Official Tutorial https://doc.rust-lang.org/book/

  20. Standard Library Documentation https://doc.rust-lang.org/std/

  21. Documentation for Crates You can build your docs by cargo

    doc API doc with your docstring autogen for your crates: https://docs.rs/
  22. Documentation for Crates https://docs.rs/

  23. Documentation for Crates https://docs.rs/

  24. Official Repos https://github.com/rust-lang/

  25. Official Repos https://github.com/rust-lang-nursery/

  26. Rust is a lower level language than Python

  27. Lower Level https://doc.rust-lang.org/std/fmt/index.html format! (like str.format)

  28. Lower Level . Rust iterator (like Python iterator) https://doc.rust-lang.org/std/iter/trait.Iterator.html

  29. Lower Level . Rust iterator (like Python iterator) https://doc.rust-lang.org/std/iter/trait.Iterator.html

  30. Lower Level . Rust iterator (like Python iterator) https://doc.rust-lang.org/std/iter/trait.Iterator.html

  31. Lower Level . Rust iterator (like Python iterator) https://doc.rust-lang.org/std/iter/trait.Iterator.html

  32. Lower Level .. Rust lambda (like Python lambda) https://doc.rust-lang.org/book/closures.html

  33. Lower Level ... Wait ! It doesn't look like that

    low level !?
  34. Lower Level ... Wait ! It doesn't look like that

    low level !? I pick what I want to show you :P
  35. Compiler

  36. Compiler

  37. Compiler

  38. Compiler

  39. Python Code • Pure Python Code • Python Code +

    Binary built from (C/C++/…) • Python Code + Other Language's Code
  40. Python + Binary Make Friends !

  41. How can Python play with binary • ctypes • CFFI

    • CPython Extension • ...
  42. ctypes • pip • IPython • PyInstaller • SymPy •

    prompt_toolkit • ... Cases CPython Extension • NumPy • Pillow • simplejson • TensorFlow • psutil • ... CFFI • Cryptography • psycopg2cffi • pyzmq • ...
  43. ctypes - quick view import ctypes from ctypes import c_char_p

    , c_void_p lib = ctypes.cdll.LoadLibrary("pycontw.so") lib.hello_world.argtypes = (c_void_p,) lib.hello.argtypes = (c_char_p,) lib.hello.restypes = c_char_p
  44. ctypes - flow C Code Shared Library Python Code ctypes

    libffi
  45. CFFI - quick view from cffi import FFI ffi =

    FFI() lib = ffi.dlopen("pycontw.so") ffi.cdef(""" void hello_world(); char* hello(const char* name); """)
  46. CFFI - flow (ABI level) C Code Shared Library Python

    Code CFFI libffi
  47. CPython Extension - import PyInit_pycontw() import pycontw Find Pure Python

    Package or Module Find pycontw.so
  48. CPython Extension - flow C Code Shared Library Python Code

    CPython C API
  49. CPython Extension - simple call path import Find <modulename>.so POSIX

    dlsym PyModule_Create() PyInit_<modulename>()
  50. CPython Extension - quick view static PyMethodDef pycontw_methods[] = {

    {"hello_world", hello_world, METH_VARARGS, NULL}, {"hello", hello, METH_VARARGS, NULL}, {NULL, NULL, 0, NULL} };
  51. CPython Extension - quick view static PyModuleDef pycontw_module = {

    PyModuleDef_HEAD_INIT, "pycontw", "PyCon TW 2017", -1, pycontw_methods, NULL, NULL, NULL, NULL };
  52. CPython Extension - quick view PyMODINIT_FUNC PyInit_pycontw() { return PyModule_Create(&pycontw_module);

    }
  53. CPython Extension - quick view And you need to implement

    functions with CPython C API
  54. CPython Extension - build from distutils.core import setup, Extension module1

    = Extension('pycontw', sources = ['cpython-extension-sample.c']) setup(name = 'pycontw', version = '1.0', description = 'This is a demo package', ext_modules = [module1])
  55. CPython Extension - build $ python setup.py build

  56. Back to things related to Python

  57. Rust with ctypes/CFFI ?

  58. Sure

  59. CPython + Rust: by ctypes or CFFI #[repr(C)] struct MyData

    { a: i32, // Other members... }
  60. CPython + Rust: by ctypes or CFFI extern "C" fn

    myfunc() { println!("It's PyCon TW 2017 !!!"); }
  61. CPython + Rust: by ctypes or CFFI • ctypes: use

    it like C • CFFI: provide a C header for Rust code
  62. Rust for CPython Extension ?

  63. not so easy, but possible

  64. CPython + Rust: by rust-cpython Repo: github.com/wdv4758h/pycontw-rust-module

  65. CPython Extension with rust-cpython Rust Code Shared Library Python Code

    rust-cpython CPython C API
  66. rust-cpython - quick view // things like Python "import" #[macro_use]

    extern crate cpython; use cpython::{PyObject, PyResult, Python};
  67. rust-cpython - quick view fn hello_world(py: Python) -> PyResult<PyObject> {

    println!("It's PyCon TW 2017"); Ok(py.None()) }
  68. rust-cpython - quick view fn hello(_: Python, name: &str) ->

    PyResult<String> { Ok(format!("It's PyCon TW 2017, hello {} !!!", name)) }
  69. rust-cpython - quick view py_module_initializer!(pycontw, initpycontw, PyInit_pycontw, |py, m| {

    m.add(py, "__doc__", "PyCon TW 2017")?; m.add(py, "hello", py_fn!(py, hello(name: &str)))?; m.add(py, "hello_world", py_fn!(py, hello_world()))?; Ok(()) });
  70. rust-cpython - Cargo.toml [package] name = "pycontw" version = "0.1.0"

    [lib] name = "pycontw" crate-type = ["cdylib"] [dependencies] cpython = { version = "0.1.0", features = ["extension-module"] }
  71. rust-cpython - build $ cargo build

  72. rust-cpython - build $ cargo build --release

  73. setuptools-rust https://github.com/PyO3/setuptools-rust

  74. setuptools-rust - setup.py from setuptools import setup from setuptools_rust import

    RustExtension setup( name='pycontw', version='2017', rust_extensions=[RustExtension('pycontw', './Cargo.toml')], packages=['pycontw'], # Rust extensions are not zip safe, just like C extensions zip_safe=False, )
  75. Build Your Rust CPython Extension $ python setup.py build

  76. CPython Extension in Rust Simple Examples

  77. Hello

  78. Print

  79. Hash

  80. Hash with random sate

  81. Rust Vec for Python

  82. Dropbox Brotli in Rust

  83. Dropbox Brotli in Rust Simple Wrapper

  84. Dropbox Brotli in Rust - Cargo.toml [dependencies] brotli = "1.0.8"

  85. Dropbox Brotli in Rust - src/lib.rs extern crate brotli as

    _brotli; use _brotli::enc::reader::CompressorReader; py_module_initializer!(pycontw, initpycontw, PyInit_pycontw, |py, m| { m.add(py, "brotli", py_fn!(py, brotli(data: Vec<u8>)))?; Ok(()) });
  86. Dropbox Brotli in Rust - src/lib.rs fn brotli(_: Python, data:

    Vec<u8>) -> PyResult<Vec<u8>> { let quality = 5_u32; let lg_window_size = 20_u32; let mut enc_data = vec![]; let mut reader = CompressorReader::new(data.as_slice(), 4096, quality, lg_window_size); let _ = reader.read_to_end(&mut enc_data); Ok(enc_data) }
  87. More Resource- Rust by Example http://rustbyexample.com/

  88. More Resource - Awesome Rust https://github.com/rust-unofficial/awesome-rust

  89. Future Work • Build module by CI for multiple platforms

    • Cross Compile module • More complete Python & Rust integration • ...
  90. Question ?