Upgrade to Pro — share decks privately, control downloads, hide ads and more …

RustとPython

46c820d33a8b4ae2bd341bbd027b35b2?s=47 Hideo Hattori
September 17, 2018

 RustとPython

PyCon JP 2018 9/17 talk session
Python Extension Module written in Rust.

46c820d33a8b4ae2bd341bbd027b35b2?s=128

Hideo Hattori

September 17, 2018
Tweet

Transcript

  1. Rust Rust と と Python Python PyCon PyCon J JP

    P 2018 2018 2018.09.17 Hideo Hattori 2018.09.17 Hideo Hattori 1
  2. whoami whoami Software Engineer @ Tech Bureau, Corp. using Python

    from version 1.5 ❤ @hhatto 2
  3. My Python Project My Python Project etc... genzshcomp pgmagick autopep8

    aiogrn pyramid_flamegraph 3
  4. AGENDA AGENDA What is Rust? Why use Rust with Python?

    Example Pros & Cons 4
  5. What is Rust? What is Rust? 5

  6. 6

  7. Hello World Hello World fn main() { let name =

    "Rust"; println!("Hello {}!", name); } 7
  8. version 1.29.0(2018.09.11) Memory Safety Blazingly Fast Rust Rust 8

  9. Installation Installation $ curl https://sh.rustup.rs -sSf | sh 9

  10. rustup rustup Installer like venv , pyenv , ... stable

    and nightly rustup install nightly 10
  11. cargo cargo & & package manager & package repository like

    setup.py , pip & PyPI.org crates.io crates.io 11
  12. cargo cargo - Start Project - Start Project $ cargo

    new myproject $ tree myproject myproject ├── Cargo.toml └── src └── main.rs 12
  13. cargo cargo - Build - Build $ cargo build $

    cargo run Hello Rust! 13
  14. Memory Safety Memory Safety Ownership Borrowing 14

  15. Ownership and Borrowing Ownership and Borrowing fn main() { let

    a = vec![1, 2, 3]; let b = a; println!("a={:?}", a); println!("b={:?}", b); } 15
  16. $ rustc ownership.rs error[E0382]: use of moved value: `a` -->

    ownership.rs:5:24 | 3 | let b = a; | - value moved here 4 | 5 | println!("a={:?}", a); | ^ value used here after move | = 16
  17. fn main() { let a = vec![1, 2, 3]; let

    b = &a; println!("a={:?}", a); println!("b={:?}", b); } 17
  18. Concurrency with Concurrency with std::thread std::thread use std::thread; fn main()

    { let mut threads = vec![]; for i in 0..4 { threads.push(thread::spawn(|| { println!("this is thread number {}", i); })); } for thread in threads { let _ = thread.join(); } } 18
  19. error[E0373]: closure may outlive the current function, but it borrows

    `i`, which is owned --> concurrency.rs:7:36 | 7 | threads.push(thread::spawn(|| { | ^^ may outlive borrowed value `i` 8 | println!("this is thread number {}", i); | - `i` is borrowed here help: to force the closure to take ownership of `i` (and any other referenced variables), u | 7 | threads.push(thread::spawn(move || { | ^^^^^^^ error: aborting due to previous error For more information about this error, try `rustc --explain E0373`. 19
  20. use std::thread; fn main() { let mut threads = vec![];

    for i in 0..4 { threads.push(thread::spawn(move || { println!("this is thread number {}", i); })); } for thread in threads { let _ = thread.join(); } } 20
  21. Blazingly Fast Blazingly Fast minimal runtime zero-cost abstractions std::thread (rayon,

    crossbeam, ...) The computer language Benchmarks Game - Rust vs C++ g++ 21
  22. Other feature Other feature Traits Pattern Match Macros ... 22

  23. Uses Uses Command-Line Tools Web Application Extension Modules ... 23

  24. Command-Line Tools Command-Line Tools - like grep tool - fuzzy

    finder in Rust - Replacement for ls written in Rust - blazingly fast CLOC program ripgrep skim exa tokei 24
  25. Web Application Web Application https://www.techempower.com/benchmarks/#section=data-r15&hw=ph&test=plaintext 25

  26. Extension Modules Extension Modules Redis, Nginx, Varnish, etc... Programming Language.

    Python, Ruby, etc... 26
  27. Python Python 27

  28. Why use Rust with Why use Rust with Python? Python?

    28
  29. Motivation Motivation more fast ... C Extension more safety write

    Extension modules use Rust crates in Python world 29
  30. How to implement How to implement ctypes + shared library

    rust-cpython PyO3 30
  31. PyO3 PyO3 (ぱいおーすりー) 31

  32. Requrements Requrements Rust(Nightly) setuptools-rust 32

  33. Start Python Extension Project Start Python Extension Project $ rustup

    default nightly $ cargo new --lib example $ cd example 33
  34. $ tree example example ├── Cargo.toml ├── setup.py └── src

    └── main.rs 34
  35. Cargo.toml Cargo.toml [package] name = "example" version = "0.1.0" authors

    = ["Hideo Hattori"] [lib] name = "example" crate-type = ["cdylib"] [dependencies] [dependencies.pyo3] version = "0.4" features = ["extension-module"] 35
  36. setup.py setup.py from setuptools import setup from setuptools_rust import Binding,

    RustExtension setup(name='example', version='0.1', rust_extensions=[ RustExtension('example', 'Cargo.toml', binding=Binding.PyO3)], zip_safe=False) 36
  37. src/lib.rs src/lib.rs #![feature(specialization)] extern crate pyo3; use pyo3::prelude::*; use pyo3::py::modinit

    as pymodinit; #[pymodinit(example)] fn init_example(py: Python, m: &PyModule) -> PyResult<()> { #[pyfn(m, "hello")] fn hello(name: &str) -> PyResult<()> { println!("Hello {}", name); Ok(()) } Ok(()) } 37
  38. Build Build $ pip install . 38

  39. Let s use!! Let s use!! $ python >>> import

    example >>> example.hello() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: Required argument ('name') (pos 1) not found >>> example.hello('Rust') Hello Rust 39
  40. Packaging Packaging use pyo3-pack 40

  41. Example of real world Example of real world 41

  42. fast-woothee-python fast-woothee-python Multi-Language User-Agent Strings Parser (Python) (Rust) (Rust+Python) Project

    Woothee woothee-python woothee-rust fast-woothee-python 42
  43. Benchmark Benchmark ## Ranking real uap 0.0813 (100.0) ******************** fast-woothee

    1.0989 ( 7.4) * woothee 3.2219 ( 2.5) * uap(non-cache) 64.3114 ( 0.1) ## Matrix real [01] [02] [03] [04] [01] uap 0.0813 100.0 1352.1 3964.4 79132.1 [02] fast-woothee 1.0989 7.4 100.0 293.2 5852.5 [03] woothee 3.2219 2.5 34.1 100.0 1996.1 [04] uap(non-cache) 64.3114 0.1 1.7 5.0 100.0 https://gist.github.com/hhatto/c951a981e8a3ee4d1bbcf96cb93d5f5e 43
  44. wrapper of Python s Class written in Rust fcsv.Reader fcsv.Writer

    fcsv fcsv Rust s csv crate 44
  45. Implement Python Class Implement Python Class use pyo3::prelude::*; #[pyclass] struct

    Writer { _wtr: csv::Writer<BufWriter<fs::File>>, token: PyToken, } #[pymodinit(_fcsv)] fn init_mod(_py: Python, m: &PyModule) -> PyResult<()> { m.add_class::<Writer>()?; Ok(()) } 45
  46. Implement Python Class Implement Python Class #[pymethods] impl Writer {

    #[new] fn __new__(obj: &PyRawObject, path: String, kwargs: Option<&PyDict>) -> PyResult<()> { : obj.init(|t| Writer { _wtr: wtr, token: t, }) } fn writerow(&mut self, py: Python, arg: PyObject) -> PyResult<()> { : Ok(()) } } 46
  47. Implement Python Iterator Implement Python Iterator #[pyproto] impl PyIterProtocol for

    Reader { fn __iter__(&mut self) -> PyResult<pyobject> { Ok(self.into()) } fn __next__(&mut self) -> PyResult<option<pyobject>> { let mut record = csv::StringRecord::new(); match self._rdr.read_record(&mut record) { Ok(true) => { let py = self.py(); Ok(Some(record.iter().collect::<vec<&str>>().to_object(py))) } _ => Err(exc::StopIteration::new("stop")), } } } 47
  48. use this use this # example.py import fcsv for row

    in fcsv.reader('hello.csv'): print(row) 48
  49. Benchmark of Benchmark of Writer Writer ## Ranking real fcsv.writerows

    0.0519 (100.0) ******************** fcsv.writerow 0.0677 ( 76.7) *************** unicodecsv.writerows 1.3566 ( 3.8) * std.writerows 1.4500 ( 3.6) * ## Matrix real [01] [02] [03] [04] [01] fcsv.writerows 0.0519 100.0 130.4 2614.4 2794.3 [02] fcsv.writerow 0.0677 76.7 100.0 2005.3 2143.4 [03] unicodecsv.writerows 1.3566 3.8 5.0 100.0 106.9 [04] std.writerows 1.4500 3.6 4.7 93.6 100.0 https://gist.github.com/hhatto/a101f904e516c0ea8519cc5a50fcf586 49
  50. Benchmark of Benchmark of Reader Reader ## Ranking real fcsv.reader

    0.5444 (100.0) ******************** unicodecsv.reader 0.6474 ( 84.1) ***************** std.reader 0.7892 ( 69.0) ************** ## Matrix real [01] [02] [03] [01] fcsv.reader 0.5444 100.0 118.9 145.0 [02] unicodecsv.reader 0.6474 84.1 100.0 121.9 [03] std.reader 0.7892 69.0 82.0 100.0 https://gist.github.com/hhatto/a101f904e516c0ea8519cc5a50fcf586 50
  51. Python s os.path module re-written in Rust already rewrite in

    Rust about 50% fpath fpath 51
  52. memchr, memrchr memchr, memrchr use fast character search memchr crate

    52
  53. fn _basename(path_str: &str, is_bytes: bool) -> PyObject { let i

    = match memchr::memrchr(MAIN_SEPARATOR as u8, path_str.as_bytes()) { Some(v) => v + 1, None => 0, }; : } 53
  54. Benchmark Benchmark methodname % real[p,r] user[p,r] sys[p,r] n abspath 45.53%

    10.15s, 5.53s 6.86s, 2.81s 3.25s, 2.69s 100000 basename 53.52% 0.71s, 0.33s 0.70s, 0.33s 0.00s, 0.00s 100000 dirname 57.43% 1.02s, 0.43s 1.01s, 0.43s 0.00s, 0.00s 100000 isabs 56.55% 0.59s, 0.25s 0.59s, 0.25s 0.00s, 0.00s 100000 islink 0.25% 3.78s, 3.77s 0.01s, 0.01s 0.01s, 0.00s 50 exists 0.25% 3.78s, 3.77s 0.01s, 0.01s 0.01s, 0.00s 50 lexists 0.70% 3.77s, 3.74s 0.01s, 0.01s 0.01s, 0.00s 50 54
  55. Benchmark (2) Benchmark (2) methodname % real[p,r] user[p,r] sys[p,r] n

    split 53.85% 1.17s, 0.54s 1.17s, 0.54s 0.00s, 0.00s 100000 splitext 62.02% 1.22s, 0.46s 1.21s, 0.46s 0.00s, 0.00s 100000 relpath 52.97% 0.02s, 0.01s 0.01s, 0.01s 0.01s, 0.00s 50 normpath 57.27% 2.02s, 0.86s 2.01s, 0.86s 0.00s, 0.00s 100000 realpath 1.08% 13.39s, 13.25s 0.05s, 0.02s 0.02s, 0.03s 50 join 23.01% 0.24s, 0.19s 0.24s, 0.18s 0.00s, 0.00s 100000 expanduser 67.45% 1.50s, 0.49s 1.49s, 0.48s 0.00s, 0.00s 100000 expandvars 61.37% 1.21s, 0.47s 1.19s, 0.47s 0.00s, 0.00s 100000 55
  56. Pros & Cons Pros & Cons 56

  57. Pros Pros Good Performance better than C (in my opinion...

    ) use Rust packages(creates) 57
  58. Cons Cons unstable Rust s Learning curve 58

  59. Conclusion Conclusion 59

  60. Thank you!! Thank you!! 60