Slide 1
Slide 1 text
Rust
Rust と
と Python
Python
PyCon
PyCon J
JP
P 2018
2018
2018.09.17 Hideo Hattori
2018.09.17 Hideo Hattori
1
Slide 2
Slide 2 text
whoami
whoami
Software Engineer @ Tech Bureau, Corp.
using Python from version 1.5 ❤
@hhatto
2
Slide 3
Slide 3 text
My Python Project
My Python Project
etc...
genzshcomp
pgmagick
autopep8
aiogrn
pyramid_flamegraph
3
Slide 4
Slide 4 text
AGENDA
AGENDA
What is Rust?
Why use Rust with Python?
Example
Pros & Cons
4
Slide 5
Slide 5 text
What is Rust?
What is Rust?
5
Slide 6
Slide 6 text
6
Slide 7
Slide 7 text
Hello World
Hello World
fn main() {
let name = "Rust";
println!("Hello {}!", name);
}
7
Slide 8
Slide 8 text
version 1.29.0（2018.09.11）
Memory Safety
Blazingly Fast
Rust
Rust
8
Slide 9
Slide 9 text
Installation
Installation
$ curl https://sh.rustup.rs -sSf | sh
9
Slide 10
Slide 10 text
rustup
rustup
Installer
like venv
, pyenv
, ...
stable and nightly
rustup install nightly
10
Slide 11
Slide 11 text
cargo
cargo
&
&
package manager & package repository
like setup.py
, pip
& PyPI.org
crates.io
crates.io
11
Slide 12
Slide 12 text
cargo
cargo
- Start Project
- Start Project
$ cargo new myproject
$ tree myproject
myproject
├── Cargo.toml
└── src
└── main.rs
12
Slide 13
Slide 13 text
cargo
cargo
- Build
- Build
$ cargo build
$ cargo run
Hello Rust!
13
Slide 14
Slide 14 text
Memory Safety
Memory Safety
Ownership
Borrowing
14
Slide 15
Slide 15 text
Ownership and Borrowing
Ownership and Borrowing
fn main() {
let a = vec![1, 2, 3];
let b = a;
println!("a={:?}", a);
println!("b={:?}", b);
}
15
Slide 16
Slide 16 text
$ rustc ownership.rs
error[E0382]: use of moved value: `a`
--> ownership.rs:5:24
|
3 | let b = a;
| - value moved here
4 |
5 | println!("a={:?}", a);
| ^ value used here after move
|
=
16
Slide 17
Slide 17 text
fn main() {
let a = vec![1, 2, 3];
let b = &a;
println!("a={:?}", a);
println!("b={:?}", b);
}
17
Slide 18
Slide 18 text
Concurrency with
Concurrency with std::thread
std::thread
use std::thread;
fn main() {
let mut threads = vec![];
for i in 0..4 {
threads.push(thread::spawn(|| {
println!("this is thread number {}", i);
}));
}
for thread in threads {
let _ = thread.join();
}
}
18
Slide 19
Slide 19 text
error[E0373]: closure may outlive the current function, but it borrows `i`, which is owned
--> concurrency.rs:7:36
|
7 | threads.push(thread::spawn(|| {
| ^^ may outlive borrowed value `i`
8 | println!("this is thread number {}", i);
| - `i` is borrowed here
help: to force the closure to take ownership of `i` (and any other referenced variables), u
|
7 | threads.push(thread::spawn(move || {
| ^^^^^^^
error: aborting due to previous error
For more information about this error, try `rustc --explain E0373`.
19
Slide 20
Slide 20 text
use std::thread;
fn main() {
let mut threads = vec![];
for i in 0..4 {
threads.push(thread::spawn(move || {
println!("this is thread number {}", i);
}));
}
for thread in threads {
let _ = thread.join();
}
}
20
Slide 21
Slide 21 text
Blazingly Fast
Blazingly Fast
minimal runtime
zero-cost abstractions
std::thread (rayon, crossbeam, ...)
The computer language Benchmarks Game -
Rust vs C++ g++
21
Slide 22
Slide 22 text
Other feature
Other feature
Traits
Pattern Match
Macros
...
22
Slide 23
Slide 23 text
Uses
Uses
Command-Line Tools
Web Application
Extension Modules
...
23
Slide 24
Slide 24 text
Command-Line Tools
Command-Line Tools
- like grep tool
- fuzzy finder in Rust
- Replacement for ls written in Rust
- blazingly fast CLOC program
ripgrep
skim
exa
tokei
24
Slide 25
Slide 25 text
Web Application
Web Application
https://www.techempower.com/benchmarks/#section=data-r15&hw=ph&test=plaintext
25
Slide 26
Slide 26 text
Extension Modules
Extension Modules
Redis, Nginx, Varnish, etc...
Programming Language.
Python, Ruby, etc...
26
Slide 27
Slide 27 text
Python
Python
27
Slide 28
Slide 28 text
Why use Rust with
Why use Rust with
Python?
Python?
28
Slide 29
Slide 29 text
Motivation
Motivation
more fast ... C Extension
more safety write Extension modules
use Rust crates in Python world
29
Slide 30
Slide 30 text
How to implement
How to implement
ctypes + shared library
rust-cpython
PyO3
30
Slide 31
Slide 31 text
PyO3
PyO3
（ぱいおーすりー）
31
Slide 32
Slide 32 text
Requrements
Requrements
Rust（Nightly）
setuptools-rust
32
Slide 33
Slide 33 text
Start Python Extension Project
Start Python Extension Project
$ rustup default nightly
$ cargo new --lib example
$ cd example
33
Slide 34
Slide 34 text
$ tree example
example
├── Cargo.toml
├── setup.py
└── src
└── main.rs
34
Slide 35
Slide 35 text
Cargo.toml
Cargo.toml
[package]
name = "example"
version = "0.1.0"
authors = ["Hideo Hattori"]
[lib]
name = "example"
crate-type = ["cdylib"]
[dependencies]
[dependencies.pyo3]
version = "0.4"
features = ["extension-module"]
35
Slide 36
Slide 36 text
setup.py
setup.py
from setuptools import setup
from setuptools_rust import Binding, RustExtension
setup(name='example',
version='0.1',
rust_extensions=[
RustExtension('example', 'Cargo.toml',
binding=Binding.PyO3)],
zip_safe=False)
36
Slide 37
Slide 37 text
src/lib.rs
src/lib.rs
#![feature(specialization)]
extern crate pyo3;
use pyo3::prelude::*;
use pyo3::py::modinit as pymodinit;
#[pymodinit(example)]
fn init_example(py: Python, m: &PyModule) -> PyResult<()> {
#[pyfn(m, "hello")]
fn hello(name: &str) -> PyResult<()> {
println!("Hello {}", name);
Ok(())
}
Ok(())
}
37
Slide 38
Slide 38 text
Build
Build
$ pip install .
38
Slide 39
Slide 39 text
Let s use!!
Let s use!!
$ python
>>> import example
>>> example.hello()
Traceback (most recent call last):
File "", line 1, in
TypeError: Required argument ('name') (pos 1) not found
>>> example.hello('Rust')
Hello Rust
39
Slide 40
Slide 40 text
Packaging
Packaging
use pyo3-pack
40
Slide 41
Slide 41 text
Example of real world
Example of real world
41
Slide 42
Slide 42 text
fast-woothee-python
fast-woothee-python
Multi-Language User-Agent Strings Parser
（Python)
（Rust）
（Rust+Python)
Project Woothee
woothee-python
woothee-rust
fast-woothee-python
42
Slide 43
Slide 43 text
Benchmark
Benchmark
## Ranking real
uap 0.0813 (100.0) ********************
fast-woothee 1.0989 ( 7.4) *
woothee 3.2219 ( 2.5) *
uap(non-cache) 64.3114 ( 0.1)
## Matrix real [01] [02] [03] [04]
[01] uap 0.0813 100.0 1352.1 3964.4 79132.1
[02] fast-woothee 1.0989 7.4 100.0 293.2 5852.5
[03] woothee 3.2219 2.5 34.1 100.0 1996.1
[04] uap(non-cache) 64.3114 0.1 1.7 5.0 100.0
https://gist.github.com/hhatto/c951a981e8a3ee4d1bbcf96cb93d5f5e
43
Slide 44
Slide 44 text
wrapper of
Python s Class written in Rust
fcsv.Reader
fcsv.Writer
fcsv
fcsv
Rust s csv crate
44
Slide 45
Slide 45 text
Implement Python Class
Implement Python Class
use pyo3::prelude::*;
#[pyclass]
struct Writer {
_wtr: csv::Writer>,
token: PyToken,
}
#[pymodinit(_fcsv)]
fn init_mod(_py: Python, m: &PyModule) -> PyResult<()> {
m.add_class::()?;
Ok(())
}
45
Slide 46
Slide 46 text
Implement Python Class
Implement Python Class
#[pymethods]
impl Writer {
#[new]
fn __new__(obj: &PyRawObject,
path: String, kwargs: Option<&PyDict>) -> PyResult<()> {
:
obj.init(|t| Writer {
_wtr: wtr,
token: t,
})
}
fn writerow(&mut self, py: Python, arg: PyObject) -> PyResult<()> {
:
Ok(())
}
}
46
Slide 47
Slide 47 text
Implement Python Iterator
Implement Python Iterator
#[pyproto]
impl PyIterProtocol for Reader {
fn __iter__(&mut self) -> PyResult {
Ok(self.into())
}
fn __next__(&mut self) -> PyResult> {
let mut record = csv::StringRecord::new();
match self._rdr.read_record(&mut record) {
Ok(true) => {
let py = self.py();
Ok(Some(record.iter().collect::>().to_object(py)))
}
_ => Err(exc::StopIteration::new("stop")),
}
}
}
47
Slide 48
Slide 48 text
use this
use this
# example.py
import fcsv
for row in fcsv.reader('hello.csv'):
print(row)
48
Slide 49
Slide 49 text
Benchmark of
Benchmark of Writer
Writer
## Ranking real
fcsv.writerows 0.0519 (100.0) ********************
fcsv.writerow 0.0677 ( 76.7) ***************
unicodecsv.writerows 1.3566 ( 3.8) *
std.writerows 1.4500 ( 3.6) *
## Matrix real [01] [02] [03] [04]
[01] fcsv.writerows 0.0519 100.0 130.4 2614.4 2794.3
[02] fcsv.writerow 0.0677 76.7 100.0 2005.3 2143.4
[03] unicodecsv.writerows 1.3566 3.8 5.0 100.0 106.9
[04] std.writerows 1.4500 3.6 4.7 93.6 100.0
https://gist.github.com/hhatto/a101f904e516c0ea8519cc5a50fcf586
49
Slide 50
Slide 50 text
Benchmark of
Benchmark of Reader
Reader
## Ranking real
fcsv.reader 0.5444 (100.0) ********************
unicodecsv.reader 0.6474 ( 84.1) *****************
std.reader 0.7892 ( 69.0) **************
## Matrix real [01] [02] [03]
[01] fcsv.reader 0.5444 100.0 118.9 145.0
[02] unicodecsv.reader 0.6474 84.1 100.0 121.9
[03] std.reader 0.7892 69.0 82.0 100.0
https://gist.github.com/hhatto/a101f904e516c0ea8519cc5a50fcf586
50
Slide 51
Slide 51 text
Python s os.path
module re-written in Rust
already rewrite in Rust about 50%
fpath
fpath
51
Slide 52
Slide 52 text
memchr, memrchr
memchr, memrchr
use
fast character search
memchr crate
52
Slide 53
Slide 53 text
fn _basename(path_str: &str, is_bytes: bool) -> PyObject {
let i = match memchr::memrchr(MAIN_SEPARATOR as u8, path_str.as_bytes()) {
Some(v) => v + 1,
None => 0,
};
:
}
53
Slide 54
Slide 54 text
Benchmark
Benchmark
methodname % real[p,r] user[p,r] sys[p,r] n
abspath 45.53% 10.15s, 5.53s 6.86s, 2.81s 3.25s, 2.69s 100000
basename 53.52% 0.71s, 0.33s 0.70s, 0.33s 0.00s, 0.00s 100000
dirname 57.43% 1.02s, 0.43s 1.01s, 0.43s 0.00s, 0.00s 100000
isabs 56.55% 0.59s, 0.25s 0.59s, 0.25s 0.00s, 0.00s 100000
islink 0.25% 3.78s, 3.77s 0.01s, 0.01s 0.01s, 0.00s 50
exists 0.25% 3.78s, 3.77s 0.01s, 0.01s 0.01s, 0.00s 50
lexists 0.70% 3.77s, 3.74s 0.01s, 0.01s 0.01s, 0.00s 50
54
Slide 55
Slide 55 text
Benchmark (2)
Benchmark (2)
methodname % real[p,r] user[p,r] sys[p,r] n
split 53.85% 1.17s, 0.54s 1.17s, 0.54s 0.00s, 0.00s 100000
splitext 62.02% 1.22s, 0.46s 1.21s, 0.46s 0.00s, 0.00s 100000
relpath 52.97% 0.02s, 0.01s 0.01s, 0.01s 0.01s, 0.00s 50
normpath 57.27% 2.02s, 0.86s 2.01s, 0.86s 0.00s, 0.00s 100000
realpath 1.08% 13.39s, 13.25s 0.05s, 0.02s 0.02s, 0.03s 50
join 23.01% 0.24s, 0.19s 0.24s, 0.18s 0.00s, 0.00s 100000
expanduser 67.45% 1.50s, 0.49s 1.49s, 0.48s 0.00s, 0.00s 100000
expandvars 61.37% 1.21s, 0.47s 1.19s, 0.47s 0.00s, 0.00s 100000
55
Slide 56
Slide 56 text
Pros & Cons
Pros & Cons
56
Slide 57
Slide 57 text
Pros
Pros
Good Performance
better than C （in my opinion... ）
use Rust packages(creates)
57
Slide 58
Slide 58 text
Cons
Cons
unstable
Rust s Learning curve
58
Slide 59
Slide 59 text
Conclusion
Conclusion
59
Slide 60
Slide 60 text
Thank you!!
Thank you!!
60