Wrapping Rust with PyO3 and Maturin

Video: https://video.linux.it/w/waFbUZBPndxW6EvHSq2bya?start=5m11&stop=1h19m
Codice: https://github.com/lu-zero/oruuid

Rust è un linguaggio di programmazione sistemico che si concentra sulla sicurezza della memoria e sulle prestazioni. È stato creato da Mozilla Research ed è stato progettato per essere utilizzato in ambienti ad alte prestazioni, come i browser web o le applicazioni server. Rust offre un'alternativa ai linguaggi di programmazione tradizionali come C e C++, fornendo una maggiore sicurezza della memoria senza sacrificare le prestazioni.

Se non hai mai toccato Rust prima d'ora, ti consigliamo vivamente di dare un'occhiata alla guida per iniziare con Rust per avere una panoramica del linguaggio e delle sue caratteristiche principali. Se vuoi provare Rust direttamente dal web, puoi utilizzare il playground di Rust.

Allo stesso modo, se non hai mai usato Python prima d'ora, ti consigliamo di dare un'occhiata alla documentazione di Python per avere una panoramica del linguaggio e delle sue caratteristiche principali. Se vuoi provare Python direttamente dal web, puoi utilizzare il playground di Python.

Durante questo workshop, esploriamo insieme l'uso di PyO3 e maturin per avvolgere le nostre rust crate e renderle utilizzabili da Python. Iniziamo con una breve introduzione a PyO3 e maturin, quindi passeremo alla loro installazione.

PyO3 è una libreria Rust che consente di scrivere estensioni Python in Rust. Con PyO3, puoi creare moduli Python scritti interamente in Rust e utilizzarli come qualsiasi altro modulo Python. Ciò significa che puoi sfruttare le prestazioni e la sicurezza della memoria di Rust all'interno del tuo codice Python.

maturin è uno strumento per creare pacchetti Python da crate Rust. Con maturin, puoi avvolgere le tue rust crate in un pacchetto Python e distribuirle su PyPI o utilizzarle come dipendenze nei tuoi progetti Python. Ciò significa che puoi sfruttare la potenza di Rust all'interno del tuo codice Python senza dover scrivere il codice in entrambi i linguaggi.

Successivamente, esploreremo insieme alcuni esempi pratici di come utilizzare queste librerie per avvolgere le nostre rust crate in pacchetti Python. Infine, cercheremo un piccolo crate che possa essere utile da avvolgere e proveremo a farlo insieme durante il workshop.

Questo workshop non è pensato per principianti assoluti in entrambi i linguaggi. Ci si aspetta che i partecipanti abbiano almeno una conoscenza minima di Rust e Python prima dell'accesso al workshop. Se sei nuovo a uno o entrambi i linguaggi, ti consigliamo vivamente di dare un'occhiata alla documentazione ufficiale e alle risorse sopra menzionate per prepararti al meglio per il workshop.

Luca Barbato — prolifico sviluppatore e contributor in progetti open source, quali FFmpeg, libav, VideoLAN, rust-av e NihAV

Transcript

1. Wrapping Rust with PyO3 and Maturin
2. Intro Wrapping Rust with PyO3 and Maturin - Luca Barbato
3. Who am I? Luca Barbato lu_zero@gentoo.org lu_zero@videolan.org Intro: Who am I? Wrapping Rust with PyO3 and Maturin - Luca Barbato
4. Why Rust? Among the most loved languages According to stack overflow Performance, Reliability, Productivity Pick 3 Write once, run almost everywhere Intro: Rust? Wrapping Rust with PyO3 and Maturin - Luca Barbato
5. Why Python? Very popular as well More than Rust? Easy to grasp Hard to master? Faster to write Slower to execute Intro: Python? Wrapping Rust with PyO3 and Maturin - Luca Barbato
6. Rust + Python = ?? It is common practice to rewrite python inner loops in C for speed But at what cost? Rust is as fast as C But with better abstractions Can Rust make Python developers more at home? Intro: Rust + Python? Wrapping Rust with PyO3 and Maturin - Luca Barbato
7. Some comparisons Rust fn foo(a: u16, b: u32) -> u64 { a as u64 + b as u64 } struct S { a: String, b: Vec<u8>, } impl S { fn blah(&self) { ... } } Python def foo(a: int, b: int) -> int: a + b class S: a: str b: bytes def blah(self) -> None: ... Intro: Rust + Python? Wrapping Rust with PyO3 and Maturin - Luca Barbato
8. They are different enough Not as different as C Python type annotations are similar to Rust types Iterators, Lamdas, and more exist cargo and pip / uv aren't that apart But not that similar Compile vs Interpret Traits vs Inheritance Static typing vs Dynamic typing Intro: Rust + Python? Wrapping Rust with PyO3 and Maturin - Luca Barbato
9. How to bridge between the two? Python has many ways to bind with C CFFI Cython and mypyc Rust can use the same C ABI easily If you want to go the hard way you can use directly cbindgen It can generate Cython bindings directly or you use cffi straight. If you want to have a better experience uniffi is good if you target multiple languages PyO3 if you want to interact with Python in both ways. In any case you want to build everything together maturin takes care of it for you Intro: Rust + Python? Wrapping Rust with PyO3 and Maturin - Luca Barbato
10. Maturin + PyO3 maturin is a Rust crate that supports PEP 621. So you can build, package and ship to pypi your Rust code. Incidentally it ship itself to pypi And since it supports PEP 621 you can use it with the usual tools (e.g. build) PyO3 is a bridge between Python and Rust that goes both direction You can expose Rust to Python You can call Python code from Rust It provides pre-made mappings between common types Wrapping Rust with PyO3 and Maturin - Luca Barbato
11. Case Study: oruuid oruuid is an implementation of Python's uuid using the uuid crate Riccardo noted that uuid_v7 is missing from Python. The API is good to start simple and gradually increase complexity Since the code itself is minimal we can use it to also try to measure the overhead Ingredients PyO3 to write the bindings maturin to build everying pytest to have a couple of tests going timeit to quickly get some numbers Wrapping Rust with PyO3 and Maturin - Luca Barbato
12. Setting up the project maturin has a new command we can use $ maturin new oruuid It will ask the kind of bindings, we are going to use PyO3 and we get ├── .github │ └── workflows │ └── CI.yml ├── .gitignore ├── Cargo.toml ├── pyproject.toml └── src └── lib.rs Notice we have a pyproject.toml and Cargo.toml generated for us. oruid: Setup Wrapping Rust with PyO3 and Maturin - Luca Barbato
13. We add a dependency on uuid: $ cargo add uuid --features v1,v4,v7 The crate let you enable specific implementation using the features system. We start enabling v1 , v4 and v7 . v1 and v7 both rely on a timestamp provided, or now is used as implicit input. v4 is just encoding a random number and in both Rust and Python it is a thin layer over the OS getrandom(2) . v7 is not implemented yet so it is our excuse to write all of this. We want to stay compatible with Python uuid Nothing to add here, it is part of the standard library. oruid: Setup Wrapping Rust with PyO3 and Maturin - Luca Barbato
14. The two APIs are fairly similar: Python class UUID: __slots__ = ('int', 'is_safe', '__weakref__') def __init__(self, hex=None, bytes=None, bytes_le=None, fields=None, int=None, version=None, *, is_safe=SafeUUID.unknown): ... def uuid1(node=None, clock_seq=None): ... if node is None: node = getnode() return UUID(fields=(time_low, time_mid, time_hi_version, clock_seq_hi_variant, clock_seq_low, node), version=1) def uuid4(): """Generate a random UUID.""" return UUID(bytes=os.urandom(16), version=4) Rust pub struct Uuid(Bytes); impl Uuid { ... pub const fn as_u128(&self) -> u128 { u128::from_be_bytes(*self.as_bytes()) } ... // in src/v1.rs pub fn new_v1(ts: Timestamp, node_id: &[u8; 6]) -> Self { ... } pub fn now_v1(node_id: &[u8; 6]) -> Self { let ts = Timestamp::now(crate::timestamp::context::shared_context()); Self::new_v1(ts, node_id) } ... // in src/v4.rs pub fn new_v4() -> Uuid { Uuid::from_u128( crate::rng::u128() & 0xFFFFFFFFFFFF4FFFBFFFFFFFFFFFFFFF | 0x40008000000000000000, ) } } oruid: API Wrapping Rust with PyO3 and Maturin - Luca Barbato
15. High level Ideally we should provide a uuid7 that returns a UUID The constructor for UUID can take a u128 (big endian) Uuid can provide a u128 We need to call the Rust Uuid::now_v7 , feed the result to the UUID(int=) and we are done! In steps we want to: i. Have our oruuid module ii. Add a uuid7() function to it iii. Make the function return the Python UUID correctly populated oruid: API Wrapping Rust with PyO3 and Maturin - Luca Barbato
16. Writing Rust for Python with PyO3 PyO3 provides all the building blocks we need An API to interact with the Python interpreter A way to map Rust types to Python types and back procedural macros to make all simpler We need to decorate our function with #[pyfunction] and register it to the module #[pyfunction] fn uuid7() -> PyResult<Py<PyAny>> { ... } #[pymodule] fn oruuid(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(uuid7, m)?)?; Ok(()) } oruid: Code Wrapping Rust with PyO3 and Maturin - Luca Barbato
17. Since we want to return the Python UUID we have to instantiate it #[pyfunction] fn uuid7() -> PyResult<Py<PyAny>> { let uuid = uuid::Uuid::now_v7(); Python::with_gil(|py| { let kwargs = PyDict::new_bound(py); kwargs.set_item("int", uuid.as_u128())?; let pyuuid = PyModule::import_bound(py, "uuid")?; pyuuid .getattr("UUID")? .call((), Some(&kwargs)) .map(|u| u.unbind()) }) } Note: for simplicity I'm glossing over receiving arguments oruid: Code Wrapping Rust with PyO3 and Maturin - Luca Barbato
18. Building using maturin So far we have our bit of code in src/lib.rs and Cargo.toml with uuid We should build it And also test it Time to use maturin build first let's set up our environment: $ python -m venv .venv $ . .venv/bin/activate we can use either maturin build $ maturin build And see if everything compiles oruid: Building and Testing Wrapping Rust with PyO3 and Maturin - Luca Barbato
19. Testing with pytest maturin is all about building If we want to test we can use any testing harness we like I picked pytest : $ pip install pytest Since we do not have a default layout I use test for Python and tests for Rust integration tests $ mkdir test/ $ touch test/__init__.py $ vim test/test_uuid7.py oruid: Building and Testing Wrapping Rust with PyO3 and Maturin - Luca Barbato
20. We cannot test much beside at least making sure it is version 7: # test/test_uuid7.py from oruuid import uuid7 def test_uuid7(): u = uuid7() assert u.version == 7 We use maturin develop to have our package reachable: $ maturin develop NOTE: if you want to benchmark remember to pass -r to it. If everything goes well a green test will greet us. $ pytest oruid: Building and Testing Wrapping Rust with PyO3 and Maturin - Luca Barbato
21. Exercise 1 Let's try to implement uuid4 by calling the Python code and exposing it as uuid4p . by calling the Rust code and exposing it as uuid4r . Let's benchmark both using timeit and compare with Python: $ maturin develop -r $ python -m timeit "import oruuid; oruuid.uuid4r()" $ python -m timeit "import oruuid; oruuid.uuid4p()" $ python -m timeit "import uuid; uuid.uuid4()" This way we can see the overhead oruid: Extra Wrapping Rust with PyO3 and Maturin - Luca Barbato
22. Exercise 2 Let's add a signature to v7 Rust has the Option enum for it PyO3 helpfully maps it to a arg=None by default #[pyo3(signature = {argument string})] can be used to express more complex signatures We have to wrap the Timestamp and ContextV7 structs and expose them Chopping the methods to those that apply to v7 . oruid: Extra Wrapping Rust with PyO3 and Maturin - Luca Barbato
23. Conclusion We saw how PyO3 makes simple to use Python from Rust and to expose Rust functions to Python We saw how maturin makes simple building Rust crates as Python packages Wrapping Rust with PyO3 and Maturin - Luca Barbato
24. Questions? Wrapping Rust with PyO3 and Maturin - Luca Barbato
25. Thank you! Wrapping Rust with PyO3 and Maturin - Luca Barbato