Upgrade to Pro — share decks privately, control downloads, hide ads and more …

vigneshwer dhinakaran - Pumping up Python modules using Rust

vigneshwer dhinakaran - Pumping up Python modules using Rust

If you’ve spent much time writing (or debugging) Python performance problems, you’ve probably had a hard time managing memory with its limited language support.

In this talk, we venture deep into the belly of the Rust Language to uncover the secret incantations for building high performance and memory safe Python extensions using Rust.

Rust has a lot to offer in terms of safety and performance for high-level programming languages such Python, Ruby, Js and more with its easy Foreign Function Interface capabilities which enables developers to easily develop bindings for foreign code.


PyCon 2018

May 11, 2018

More Decks by PyCon 2018

Other Decks in Programming


  1. Pumping Python modules using Rust PyCon US 2017 Vigneshwer Dhinakaran

    | Mozilla TechSpeaker | @dvigneshwer
  2. Need for native extension Performance Freedom in memory management Access

    to low-level APIs
  3. Solutions to native extension Python community & it’s strong lib

    ecosystem to the rescue
  4. Native extension aren’t new to Python devs

  5. Native extension aren’t new to Python devs

  6. Native extension aren’t new to Python devs

  7. Native extension aren’t new to Python devs

  8. Reality: Not many of us can write C/C++ extensions

  9. Writing C/C++ extension is scary & painful Manual memory management

    Segfaults Data Races
  10. Other options? • Cython • Numba • PyPy • ...

  11. What’s the main ask here for us? C/C++ ‘s Performance,

    Portability & Embeddability With Guaranteed Safety
  12. Rust is a system programming language that has great control

    like C/C++, delivers productivity like in Python, and is super safe.
  13. Zen of Rust Memory Safety without Garbage Collection Rich Typesystem

    High level iterators Freedom from data race Welcoming Community
  14. Facts & Figures

  15. Speeding up crypto on Wire desktop apps https://medium.com/@wireapp/speeding-up-crypto-on-wire-desktop-apps-3ff37fc98c3f Up to

    141× faster
  16. Fixing Python Performance with Rust https://blog.sentry.io/2016/10/19/fixing-python-performance-with-rust.html

  17. Snips Uses Rust to Build an Embedded Voice Assistant

  18. Tokio MiniHTTP The library is a proof-of-concept implementation of a

    simple HTTP/1.1 server using Tokio. http://aturon.github.io/blog/2016/08/11/futures/
  19. 2016 -18 For the third year in a row, Rust

    is stackoverflow's most loved language https://insights.stackoverflow.com/survey/2018/
  20. Friends of Rust Organizations running Rust in production. (https://www.rust-lang.org/en-US/friends.html)

  21. History: How did it all begin?

  22. Browsers Compete for performance & need Safety at its core

  23. None
  24. 24

  25. None
  26. Gradual adoption works.

  27. Rust is easy to learn!

  28. Variables

  29. Functions

  30. Flow control

  31. Flow control

  32. Looping

  33. Types

  34. Package Manager

  35. Ownership The act, state, or right of possessing something

  36. Garbage Collection Heap Memory Circle(5) Circle(5) Circle(5)

  37. Referencing Now both bookingService and repairService hold a reference to

  38. Same implementation in Rust Doesn't compile!

  39. Ownership • Rust moves ownership by default • The owner

    has the right to destroy the thing it owns • The memory is freed as soon as the owned variable goes out of it's scope • Hence vehicle may already be destroyed at the point when it's passed to repair_service • Rust catches these errors at compile time
  40. Borrowing The act, state, or right of possessing something

  41. Making it work in Rust Lending things is the key!

  42. Borrowing • A reference is passed without transferring ownership •

    One can borrow immutable (&) or mutable (&mut) but not both at the same time • Shared ownership can be achieved through special pointers with runtime checks
  43. Mutability Rules • All variables are immutable by default •

    Only one mutable reference at a time ( But as many immutable &’s as you want ) • Mutable references block all other access (The &mut must go out of scope before using other &’s )
  44. Yes !!! Safe memory management without GC

  45. High-level iterators in Rust

  46. High-level iterators Execution time : ~ 250 ms

  47. High-level iterators

  48. Rayon ~ 50 ms

  49. custom data type in Rust

  50. Struct • Structs are like lightweight classes • They can

    have fields • They can be instantiated
  51. Struct But not partially !

  52. Struct Rust has option enum for such cases. Every Option

    is either Some and contains a value, or None, and does not.
  53. Struct can have methods

  54. Traits Traits are Rust’s sole notion of interface Traits must

    be brought into scope with use It has to be looked up in a vtable at run time ~ DYNAMIC DISPATCH
  55. Generics talk is now generic with a Talk trait bound

    The compiler generates specialized code ~ Static Dispatch Zero -Cost Abstraction !!
  56. Developing & Shipping Rust ext. in Python modules

  57. How does this work? • A Foreign Function Interface is

    a way for a programming language to define functions and enable a different (foreign) programming language to call those functions • Based on platform-dependant C application binary interface • Rust produces efficient C bindings
  58. Rust is ideal for creating and using FFI

  59. Rust FFI capabilities • Building the project • Create the

  60. Interfacing & building the python module • build.py • setup.py

  61. Other libs • setuptools-rust https://pypi.org/project/setuptools-rust/ • milksnake https://github.com/getsentry/milksnake

  62. Rust-Cpython

  63. Rust-Cpython : Integration • Import the dependencies • Write the

    rust equivalent logic • Export the rust function
  64. Rust-Cpython : Benchmarks

  65. None
  66. Tools • rustup: the Rust toolchain installer • rustfmt :

    the Rust code formatting tool • rust-clippy : Rust code linter • Cargo : package manager • ...
  67. 67 Community Photo credit: David McSpadden https://www.flickr.com/photos/familyclan/15535822737/

  68. 68 RFC Process

  69. 69

  70. Summary

  71. When should you use Rust? • Rewrite modules of your

    code that are computationally heavy (Complex math optimizations etc) • For accessing the hardware(ex :- IOT applications, device drivers, GPU’s etc) • Implementing advance concurrency paradigms • Python to Rust calls are non-trivial, so be wise!
  72. What does Rust have to offer? • Modern C/C++ replacement

    • Memory Safety without Garbage Collection • No runtime needed • Rich Typesystem • Strong Functional Programming influence
  73. Feel Awesome

  74. Thanks! @dvigneshwer dvigneshwer@gmail.com https://tinyurl.com/PyConUS2018