Deep Postgres Extensions in Rust | PGCon 2019 | Jeff Davis

Postgres relies heavily on an extension ecosystem, but that is almost 100% dependent on C; which cuts out developers, libraries, and ideas from the world of Postgres. postgres-extension.rs changes that by supporting development of extensions in Rust. Rust is a memory-safe language that integrates nicely in any environment, has powerful libraries, a vibrant ecosystem, and a prolific developer community.

Rust is a unique language because it supports high-level features but all the magic happens at compile-time, and the resulting code is not dependent on an intrusive or bulky runtime. That makes it ideal for integrating with postgres, which has a lot of its own runtime, like memory contexts and signal handlers. postgres-extension.rs offers this integration, allowing the development of extensions in rust, even if deeply-integrated into the postgres internals, and helping handle tricky issues like error handling. This is done through a collection of Rust function declarations, macros, and utility functions that allow rust code to call into postgres, and safely handle resulting errors.

Transcript

1. Deep Postgres Extensions in Rust:
   postgres-extension.rs
   Jeff Davis
   Citus Data / Microsoft
   

   View Slide

2. 2
   Motivation
   ●
   Postgres relies on an ecosystem of extensions
   ●
   This is a good thing!
   ●
   Extensions allow domain-specific or
   experimental development
   ●
   We need to encourage new developers to get
   involved and new types of extension
   development
   ●
   Rust offers a different language and
   environment
   ●
   And brings new ideas!
   

   View Slide

3. 3
   Why Rust?
   ●
   More memory safety features than C
   ●
   Minimal runtime like C:
   ●
   No garbage collector or refcounting
   ●
   No “extra” code
   ●
   No “extra” data held in structs
   ●
   Not even a vtable pointer!
   ●
   Modern features
   ●
   Growing developer community
   ●
   Awesome ecosystem
   

   View Slide

4. 4
   The Postgres World is C
   ●
   Real extensions used to require C:
   ●
   Foreign Data Wrappers
   ●
   Custom Data Types
   ●
   Index and Sort Support Functions
   ●
   Background Workers
   ●
   UDFs calling internal functions
   

   View Slide

5. 5
   What About Procedural Languages?
   ●
   PL/pgSQL, Perl, Python, v8, etc.
   ●
   Essentially sandboxes
   ●
   Only for UDFs and SPI
   ●
   SPI: Server Programming Interface allows execution of
   arbitrary SQL within a UDF
   ●
   We need something more
   

   View Slide

6. 6
   Let’s see what rust can do
   ●
   Go beyond the Rust marketing and see how to
   use it to work with a complex system like
   postgres:
   ●
   Memory Contexts
   ●
   Error handling using setjmp/longjmp
   ●
   Global variables
   ●
   Intricate APIs
   

   View Slide

7. 7
   So what is postgres-extension.rs?
   ●
   Allows close integration into the backend as an
   extension, just like C
   ●
   But it’s a pure Rust crate
   ●
   A collection of function declarations, macros,
   and utility functions
   ●
   Link seamlessly with C
   

   View Slide

8. 8
   Not a Client Driver, PL, or ORM
   ●
   There’s already an excellent pure-rust client
   library: rust-postgres
   ●
   Interact with postgresql from client application
   ●
   Thanks Steven Fackler!
   ●
   postgres-extension.rs is for deeper integration
   into the postgres server, like a C extension
   

   View Slide

9. 9
   Features 1
   ●
   Can construct and operate directly on Postgres
   structures
   ●
   No copying or translation of data going from C to Rust or Rust
   to C
   ●
   Structure format is declared to be C-compatible
   ●
   Uses palloc()/pfree() for all heap allocations
   ●
   Even rust standard library calls
   ●
   Means you can safely pass back data that postgres will free
   with a memory context reset
   ●
   elog()/ereport() support
   

   View Slide

10. 10
    Features 2: Solves Error-Handling
    Mismatch
    ●
    If Rust panics, catch it before it returns to C,
    and turn it into a postgres ERROR
    ●
    If postgres calls rust, and rust calls a postgres
    function, and the postgres function throws an
    ERROR:
    ●
    catch it and turn it into a rust panic before skipping over any
    rust frames
    ●
    Important so that rust destructors are called
    ●
    This problem was a stumbling block preventing
    better support for C++ extensions, but is
    solved in postgres-extension.rs
    

    View Slide

11. 11
    Demo 1: UDFs and error handling
    ●
    DEMO
    

    View Slide

12. 12
    Demo 2: UDF with SPI
    ●
    DEMO
    

    View Slide

13. 13
    Demo 3: Concurrent Server with Tokio
    ●
    Tokio is an async framework
    ●
    Runtime for futures
    ●
    Build a background worker extension that:
    ●
    Accepts simple SQL statements from concurrent connections
    to port 8080
    ●
    Executes SQL with SPI
    ●
    Returns results
    

    View Slide

14. 14
    Potential Sources of Overhead
    ●
    Array bounds checks
    ●
    Catching longjmp() at C→Rust boundary
    ●
    Catching rust panics at Rust→C boundary
    ●
    Converting rust strings to C strings
    ●
    All avoidable if you are careful, just like C
    

    View Slide

15. 15
    C and Rust, not C or Rust
    ●
    Make rust developers more welcome
    ●
    Without making C developers less welcome
    ●
    Fitting for a bilingual city like Ottawa!
    

    View Slide

16. 16
    Conclusion
    ●
    http://github.com/jeff-davis/postgres-extension.
    rs
    ●
    Try out writing extensions in a new language
    ●
    Only some internal postgres interfaces are
    supported for now
    ●
    Rust seems to have passed the test for real
    database internals
    ●
    Rust and Postgres have great potential
    together
    

    View Slide