$30 off During Our Annual Pro Sale. View Details »

Deep Postgres Extensions in Rust | PGConf EU | Jeff Davis

Citus Data
October 16, 2019

Deep Postgres Extensions in Rust | PGConf EU | Jeff Davis

Postgres relies heavily on an extension ecosystem, but that is almost 100% dependent on C; which cuts out developers, libraries, and ideas from the world of Postgres. postgres-extension.rs changes that by supporting development of extensions in Rust. Rust is a memory-safe language that integrates nicely in any environment, has powerful libraries, a vibrant ecosystem, and a prolific developer community.

Rust is a unique language because it supports high-level features but all the magic happens at compile-time, and the resulting code is not dependent on an intrusive or bulky runtime. That makes it ideal for integrating with postgres, which has a lot of its own runtime, like memory contexts and signal handlers. postgres-extension.rs offers this integration, allowing the development of extensions in rust, even if deeply-integrated into the postgres internals, and helping handle tricky issues like error handling. This is done through a collection of Rust function declarations, macros, and utility functions that allow rust code to call into postgres, and safely handle resulting errors.

Citus Data

October 16, 2019
Tweet

More Decks by Citus Data

Other Decks in Technology

Transcript

  1. Deep Postgres Extensions in Rust:
    postgres-extension.rs
    Jeff Davis
    Citus Data / Microsoft

    View Slide

  2. 2
    Motivation

    Postgres relies on an ecosystem of extensions

    This is a good thing!

    Extensions allow domain-specific or
    experimental development

    Encourage new developers to get involved and
    new types of extension development

    Rust offers a different language and
    environment

    And brings new ideas!

    View Slide

  3. 3
    Why Rust?

    Minimal runtime like C:

    No garbage collector or refcounting

    No “extra” code

    No “extra” data held in structs

    Not even a vtable pointer!

    Modern features, safety

    Growing developer community

    Awesome ecosystem

    View Slide

  4. 4
    The Postgres World is C

    Real extensions used to require C:

    Foreign Data Wrappers

    Custom Data Types

    Index and Sort Support Functions

    Background Workers

    UDFs calling internal functions

    View Slide

  5. 5
    What About Procedural Languages?

    PL/pgSQL, Perl, Python, v8, etc.

    Essentially sandboxes

    Only for UDFs and SPI

    SPI: Server Programming Interface allows execution of
    arbitrary SQL within a UDF

    We need something more

    View Slide

  6. 6
    Let’s see what rust can do

    Go beyond the Rust marketing and see how to
    use it to work with a complex system like
    postgres:

    Memory Contexts

    Error handling using setjmp/longjmp

    Global variables

    Intricate APIs

    View Slide

  7. 7
    So what is postgres-extension.rs?

    Allows close integration into the backend as an
    extension, just like C

    But it’s a pure Rust crate

    A collection of function declarations, macros,
    and utility functions

    Link seamlessly with C

    Only a subset of support for Postgres internals.
    Takes on the hardest challenges but many APIs
    are not yet implemented.

    View Slide

  8. 8
    Not a Client Driver, PL, or ORM

    There’s already an excellent pure-rust client
    library: rust-postgres

    Interact with postgresql from client application

    Thanks Steven Fackler!

    postgres-extension.rs is for deeper integration
    into the postgres server, like a C extension

    View Slide

  9. 9
    Features 1

    Can construct and operate directly on Postgres
    structures

    No copying or translation of data going from C to Rust or Rust
    to C

    Structure format is declared to be C-compatible

    Uses palloc()/pfree() for all heap allocations

    Even rust standard library calls

    inspect memory usage of rust code separate from other
    allocations

    elog()/ereport() support

    View Slide

  10. 10
    Features 2: Solves Error-Handling
    Mismatch

    If Rust panics, catch it before it returns to C,
    and turn it into a postgres ERROR

    If postgres calls rust, and rust calls a postgres
    function, and the postgres function throws an
    ERROR:

    catch it and turn it into a rust panic before skipping over any
    rust frames

    Important so that rust destructors are called

    View Slide

  11. 11
    Demo 1: UDFs and error handling

    DEMO

    View Slide

  12. 12
    Demo 2: UDF with SPI

    DEMO

    View Slide

  13. 13
    Demo 3: Concurrent Server with Tokio

    Tokio is an async framework

    Runtime for futures

    Build a background worker extension that:

    Accepts simple SQL statements from concurrent connections
    to port 8080

    Executes SQL with SPI

    Returns results

    View Slide

  14. 14
    Potential Sources of Overhead

    Array bounds checks

    Catching longjmp() at C→Rust boundary

    Catching rust panics at Rust→C boundary

    Converting rust strings to C strings

    View Slide

  15. 15
    C and Rust, not C or Rust

    Make rust developers more welcome

    Without making C developers less welcome

    View Slide

  16. 16
    Conclusion

    http://github.com/jeff-davis/postgres-extension.
    rs

    Try out writing extensions in a new language

    Only some internal postgres interfaces are
    supported for now

    Rust seems to have passed the test for real
    database internals

    Rust and Postgres have great potential
    together

    View Slide