Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Barrel. Using Erlang to build a Database

Barrel. Using Erlang to build a Database

Barrel (https://barrel-db.org) is a modern document-oriented database in Erlang focusing on data locality (put/match the data next to you) and P2P with an effort to maintain a compatibility with the Apache CouchDB API.

In this talk, I will begin with an overview of Barrel, covering the problems that it is intended to solve and how its data model, transaction model, query model, and deployment model work together to solve those problems. I will then use Barrel to illustrate how to design and implement a production software in Erlang.

Benoit Chesneau

January 29, 2016
Tweet

More Decks by Benoit Chesneau

Other Decks in Technology

Transcript

  1. BARREL
    DISTRIBUTED DATABASE
    FOR THE MODERN WORLD
    https://barrel-db.org
    Erlang Factory Lite Brussel 2016

    View Slide

  2. VISION
    AND CONCEPT

    View Slide

  3. local database
    mobile
    sensor
    "cloud" database
    local
    database
    mobile
    DATA IS MOBILE

    View Slide

  4. share
    discover
    replicate
    PEER TO PEER (P2P)

    View Slide

  5. ▸ Local first
    ▸ Put/Match the data next to you
    ▸ Query Locally
    ▸ Replicate a view of the data you need

    View Slide

  6. WHAT
    IS BARREL

    View Slide

  7. WHAT IS BARREL
    ▸ a document database
    ▸ revision tree & atomic document updates
    ▸ changes feed
    ▸ replication between any nodes in both way
    ▸ views (~ map)

    View Slide

  8. AN EFFORT TO MAINTAIN
    THE COUCHDB API

    View Slide

  9. ▸ view changes
    ▸ replication based on a view
    ▸ bulk get
    ▸ pluggable
    WITH MODERN FEATURES

    View Slide

  10. ▸ DATA are not only blobs
    ▸ Replicated APPs
    ▸ Couchapps but extended and revisited
    REPLICATED APPS

    View Slide

  11. BUILT
    IN ERLANG
    />
    <

    View Slide

  12. ▸ append only file
    ▸ MVCC
    ▸ 2 indexes (btree): by sequence, by id
    ▸ 1 index for local documents without conflict handling
    ▸ A revision tree is stored in indexes pointed to the revision offset
    ▸ The revision is stored in the file separately
    HOW ARE STORED DOCUMENTS

    View Slide

  13. REVISION TREE

    View Slide

  14. ▸ Write is slow
    ▸ Read should not being blocked by writes
    ▸ No shared memory
    ▸ No atomic integer trick
    ▸ Only actors and message passing
    ▸ Operations on a doc are atomic
    CHALLENGES

    View Slide

  15. READ/WRITE OPERATIONS
    DB STATE
    READER
    READER
    writer
    writer
    write process
    update
    share state

    View Slide

  16. ▸ LRU to cache blocks

    https://github.com/barrel-db/erlang-lru
    ▸ 1 File process, Operations are limited
    ▸ DB users are linked to the database process
    ▸ Write buffer to reduce the latency
    ▸ Optional wal (barrel 014)
    ▸ NIF for the collation
    READ/WRITE OPERATIONS

    View Slide

  17. INDEX OPERATIONS
    View Group
    READER
    READER
    change reader
    indexer
    update
    share state
    send /collect changes
    DB
    get changes

    View Slide

  18. ▸ Credit Flow Based
    ▸ The View group keep the state
    ▸ View group is created on demande
    ▸ kept open until it has readers
    ▸ Indexer ask for updates
    ▸ Read functions (Map functions) are processed in //
    INDEX OPERATIONS

    View Slide

  19. ▸ Create a new file to remove the fragmentation
    ▸ A race between copy and the addition of new data
    ▸ Require at least twice of the storage
    THE COMPACTION ISSUE

    View Slide

  20. ▸ Added 2 features:
    ▸ MOVE: move doc(s) to another node or database
    ▸ User hooks functions (run in background) using hooks:

    https://github.com/barrel-db/hooks
    ▸ Partition on demand
    ▸ Decision depends on the application needs
    NEW FUNCTIONS

    View Slide

  21. CHANGES HANDLER
    subscriber
    change dispatcher
    broadcast changes
    DB

    View Slide

  22. ▸ changes load balancing
    ▸ consumer subscribe on patterns (delete, update, …)
    ▸ Create changes Load Balancer on demand
    ▸ Allows remote nodes to subscribe to a queue
    ▸ Based on primer (released on February 2016)
    CHANGES EVENTS

    View Slide

  23. ▸ fetch the revisions and their attachments

    not present on the node
    ▸ continuous or not
    ▸ try to collect multiple docs at once
    ▸ use hackney:

    http://github.com/benoitc/hackney
    ▸ Use a Flow-based pattern instead of a classic pool
    REPLICATION

    View Slide

  24. View Slide

  25. HTTPS://BARREL-DB.ORG
    Barrel
    HTTP://ENKIM.EU
    Enki Multimedia

    View Slide