Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Barrel. Using Erlang to build a Database

Barrel. Using Erlang to build a Database

Barrel (https://barrel-db.org) is a modern document-oriented database in Erlang focusing on data locality (put/match the data next to you) and P2P with an effort to maintain a compatibility with the Apache CouchDB API.

In this talk, I will begin with an overview of Barrel, covering the problems that it is intended to solve and how its data model, transaction model, query model, and deployment model work together to solve those problems. I will then use Barrel to illustrate how to design and implement a production software in Erlang.

F04edc7cb2099745e5413c754d3d22f5?s=128

Benoit Chesneau

January 29, 2016
Tweet

Transcript

  1. BARREL DISTRIBUTED DATABASE FOR THE MODERN WORLD https://barrel-db.org Erlang Factory

    Lite Brussel 2016
  2. VISION AND CONCEPT

  3. local database mobile sensor "cloud" database local database mobile DATA

    IS MOBILE
  4. share discover replicate PEER TO PEER (P2P)

  5. ▸ Local first ▸ Put/Match the data next to you

    ▸ Query Locally ▸ Replicate a view of the data you need
  6. WHAT IS BARREL

  7. WHAT IS BARREL ▸ a document database ▸ revision tree

    & atomic document updates ▸ changes feed ▸ replication between any nodes in both way ▸ views (~ map)
  8. AN EFFORT TO MAINTAIN THE COUCHDB API

  9. ▸ view changes ▸ replication based on a view ▸

    bulk get ▸ pluggable WITH MODERN FEATURES
  10. ▸ DATA are not only blobs ▸ Replicated APPs ▸

    Couchapps but extended and revisited REPLICATED APPS
  11. BUILT IN ERLANG /> <

  12. ▸ append only file ▸ MVCC ▸ 2 indexes (btree):

    by sequence, by id ▸ 1 index for local documents without conflict handling ▸ A revision tree is stored in indexes pointed to the revision offset ▸ The revision is stored in the file separately HOW ARE STORED DOCUMENTS
  13. REVISION TREE

  14. ▸ Write is slow ▸ Read should not being blocked

    by writes ▸ No shared memory ▸ No atomic integer trick ▸ Only actors and message passing ▸ Operations on a doc are atomic CHALLENGES
  15. READ/WRITE OPERATIONS DB STATE READER READER writer writer write process

    update share state
  16. ▸ LRU to cache blocks
 https://github.com/barrel-db/erlang-lru ▸ 1 File process,

    Operations are limited ▸ DB users are linked to the database process ▸ Write buffer to reduce the latency ▸ Optional wal (barrel 014) ▸ NIF for the collation READ/WRITE OPERATIONS
  17. INDEX OPERATIONS View Group READER READER change reader indexer update

    share state send /collect changes DB get changes
  18. ▸ Credit Flow Based ▸ The View group keep the

    state ▸ View group is created on demande ▸ kept open until it has readers ▸ Indexer ask for updates ▸ Read functions (Map functions) are processed in // INDEX OPERATIONS
  19. ▸ Create a new file to remove the fragmentation ▸

    A race between copy and the addition of new data ▸ Require at least twice of the storage THE COMPACTION ISSUE
  20. ▸ Added 2 features: ▸ MOVE: move doc(s) to another

    node or database ▸ User hooks functions (run in background) using hooks:
 https://github.com/barrel-db/hooks ▸ Partition on demand ▸ Decision depends on the application needs NEW FUNCTIONS
  21. CHANGES HANDLER subscriber change dispatcher broadcast changes DB

  22. ▸ changes load balancing ▸ consumer subscribe on patterns (delete,

    update, …) ▸ Create changes Load Balancer on demand ▸ Allows remote nodes to subscribe to a queue ▸ Based on primer (released on February 2016) CHANGES EVENTS
  23. ▸ fetch the revisions and their attachments
 not present on

    the node ▸ continuous or not ▸ try to collect multiple docs at once ▸ use hackney:
 http://github.com/benoitc/hackney ▸ Use a Flow-based pattern instead of a classic pool REPLICATION
  24. None
  25. HTTPS://BARREL-DB.ORG Barrel HTTP://ENKIM.EU Enki Multimedia