Barrel. Using Erlang to build a Database

Barrel (https://barrel-db.org) is a modern document-oriented database in Erlang focusing on data locality (put/match the data next to you) and P2P with an effort to maintain a compatibility with the Apache CouchDB API.

In this talk, I will begin with an overview of Barrel, covering the problems that it is intended to solve and how its data model, transaction model, query model, and deployment model work together to solve those problems. I will then use Barrel to illustrate how to design and implement a production software in Erlang.

Transcript

1. BARREL DISTRIBUTED DATABASE FOR THE MODERN WORLD https://barrel-db.org Erlang Factory Lite Brussel 2016
2. VISION AND CONCEPT
3. local database mobile sensor "cloud" database local database mobile DATA IS MOBILE
4. share discover replicate PEER TO PEER (P2P)
5. ▸ Local first ▸ Put/Match the data next to you ▸ Query Locally ▸ Replicate a view of the data you need
6. WHAT IS BARREL
7. WHAT IS BARREL ▸ a document database ▸ revision tree & atomic document updates ▸ changes feed ▸ replication between any nodes in both way ▸ views (~ map)
8. AN EFFORT TO MAINTAIN THE COUCHDB API
9. ▸ view changes ▸ replication based on a view ▸ bulk get ▸ pluggable WITH MODERN FEATURES
10. ▸ DATA are not only blobs ▸ Replicated APPs ▸ Couchapps but extended and revisited REPLICATED APPS
11. BUILT IN ERLANG /> <
12. ▸ append only file ▸ MVCC ▸ 2 indexes (btree): by sequence, by id ▸ 1 index for local documents without conflict handling ▸ A revision tree is stored in indexes pointed to the revision offset ▸ The revision is stored in the file separately HOW ARE STORED DOCUMENTS
13. REVISION TREE
14. ▸ Write is slow ▸ Read should not being blocked by writes ▸ No shared memory ▸ No atomic integer trick ▸ Only actors and message passing ▸ Operations on a doc are atomic CHALLENGES
15. READ/WRITE OPERATIONS DB STATE READER READER writer writer write process update share state
16. ▸ LRU to cache blocks
 https://github.com/barrel-db/erlang-lru ▸ 1 File process, Operations are limited ▸ DB users are linked to the database process ▸ Write buffer to reduce the latency ▸ Optional wal (barrel 014) ▸ NIF for the collation READ/WRITE OPERATIONS
17. INDEX OPERATIONS View Group READER READER change reader indexer update share state send /collect changes DB get changes
18. ▸ Credit Flow Based ▸ The View group keep the state ▸ View group is created on demande ▸ kept open until it has readers ▸ Indexer ask for updates ▸ Read functions (Map functions) are processed in // INDEX OPERATIONS
19. ▸ Create a new file to remove the fragmentation ▸ A race between copy and the addition of new data ▸ Require at least twice of the storage THE COMPACTION ISSUE
20. ▸ Added 2 features: ▸ MOVE: move doc(s) to another node or database ▸ User hooks functions (run in background) using hooks:
 https://github.com/barrel-db/hooks ▸ Partition on demand ▸ Decision depends on the application needs NEW FUNCTIONS
21. CHANGES HANDLER subscriber change dispatcher broadcast changes DB
22. ▸ changes load balancing ▸ consumer subscribe on patterns (delete, update, …) ▸ Create changes Load Balancer on demand ▸ Allows remote nodes to subscribe to a queue ▸ Based on primer (released on February 2016) CHANGES EVENTS
23. ▸ fetch the revisions and their attachments
 not present on the node ▸ continuous or not ▸ try to collect multiple docs at once ▸ use hackney:
 http://github.com/benoitc/hackney ▸ Use a Flow-based pattern instead of a classic pool REPLICATION
25. HTTPS://BARREL-DB.ORG Barrel HTTP://ENKIM.EU Enki Multimedia