BARREL, KEEP YOUR DATA IN SYNC IN YOUR ERLANG APPLICATION

Barrel is a database I wrote from scratch in Erlang that can be used embedded in an Erlang or Elixir application like Mnesia or standalone over HTTP from any applications or service. With Barrel, you can easily bring and keep a view of your data inside your application and replicate it between your different machines. After briefly describing the architecture of barrel and its key features, the talk will show how we use it to create a custom data platform in Erlang or Elixir for a consumer-oriented product, mixing local and remote storages and different peer-to-peer replications strategies.

Transcript

1. KEEP YOUR DATA IN SYNC IN YOUR ERLANG APPLICATION CodeBeam Munich - 2018
2. benoît chesneau craftsman working on P2P and custom data endpoints solution Created 11 years ago, Enki Multimedia member of the Erlang Industrial User Group about me
3. more and more of our data is in the "Cloud" we let to third parties the control over our data we live in the time where Making us more and more dependant
4. Barrel is not a database. It is a framework / product that synchronizes 
 and replicates data between machines and applications.
5. keep a partial state of your data locally replicate changes in your data route automatically or implicitly a view of your data Built in Erlang Opensource Features
6. Barrel, core concepts
7. core data structure of barrel are documents stored in a barrel can be represented as a an Erlang map or JSON with a revision Document oriented { "id" : "ID", "key" : "value", "_rev": "REVISIONID" }
8. attachments are binary contents updated or linked (http) to a document attach anything { "id" : "ID", "key" : "value", "_attachments": [ { "id" : "AttachmentID" } ] }
9. by document IDs by path automatically indexed { "id" : "ID", "locations" : [ { "country" : "France", "city" : "creil" } ], "headquarter" : "France" } $/id/ID $/locations/0/country/france $/locations/0/city/creil $/headquarter/france
10. listen on any document changes changes are incremental filter changes persist view changes database changes { "id" : "ID", "rev" : "REVID", "seq" : 1, "changes" : ["REVID"] }
11. uses the changes feed handle the conflicts push/pull replication push/pull replication
12. built in Erlang good resilience due to an efficient supervision. (aten from rabbitmq for local nodes monitoring) Each steps of the write workflow is done in different isolated processes local document db using Rocksdb using dirty-nifs:
 https://gitlab.com/barrel-db/erlang-rocksdb new improvements in the TCP/IO stack improves the performance for free.
13. create a custom data platform mixing local and remote storages and different peer-to-peer replications strategies.
14. HTTP(s) & Websocket for any one lib can be embedded in an Erlang Node if you want to remote nodes uses websockets to exchange the data embedded or remote
15. barrels have to be created per nodes Or via the HTTP API create a barrel Name = <<"mybarrel">>, Params = #{ store_provider => <<"MyLocalRocksdb">> }, ok = create_barrel(Name, Params) POST /barrels/name { "name": "mybarrel", "params" : { "store_provider": "MyLocalRocksdb" } }
16. once opened , you can use a barrel to create and retrieve a doc simple put and get a barrel {ok, Barrel} = barrel:open(Name) {ok, DocId, Rev} = barrel:save_doc(Barrel, {}), {ok, Doc = #{ <<"id">> := DocId, <<"_rev">> := Rev }} = barrel:fetch_doc(Barrel, DocId)
17. once opened , you can use a barrel to create and retrieve a doc can be nested "Peer to Peer" Put Params = #{ p2p_write => { host => "HOST:PORT", name => <<"RemoteBarrel">> }} {ok, DocId, Rev} = barrel:save_doc(Barrel, {}, Params) Params = #{p2p_write => { host => "HOST:PORT", name => <<"RemoteBarrel">>, p2p_write => { ... } }}
18. similar syntax to Put Read first locally then remote "Peer to Peer" Get Params = #{p2p_read => { host => "HOST:PORT", name => <<"RemoteBarrel">>} {ok, Doc} = barrel:fetch_doc(Barrel, {}, Params),
19. once opened , you can use a barrel to create and retrieve a doc
 
 Source and Target can be remote or local. Or both. Can work in Push and Pull mode at the same time (synchronisation). REPLICATION Params = #{ <<"source">> => <<"MyLocalBarrel">>, <<"target" => << "ws://HOST:PORT/RemoteBarrel" >> } {ok, RepId} = barrel:replicate(Params),
20. A listener is a simple process on which you can pass a callback. you can subscribe to different pattern of path with a semantic similar to the one in MQTT Listen on change Callback = fun(Change) -> ok end, Params => #{ callback_fun => MyCallback, subscribe_to => [<<"/*">>], since => 0 }, {ok, ListenerPid} = start_listener(Barrel, Params).
21. what's next?
22. contact@barrel-db.org https://barrel-db.org