BARREL, KEEP YOUR DATA IN SYNC IN YOUR ERLANG APPLICATION

Barrel is a database I wrote from scratch in Erlang that can be used embedded in an Erlang or Elixir application like Mnesia or standalone over HTTP from any applications or service. With Barrel, you can easily bring and keep a view of your data inside your application and replicate it between your different machines. After briefly describing the architecture of barrel and its key features, the talk will show how we use it to create a custom data platform in Erlang or Elixir for a consumer-oriented product, mixing local and remote storages and different peer-to-peer replications strategies.

Transcript

1. KEEP YOUR DATA IN SYNC IN
   YOUR ERLANG APPLICATION
   CodeBeam Munich - 2018
   

   View Slide

2. benoît chesneau
   craftsman working on P2P and custom data
   endpoints solution
   Created 11 years ago, Enki Multimedia
   member of the Erlang Industrial User Group
   about me
   

   View Slide

3. more and more of our data is in the "Cloud"
   we let to third parties the control over our data
   we live in the time where
   Making us more and
   more dependant
   

   View Slide

4. Barrel is not a database. It is a
   framework / product that synchronizes 

   and replicates data between machines
   and applications.
   

   View Slide

5. keep a partial state of your data locally
   replicate changes in your data
   route automatically or implicitly a view of your data
   Built in Erlang
   Opensource
   Features
   

   View Slide

6. Barrel, core concepts
   

   View Slide

7. core data structure of barrel are documents
   stored in a barrel
   can be represented as a an Erlang map or JSON
   with a revision
   Document oriented
   {
   "id" : "ID",
   "key" : "value",
   "_rev": "REVISIONID"
   }
   

   View Slide

8. attachments are binary contents
   updated or linked (http) to a document
   attach anything
   {
   "id" : "ID",
   "key" : "value",
   "_attachments": [
   {
   "id" : "AttachmentID"
   }
   ]
   }
   

   View Slide

9. by document IDs
   by path
   automatically indexed
   {
   "id" : "ID",
   "locations" : [
   {
   "country" : "France",
   "city" : "creil"
   }
   ],
   "headquarter" : "France"
   }
   $/id/ID
   $/locations/0/country/france
   $/locations/0/city/creil
   $/headquarter/france
   

   View Slide

10. listen on any document changes
    changes are incremental
    filter changes
    persist view changes
    database changes
    {
    "id" : "ID",
    "rev" : "REVID",
    "seq" : 1,
    "changes" : ["REVID"]
    }
    

    View Slide

11. uses the changes feed
    handle the conflicts
    push/pull replication
    push/pull replication
    

    View Slide

12. built in Erlang
    good resilience due to an efficient supervision. (aten
    from rabbitmq for local nodes monitoring)
    Each steps of the write workflow is done in different
    isolated processes
    local document db using Rocksdb using dirty-nifs:

    https://gitlab.com/barrel-db/erlang-rocksdb
    new improvements in the TCP/IO stack improves
    the performance for free.
    

    View Slide

13. create a custom data platform
    mixing local and remote storages
    and different peer-to-peer
    replications strategies.
    

    View Slide

14. HTTP(s) & Websocket for any one
    lib can be embedded in an Erlang Node if you want
    to
    remote nodes uses websockets to exchange the
    data
    embedded or remote
    

    View Slide

15. barrels have to be created per nodes
    Or via the HTTP API
    create a barrel
    Name = <>,
    Params = #{ store_provider => <> },
    ok = create_barrel(Name, Params)
    POST /barrels/name
    {
    "name": "mybarrel",
    "params" : {
    "store_provider": "MyLocalRocksdb"
    }
    }
    

    View Slide

16. once opened , you can use a barrel to create and
    retrieve a doc
    simple put and get a barrel
    {ok, Barrel} = barrel:open(Name)
    {ok, DocId, Rev} = barrel:save_doc(Barrel, {}),
    {ok, Doc = #{ <> := DocId,
    <> := Rev }} = barrel:fetch_doc(Barrel, DocId)
    

    View Slide

17. once opened , you can use a barrel to create and
    retrieve a doc
    can be nested
    "Peer to Peer" Put
    Params = #{ p2p_write => { host => "HOST:PORT",
    name => <> }}
    {ok, DocId, Rev} = barrel:save_doc(Barrel, {}, Params)
    Params = #{p2p_write => { host => "HOST:PORT",
    name => <>,
    p2p_write => { ... } }}
    

    View Slide

18. similar syntax to Put
    Read first locally then remote
    "Peer to Peer" Get
    Params = #{p2p_read => { host => "HOST:PORT",
    name => <>}
    {ok, Doc} = barrel:fetch_doc(Barrel, {}, Params),
    

    View Slide

19. once opened , you can use a barrel to create and
    retrieve a doc

    

    Source and Target can be remote or local. Or both.
    Can work in Push and Pull mode at the same time
    (synchronisation).
    REPLICATION
    Params = #{
    <> => <>,
    < << "ws://HOST:PORT/RemoteBarrel" >>
    }
    {ok, RepId} = barrel:replicate(Params),
    

    View Slide

20. A listener is a simple process on which you can
    pass a callback.
    you can subscribe to different pattern of path with a
    semantic similar to the one in MQTT
    Listen on change
    Callback = fun(Change) -> ok end,
    Params => #{ callback_fun => MyCallback,
    subscribe_to => [<>],
    since => 0
    },
    {ok, ListenerPid} = start_listener(Barrel, Params).
    

    View Slide

21. what's next?
    

    View Slide

22. contact@barrel-db.org
    https://barrel-db.org
    

    View Slide