Upgrade to Pro — share decks privately, control downloads, hide ads and more …

BARREL, KEEP YOUR DATA IN SYNC IN YOUR ERLANG APPLICATION

BARREL, KEEP YOUR DATA IN SYNC IN YOUR ERLANG APPLICATION

Barrel is a database I wrote from scratch in Erlang that can be used embedded in an Erlang or Elixir application like Mnesia or standalone over HTTP from any applications or service. With Barrel, you can easily bring and keep a view of your data inside your application and replicate it between your different machines. After briefly describing the architecture of barrel and its key features, the talk will show how we use it to create a custom data platform in Erlang or Elixir for a consumer-oriented product, mixing local and remote storages and different peer-to-peer replications strategies.

Benoit Chesneau

December 07, 2018
Tweet

More Decks by Benoit Chesneau

Other Decks in Programming

Transcript

  1. KEEP YOUR DATA IN SYNC IN
    YOUR ERLANG APPLICATION
    CodeBeam Munich - 2018

    View full-size slide

  2. benoît chesneau
    craftsman working on P2P and custom data
    endpoints solution
    Created 11 years ago, Enki Multimedia
    member of the Erlang Industrial User Group
    about me

    View full-size slide

  3. more and more of our data is in the "Cloud"
    we let to third parties the control over our data
    we live in the time where
    Making us more and
    more dependant

    View full-size slide

  4. Barrel is not a database. It is a
    framework / product that synchronizes 

    and replicates data between machines
    and applications.

    View full-size slide

  5. keep a partial state of your data locally
    replicate changes in your data
    route automatically or implicitly a view of your data
    Built in Erlang
    Opensource
    Features

    View full-size slide

  6. Barrel, core concepts

    View full-size slide

  7. core data structure of barrel are documents
    stored in a barrel
    can be represented as a an Erlang map or JSON
    with a revision
    Document oriented
    {
    "id" : "ID",
    "key" : "value",
    "_rev": "REVISIONID"
    }

    View full-size slide

  8. attachments are binary contents
    updated or linked (http) to a document
    attach anything
    {
    "id" : "ID",
    "key" : "value",
    "_attachments": [
    {
    "id" : "AttachmentID"
    }
    ]
    }

    View full-size slide

  9. by document IDs
    by path
    automatically indexed
    {
    "id" : "ID",
    "locations" : [
    {
    "country" : "France",
    "city" : "creil"
    }
    ],
    "headquarter" : "France"
    }
    $/id/ID
    $/locations/0/country/france
    $/locations/0/city/creil
    $/headquarter/france

    View full-size slide

  10. listen on any document changes
    changes are incremental
    filter changes
    persist view changes
    database changes
    {
    "id" : "ID",
    "rev" : "REVID",
    "seq" : 1,
    "changes" : ["REVID"]
    }

    View full-size slide

  11. uses the changes feed
    handle the conflicts
    push/pull replication
    push/pull replication

    View full-size slide

  12. built in Erlang
    good resilience due to an efficient supervision. (aten
    from rabbitmq for local nodes monitoring)
    Each steps of the write workflow is done in different
    isolated processes
    local document db using Rocksdb using dirty-nifs:

    https://gitlab.com/barrel-db/erlang-rocksdb
    new improvements in the TCP/IO stack improves
    the performance for free.

    View full-size slide

  13. create a custom data platform
    mixing local and remote storages
    and different peer-to-peer
    replications strategies.

    View full-size slide

  14. HTTP(s) & Websocket for any one
    lib can be embedded in an Erlang Node if you want
    to
    remote nodes uses websockets to exchange the
    data
    embedded or remote

    View full-size slide

  15. barrels have to be created per nodes
    Or via the HTTP API
    create a barrel
    Name = <<"mybarrel">>,
    Params = #{ store_provider => <<"MyLocalRocksdb">> },
    ok = create_barrel(Name, Params)
    POST /barrels/name
    {
    "name": "mybarrel",
    "params" : {
    "store_provider": "MyLocalRocksdb"
    }
    }

    View full-size slide

  16. once opened , you can use a barrel to create and
    retrieve a doc
    simple put and get a barrel
    {ok, Barrel} = barrel:open(Name)
    {ok, DocId, Rev} = barrel:save_doc(Barrel, {}),
    {ok, Doc = #{ <<"id">> := DocId,
    <<"_rev">> := Rev }} = barrel:fetch_doc(Barrel, DocId)

    View full-size slide

  17. once opened , you can use a barrel to create and
    retrieve a doc
    can be nested
    "Peer to Peer" Put
    Params = #{ p2p_write => { host => "HOST:PORT",
    name => <<"RemoteBarrel">> }}
    {ok, DocId, Rev} = barrel:save_doc(Barrel, {}, Params)
    Params = #{p2p_write => { host => "HOST:PORT",
    name => <<"RemoteBarrel">>,
    p2p_write => { ... } }}

    View full-size slide

  18. similar syntax to Put
    Read first locally then remote
    "Peer to Peer" Get
    Params = #{p2p_read => { host => "HOST:PORT",
    name => <<"RemoteBarrel">>}
    {ok, Doc} = barrel:fetch_doc(Barrel, {}, Params),

    View full-size slide

  19. once opened , you can use a barrel to create and
    retrieve a doc


    Source and Target can be remote or local. Or both.
    Can work in Push and Pull mode at the same time
    (synchronisation).
    REPLICATION
    Params = #{
    <<"source">> => <<"MyLocalBarrel">>,
    <<"target" => << "ws://HOST:PORT/RemoteBarrel" >>
    }
    {ok, RepId} = barrel:replicate(Params),

    View full-size slide

  20. A listener is a simple process on which you can
    pass a callback.
    you can subscribe to different pattern of path with a
    semantic similar to the one in MQTT
    Listen on change
    Callback = fun(Change) -> ok end,
    Params => #{ callback_fun => MyCallback,
    subscribe_to => [<<"/*">>],
    since => 0
    },
    {ok, ListenerPid} = start_listener(Barrel, Params).

    View full-size slide

  21. what's next?

    View full-size slide