Building and Integrating A Data Platform

Slide 1

Slide 1 text

BUILDING AND INTEGRATING A DATA PLATFORM CodeBeam SF - 2018

Slide 2

Slide 2 text

benoît chesneau craftsman working on P2P and custom data endpoints solution enki multimedia: the corporate interface member of the Erlang Industrial User Group about me

Slide 3

Slide 3 text

micromobile services

Slide 4

Slide 4 text

1. Does my service do only one thing? 2. Is my service autonomous? 3. Does this service own its own data? a good micro-service?

Slide 5

Slide 5 text

isolated own its own data resilient communicate with other by asynchronous messages micro-servcice

Slide 6

Slide 6 text

sharing data in the mobile age between and across micro-services make applications more scalable and resilient Ex: messaging systems, sharing data

Slide 7

Slide 7 text

sharing data update   and query microservice standard solution: client call a webservices to query and update the data problem: if connection is slow or absent the microservice stops cloud   storage

Slide 8

Slide 8 text

sharing data microservice local storage replicated always available eventually consistent cloud   storage update   and query synchronize local   storage

Slide 9

Slide 9 text

barrel Bring and keep a view of your data near your application

Slide 10

Slide 10 text

a database focusing on simplicity document oriented Automatic indexing Focusing on simplicity

Slide 11

Slide 11 text

Docs are maps { “id” : “someid”, “Key” : “value” }

Slide 12

Slide 12 text

automatic indexing Access by path: /locations/country/Germany

Slide 13

Slide 13 text

Local ﬁrst local ﬁrst: bring and keep a view of your data near your application data is synchronised with other storages Replication to and from any sources

Slide 14

Slide 14 text

partial view query node Title Text

Slide 15

Slide 15 text

library embedded in your Erlang application(*) available as a micro-service via HTTP(1,2) or via the Erlang distribution Peer to peer: a barrel is the unit Semantic to allow distributed transactions P2p (*) including elixir or lfe, or ….

Slide 16

Slide 16 text

every peers fork the master, updates are ofﬂine peers pull and merge from the main server works well for back pressure (writes can be delayed) CRDT semantic for conﬂict-free data structures

Slide 17

Slide 17 text

causality no vector clock revision tree

Slide 18

Slide 18 text

a a a b c commit Alice Iko Bob state t0 pull operations manual automatic rejected

Slide 19

Slide 19 text

a a a b b Alice Iko Bob state t0 rejected pull pull t1 c b d merge operations

Slide 20

Slide 20 text

a a a b b Alice Iko Bob state t2 pull t3 c b d pull operations

Slide 21

Slide 21 text

? ?? Erlang

Slide 22

Slide 22 text

Erlang is slow Erlang is only for communications protocols I should do it rust… No access to low level memory and ﬁle systems APIS Why not Erlang

Slide 23

Slide 23 text

Barrel is more a data orchestration service than a database Basic indexing Focus on replicating the data Nifs to help Why Erlang

Slide 24

Slide 24 text

Doc: Revision + Metadata data: Read-Modify-Write: concurrency issue Incremental changes log: append only Indexes: when a new winning version is found the doc is indexed. Blobs (attachments) What we write

Slide 25

Slide 25 text

Provides connectors for other storages RocksdDB for local persistent storage  https://gitlab.com/barrel-db/erlang-rocksdb.git Dirty-nifs ETS? Use the right tool for …

Slide 26

Slide 26 text

Goal: anticipate the resource usage at the node level Return early to the client Control applied to all resources in the nodes Back-pressure let it bend: be resilient

Slide 27

Slide 27 text

worker_pool  https://github.com/inaka/worker_pool Hard to debug your program Little control on the pending requests Ecpocxy but handle back-pressure the reverse way Simple pooling

Slide 28

Slide 28 text

Clients and Jobs should be handled independently Active and passive regularion Request unit: to set the number of requests we want to serve / seconds Flow-Based programming? sbroker, partially ﬁt the bill:   https://github.com/ﬁshcakez/sbroker Dynamic regulation

Slide 29

Slide 29 text

Started with a simple “Single Writer Multiple Readers” pattern bottleneck: A process to handle the ﬁnal write to the database We do and // most of the work out of the write process Indexes are processed asynchronously (but a session can read its own writes if needed) Concurrency challenge

Slide 30

Slide 30 text

Read access is shared via ETS On request a monitor to the db is created ets: to share the state between readers

Slide 31

Slide 31 text

When using the erlang distribution, events are dispatched by nodes, processes always subscribe locally Events

Slide 32

Slide 32 text

Erlang distribution is not used to share the data Erlang distribution can be switched HTTP transports Transport the data

Slide 33

Slide 33 text

Roadmap

Slide 34

Slide 34 text

1.0: 24 march 2018 1.1: 24 april 2018 … Milestones

Slide 35

Slide 35 text

1.0: Websockets support (with new hackney) 1.1: Experimental: GRPC Coming features