Primer- A distributed and persistent Message Queue System for Erlang Applications

Slide 1

Slide 1 text

Primer A distributed message queue system for Erlang applications ENKI MULTIMEDIA http://enkim.eu Erlang User Conference - 2015  Benoît CHESNEAU - @benoitc

Slide 2

Slide 2 text

• RCOUCH usages in mind • needed something simple • that I can embed in my Erlang app • without external overhead WHY http://rcouch.org

Slide 3

Slide 3 text

• execution of jobs (sync or not) • realtime pub/sub • Chat • IOT USES

Slide 4

Slide 4 text

FEATURES

Slide 5

Slide 5 text

• embeddable • master-master • versatile • resistant to net-splits • optional persistence FEATURES

Slide 6

Slide 6 text

• can be added to my application • do not rely on external API • no C binding • not related to a speciﬁc protocol EMBEDDABLE

Slide 7

Slide 7 text

• each nodes have the same role • distributed & decentralised • easy to scale MASTER  MASTER

Slide 8

Slide 8 text

• Fast queuing: best effort to order • But also replicated log • one or many consumers • at least/most once semantic • message mapping versatile

Slide 9

Slide 9 text

DESIGN

Slide 10

Slide 10 text

• Full mesh • use gossip to broadcast messages • replicated message queue • synchronous or chained replication distributed

Slide 11

Slide 11 text

• Epidemic broadcast protocol • extracted from riak_core • standalone membership protocol  (riak_dt + ORSWOT) • metadata • broadcast handlers plumtree http://github.com/helium/plumtree

Slide 12

Slide 12 text

• get W nodes to replicate • shufﬂe nodes • replication is synchronous • queue are created dynamically • read queue is local • read from any node • group to fan-out add message best effort to order

Slide 13

Slide 13 text

QUEUE ADD MESSAGE update 1 QUEUE QUEUE replicate 2 Wn W1 got message 3 updated 4 replicate QUEUE MANAGER notify add a message cluster metadata notify best effort to order

Slide 14

Slide 14 text

QUEUE ACK 3 QUEUE QUEUE update message state 3 Wn W1 broadcast ACK 4 fetch 2 read a message cluster metadata ask best effort to order GET MESSAGE METADATA GET NODES 1

Slide 15

Slide 15 text

• log queues should be created ﬁrst • Period of retention • dynamic creation of ranges • a chain of replicated nodes • group of consumers for concurrency add message replicated log

Slide 16

Slide 16 text

QUEUE ADD MESSAGE update 1 QUEUE QUEUE replicate 2 Wn W1 got message 3 updated 4 replicate QUEUE MANAGER notify about new range add a message cluster metadata notify replicated log QUEUE RANGE

Slide 17

Slide 17 text

ACK 3 send 3 read a message cluster metadata ask GET MESSAGE METADATA GET NODES 1 CONSUMER GROUP QUEUE RANGE fetch 2 checkpoint 3 replicated log QUEUE QUEUE Wn W1

Slide 18

Slide 18 text

• by default is off • Append-Only ﬁle • high watermark support • possible to dump data persistence

Slide 19

Slide 19 text

• like a consumer group • to handle custom routing • based on a dsl • dsl can be extended • possibility to re-enqueue a message in another queue mapping

Slide 20

Slide 20 text

mapping [ {map_id, Pattern, ok}, {map, fun(Msg) -> ok end} ]

Slide 21

Slide 21 text

Miscellaneous

Slide 22

Slide 22 text

• metrics • dashboard to watch the activity • a queue can be browsed Other features