Verne.mq

André Graf is co-founder and CTO of erl.io, a swiss company with a focus on Erlang and distributed systems engineering.

He presented VerneMQ, a new Erlang powered MQTT message broker his company opensourced a few weeks ago. His talk covers a quick introduction to the MQTT protocol and some of its application scenarios. Special emphasis will then be laid on the VerneMQ architecture and its take on scalability and extensibility.

Transcript

1. erlang.paris Meetup / June 29th 2015, Paris / André Graf (@der_graf) VerneMQ - A distributed MQTT message broker developed in Erlang
2. Intro - André Graf, Co-Founder at Erl.io (2012) - Programming in Erlang since 2008 - Basel Switzerland @der_graf
3. Simple Publish/Subscribe protocol over TCP/IP that utilizes a centralized message broker MQTT
4. Different use cases… MQTT
5. https://www.flickr.com/mmmavocado/
6. https://www.flickr.com/photos/christianhaugen
7. https://www.flickr.com/photos/janitors
8. Different use cases… ...but similar requirements. MQTT
9. constrained devices in constrained environments MQTT
10. | different QoS Levels | small header sizes (min 2B) | durable sessions | last will testament MQTT
11. Some MQTT Facts - Current Spec Version 3.1.1 - open OASIS Standard since 2014 - Invented 1998 at IBM and Arcom
12. VerneMQ Architecture from a bird’s eye perspective
13. Erlang VM ninenines/ranch ninenines/cowboy vmq_server vmq_plugin helium/eleveldb helium/plumtree uwiger/jobs vmq_commons feuerlabs/ exometer_core vmq_passwd vmq_acl vmq_bridge vmq_systree vmq_snmp vmq_graphite larshesel/ vmq_elixir
14. erlio/vernemq release project basho/nodepackage basho/cuttlefish basho/clique rebar/rebar3
15. vmq_reg_trie <<gen_server>> 3 processes / MQTT connection vmq_server vmq_session <<gen_fsm>> vmq_ranch <<ranch_protocol>> vmq_queue <<gen_fsm>> start_link start_link vmq_reg <<gen_server>> vmq_reg_sup <<supervisor>> vmq_reg_trie <<gen_server>> monitor ETS ETS
16. vmq_reg_trie <<gen_server>> vmq_server vmq_session <<gen_fsm>> vmq_ranch <<ranch_protocol>> sync vmq_reg <<gen_server>> vmq_reg_sup <<supervisor>> vmq_reg_trie <<gen_server>> ETS vmq_session <<gen_fsm>> vmq_ranch <<ranch_protocol>> vmq_queue <<gen_fsm>> async batch/async match fold MQTT publish example TCP Conn Client A TCP Conn Client B ETS PUB /a/b/c SUB /a/b/+
17. Clustering VerneMQ
18. Distribution What is ACTIVELY distributed? | Subscriber Data | Online Messages | Cluster Status Information | ...
19. Distribution What is NOT distributed? | offline messages (future)
20. Distributing Subscriber Data - Initially using Mnesia - Move to Plumtree/LevelDB Paradigm Shift: Consistent -> eventual Consistent
21. Distributing online Messages - Initially over Erlang Distribution layer - Move to own communication layer - buffering in case of ‘nodedown’s - batching requests for higher throughput
22. Extending VerneMQ
23. erlio/vmq_plugin %% iterates over all Plugins serving the Hook 'HookName' vmq_plugin:all(HookName :: atom(), HookArgs :: [any()]). %% iterates over all Plugins serving the Hook 'HookName' %% until one returns 'ok' | {'ok', any()} vmq_plugin:all_till_ok(HookName :: atom(), HookArgs :: [any()]). %% calls the top priority Plugin that serves the Hook 'HookName' vmq_plugin:only(HookName :: atom(), HookArgs :: [any()]).
24. erlio/vmq_plugin %% enabling an OTP Application Plugin vmq_plugin_mgr:enable_plugin(Application :: atom()). %% in Application.app.src {env, [ {vmq_plugin_hooks, [ {HookName, Module, Function, Arity, []} ... ]} ]}
25. erlio/vmq_plugin Available Hooks: - auth_on_register/5 - auth_on_publish/6 - auth_on_subscribe/3 - on_register/3 - on_publish/6 - on_subscribe/3 - on_deliver/6 (planned)
26. erlio/vmq_plugin Available Hooks: - msg_store_write_sync/2 - msg_store_write_async/2 - msg_store_delete_sync/1 - msg_store_delete_async/1 - msg_store_fold/2 - msg_store_read/1
27. erlio/vmq_plugin Checkout vmq_plugin and the Demo Plugin App https://github.com/erlio/vmq_plugin https://github.com/dergraf/vernemq_demo_plugin
28. To Sum Up - MQTT is a great Protocol for various use cases (monitoring/metering/mobile/..) - VerneMQ brings scalability and powerful extensibility to the MQTT world. - Erl.io provides support and development services for VerneMQ
29. http://verne.mq | http://erl.io @der_graf | @vernemq | @erlio_basel Merci Beaucoup! Je suis content de répondre a vos questions.
30. References https://verne.mq http://mqtt.org https://github.com/erlio/vernemq https://github.com/erlio/vmq_plugin https://github.com/dergraf/vernemq_demo_plugin https://github.com/helium/plumtree http://asc.di.fct.unl.pt/~jleitao/pdf/srds07-leitao.pdf https://groups.google.com/forum/#!topic/mqtt/c1dlCdA5eOc