Verne.mq

 Verne.mq

André Graf is co-founder and CTO of erl.io, a swiss company with a focus on Erlang and distributed systems engineering.

He presented VerneMQ, a new Erlang powered MQTT message broker his company opensourced a few weeks ago. His talk covers a quick introduction to the MQTT protocol and some of its application scenarios. Special emphasis will then be laid on the VerneMQ architecture and its take on scalability and extensibility.

89261d2bbe7220dfde8ec1be0aa0f52f?s=128

erlang.paris

June 29, 2015
Tweet

Transcript

  1. erlang.paris Meetup / June 29th 2015, Paris / André Graf

    (@der_graf) VerneMQ - A distributed MQTT message broker developed in Erlang
  2. Intro - André Graf, Co-Founder at Erl.io (2012) - Programming

    in Erlang since 2008 - Basel Switzerland @der_graf
  3. Simple Publish/Subscribe protocol over TCP/IP that utilizes a centralized message

    broker MQTT
  4. Different use cases… MQTT

  5. https://www.flickr.com/mmmavocado/

  6. https://www.flickr.com/photos/christianhaugen

  7. https://www.flickr.com/photos/janitors

  8. Different use cases… ...but similar requirements. MQTT

  9. constrained devices in constrained environments MQTT

  10. | different QoS Levels | small header sizes (min 2B)

    | durable sessions | last will testament MQTT
  11. Some MQTT Facts - Current Spec Version 3.1.1 - open

    OASIS Standard since 2014 - Invented 1998 at IBM and Arcom
  12. VerneMQ Architecture from a bird’s eye perspective

  13. Erlang VM ninenines/ranch ninenines/cowboy vmq_server vmq_plugin helium/eleveldb helium/plumtree uwiger/jobs vmq_commons

    feuerlabs/ exometer_core vmq_passwd vmq_acl vmq_bridge vmq_systree vmq_snmp vmq_graphite larshesel/ vmq_elixir
  14. erlio/vernemq release project basho/nodepackage basho/cuttlefish basho/clique rebar/rebar3

  15. vmq_reg_trie <<gen_server>> 3 processes / MQTT connection vmq_server vmq_session <<gen_fsm>>

    vmq_ranch <<ranch_protocol>> vmq_queue <<gen_fsm>> start_link start_link vmq_reg <<gen_server>> vmq_reg_sup <<supervisor>> vmq_reg_trie <<gen_server>> monitor ETS ETS
  16. vmq_reg_trie <<gen_server>> vmq_server vmq_session <<gen_fsm>> vmq_ranch <<ranch_protocol>> sync vmq_reg <<gen_server>>

    vmq_reg_sup <<supervisor>> vmq_reg_trie <<gen_server>> ETS vmq_session <<gen_fsm>> vmq_ranch <<ranch_protocol>> vmq_queue <<gen_fsm>> async batch/async match fold MQTT publish example TCP Conn Client A TCP Conn Client B ETS PUB /a/b/c SUB /a/b/+
  17. Clustering VerneMQ

  18. Distribution What is ACTIVELY distributed? | Subscriber Data | Online

    Messages | Cluster Status Information | ...
  19. Distribution What is NOT distributed? | offline messages (future)

  20. Distributing Subscriber Data - Initially using Mnesia - Move to

    Plumtree/LevelDB Paradigm Shift: Consistent -> eventual Consistent
  21. Distributing online Messages - Initially over Erlang Distribution layer -

    Move to own communication layer - buffering in case of ‘nodedown’s - batching requests for higher throughput
  22. Extending VerneMQ

  23. erlio/vmq_plugin %% iterates over all Plugins serving the Hook 'HookName'

    vmq_plugin:all(HookName :: atom(), HookArgs :: [any()]). %% iterates over all Plugins serving the Hook 'HookName' %% until one returns 'ok' | {'ok', any()} vmq_plugin:all_till_ok(HookName :: atom(), HookArgs :: [any()]). %% calls the top priority Plugin that serves the Hook 'HookName' vmq_plugin:only(HookName :: atom(), HookArgs :: [any()]).
  24. erlio/vmq_plugin %% enabling an OTP Application Plugin vmq_plugin_mgr:enable_plugin(Application :: atom()).

    %% in Application.app.src {env, [ {vmq_plugin_hooks, [ {HookName, Module, Function, Arity, []} ... ]} ]}
  25. erlio/vmq_plugin Available Hooks: - auth_on_register/5 - auth_on_publish/6 - auth_on_subscribe/3 -

    on_register/3 - on_publish/6 - on_subscribe/3 - on_deliver/6 (planned)
  26. erlio/vmq_plugin Available Hooks: - msg_store_write_sync/2 - msg_store_write_async/2 - msg_store_delete_sync/1 -

    msg_store_delete_async/1 - msg_store_fold/2 - msg_store_read/1
  27. erlio/vmq_plugin Checkout vmq_plugin and the Demo Plugin App https://github.com/erlio/vmq_plugin https://github.com/dergraf/vernemq_demo_plugin

  28. To Sum Up - MQTT is a great Protocol for

    various use cases (monitoring/metering/mobile/..) - VerneMQ brings scalability and powerful extensibility to the MQTT world. - Erl.io provides support and development services for VerneMQ
  29. http://verne.mq | http://erl.io @der_graf | @vernemq | @erlio_basel Merci Beaucoup!

    Je suis content de répondre a vos questions.
  30. References https://verne.mq http://mqtt.org https://github.com/erlio/vernemq https://github.com/erlio/vmq_plugin https://github.com/dergraf/vernemq_demo_plugin https://github.com/helium/plumtree http://asc.di.fct.unl.pt/~jleitao/pdf/srds07-leitao.pdf https://groups.google.com/forum/#!topic/mqtt/c1dlCdA5eOc