Upgrade to Pro — share decks privately, control downloads, hide ads and more …



André Graf is co-founder and CTO of erl.io, a swiss company with a focus on Erlang and distributed systems engineering.

He presented VerneMQ, a new Erlang powered MQTT message broker his company opensourced a few weeks ago. His talk covers a quick introduction to the MQTT protocol and some of its application scenarios. Special emphasis will then be laid on the VerneMQ architecture and its take on scalability and extensibility.



June 29, 2015

More Decks by erlang.paris

Other Decks in Technology


  1. erlang.paris Meetup / June 29th 2015, Paris / André Graf

    (@der_graf) VerneMQ - A distributed MQTT message broker developed in Erlang
  2. Intro - André Graf, Co-Founder at Erl.io (2012) - Programming

    in Erlang since 2008 - Basel Switzerland @der_graf
  3. Simple Publish/Subscribe protocol over TCP/IP that utilizes a centralized message

    broker MQTT
  4. Different use cases… MQTT

  5. https://www.flickr.com/mmmavocado/

  6. https://www.flickr.com/photos/christianhaugen

  7. https://www.flickr.com/photos/janitors

  8. Different use cases… ...but similar requirements. MQTT

  9. constrained devices in constrained environments MQTT

  10. | different QoS Levels | small header sizes (min 2B)

    | durable sessions | last will testament MQTT
  11. Some MQTT Facts - Current Spec Version 3.1.1 - open

    OASIS Standard since 2014 - Invented 1998 at IBM and Arcom
  12. VerneMQ Architecture from a bird’s eye perspective

  13. Erlang VM ninenines/ranch ninenines/cowboy vmq_server vmq_plugin helium/eleveldb helium/plumtree uwiger/jobs vmq_commons

    feuerlabs/ exometer_core vmq_passwd vmq_acl vmq_bridge vmq_systree vmq_snmp vmq_graphite larshesel/ vmq_elixir
  14. erlio/vernemq release project basho/nodepackage basho/cuttlefish basho/clique rebar/rebar3

  15. vmq_reg_trie <<gen_server>> 3 processes / MQTT connection vmq_server vmq_session <<gen_fsm>>

    vmq_ranch <<ranch_protocol>> vmq_queue <<gen_fsm>> start_link start_link vmq_reg <<gen_server>> vmq_reg_sup <<supervisor>> vmq_reg_trie <<gen_server>> monitor ETS ETS
  16. vmq_reg_trie <<gen_server>> vmq_server vmq_session <<gen_fsm>> vmq_ranch <<ranch_protocol>> sync vmq_reg <<gen_server>>

    vmq_reg_sup <<supervisor>> vmq_reg_trie <<gen_server>> ETS vmq_session <<gen_fsm>> vmq_ranch <<ranch_protocol>> vmq_queue <<gen_fsm>> async batch/async match fold MQTT publish example TCP Conn Client A TCP Conn Client B ETS PUB /a/b/c SUB /a/b/+
  17. Clustering VerneMQ

  18. Distribution What is ACTIVELY distributed? | Subscriber Data | Online

    Messages | Cluster Status Information | ...
  19. Distribution What is NOT distributed? | offline messages (future)

  20. Distributing Subscriber Data - Initially using Mnesia - Move to

    Plumtree/LevelDB Paradigm Shift: Consistent -> eventual Consistent
  21. Distributing online Messages - Initially over Erlang Distribution layer -

    Move to own communication layer - buffering in case of ‘nodedown’s - batching requests for higher throughput
  22. Extending VerneMQ

  23. erlio/vmq_plugin %% iterates over all Plugins serving the Hook 'HookName'

    vmq_plugin:all(HookName :: atom(), HookArgs :: [any()]). %% iterates over all Plugins serving the Hook 'HookName' %% until one returns 'ok' | {'ok', any()} vmq_plugin:all_till_ok(HookName :: atom(), HookArgs :: [any()]). %% calls the top priority Plugin that serves the Hook 'HookName' vmq_plugin:only(HookName :: atom(), HookArgs :: [any()]).
  24. erlio/vmq_plugin %% enabling an OTP Application Plugin vmq_plugin_mgr:enable_plugin(Application :: atom()).

    %% in Application.app.src {env, [ {vmq_plugin_hooks, [ {HookName, Module, Function, Arity, []} ... ]} ]}
  25. erlio/vmq_plugin Available Hooks: - auth_on_register/5 - auth_on_publish/6 - auth_on_subscribe/3 -

    on_register/3 - on_publish/6 - on_subscribe/3 - on_deliver/6 (planned)
  26. erlio/vmq_plugin Available Hooks: - msg_store_write_sync/2 - msg_store_write_async/2 - msg_store_delete_sync/1 -

    msg_store_delete_async/1 - msg_store_fold/2 - msg_store_read/1
  27. erlio/vmq_plugin Checkout vmq_plugin and the Demo Plugin App https://github.com/erlio/vmq_plugin https://github.com/dergraf/vernemq_demo_plugin

  28. To Sum Up - MQTT is a great Protocol for

    various use cases (monitoring/metering/mobile/..) - VerneMQ brings scalability and powerful extensibility to the MQTT world. - Erl.io provides support and development services for VerneMQ
  29. http://verne.mq | http://erl.io @der_graf | @vernemq | @erlio_basel Merci Beaucoup!

    Je suis content de répondre a vos questions.
  30. References https://verne.mq http://mqtt.org https://github.com/erlio/vernemq https://github.com/erlio/vmq_plugin https://github.com/dergraf/vernemq_demo_plugin https://github.com/helium/plumtree http://asc.di.fct.unl.pt/~jleitao/pdf/srds07-leitao.pdf https://groups.google.com/forum/#!topic/mqtt/c1dlCdA5eOc