Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Verne.mq

 Verne.mq

André Graf is co-founder and CTO of erl.io, a swiss company with a focus on Erlang and distributed systems engineering.

He presented VerneMQ, a new Erlang powered MQTT message broker his company opensourced a few weeks ago. His talk covers a quick introduction to the MQTT protocol and some of its application scenarios. Special emphasis will then be laid on the VerneMQ architecture and its take on scalability and extensibility.

erlang.paris

June 29, 2015
Tweet

More Decks by erlang.paris

Other Decks in Technology

Transcript

  1. erlang.paris Meetup / June 29th 2015, Paris / André Graf (@der_graf)
    VerneMQ - A distributed MQTT message broker
    developed in Erlang

    View full-size slide

  2. Intro
    - André Graf, Co-Founder at Erl.io (2012)
    - Programming in Erlang since 2008
    - Basel Switzerland
    @der_graf

    View full-size slide

  3. Simple Publish/Subscribe protocol
    over TCP/IP that utilizes
    a centralized message broker
    MQTT

    View full-size slide

  4. Different use cases…
    MQTT

    View full-size slide

  5. https://www.flickr.com/mmmavocado/

    View full-size slide

  6. https://www.flickr.com/photos/christianhaugen

    View full-size slide

  7. https://www.flickr.com/photos/janitors

    View full-size slide

  8. Different use cases…
    ...but similar requirements.
    MQTT

    View full-size slide

  9. constrained devices in
    constrained environments
    MQTT

    View full-size slide

  10. | different QoS Levels
    | small header sizes (min 2B)
    | durable sessions
    | last will testament
    MQTT

    View full-size slide

  11. Some MQTT Facts
    - Current Spec Version 3.1.1
    - open OASIS Standard since 2014
    - Invented 1998 at IBM and Arcom

    View full-size slide

  12. VerneMQ Architecture from a
    bird’s eye perspective

    View full-size slide

  13. Erlang VM
    ninenines/ranch
    ninenines/cowboy
    vmq_server
    vmq_plugin
    helium/eleveldb
    helium/plumtree
    uwiger/jobs
    vmq_commons
    feuerlabs/
    exometer_core
    vmq_passwd
    vmq_acl
    vmq_bridge
    vmq_systree
    vmq_snmp
    vmq_graphite
    larshesel/
    vmq_elixir

    View full-size slide

  14. erlio/vernemq release project
    basho/nodepackage
    basho/cuttlefish
    basho/clique
    rebar/rebar3

    View full-size slide

  15. vmq_reg_trie
    <>
    3 processes / MQTT connection
    vmq_server
    vmq_session
    <>
    vmq_ranch
    <>
    vmq_queue
    <>
    start_link start_link
    vmq_reg
    <>
    vmq_reg_sup
    <>
    vmq_reg_trie
    <>
    monitor
    ETS
    ETS

    View full-size slide

  16. vmq_reg_trie
    <>
    vmq_server
    vmq_session
    <>
    vmq_ranch
    <> sync
    vmq_reg
    <>
    vmq_reg_sup
    <>
    vmq_reg_trie
    <>
    ETS
    vmq_session
    <>
    vmq_ranch
    <>
    vmq_queue
    <>
    async batch/async
    match
    fold
    MQTT publish example
    TCP Conn Client A
    TCP Conn Client B
    ETS
    PUB
    /a/b/c
    SUB
    /a/b/+

    View full-size slide

  17. Clustering VerneMQ

    View full-size slide

  18. Distribution
    What is ACTIVELY distributed?
    | Subscriber Data
    | Online Messages
    | Cluster Status Information
    | ...

    View full-size slide

  19. Distribution
    What is NOT distributed?
    | offline messages (future)

    View full-size slide

  20. Distributing Subscriber Data
    - Initially using Mnesia
    - Move to Plumtree/LevelDB
    Paradigm Shift:
    Consistent -> eventual Consistent

    View full-size slide

  21. Distributing online Messages
    - Initially over Erlang Distribution layer
    - Move to own communication layer
    - buffering in case of ‘nodedown’s
    - batching requests for higher throughput

    View full-size slide

  22. Extending VerneMQ

    View full-size slide

  23. erlio/vmq_plugin
    %% iterates over all Plugins serving the Hook 'HookName'
    vmq_plugin:all(HookName :: atom(), HookArgs :: [any()]).
    %% iterates over all Plugins serving the Hook 'HookName'
    %% until one returns 'ok' | {'ok', any()}
    vmq_plugin:all_till_ok(HookName :: atom(), HookArgs :: [any()]).
    %% calls the top priority Plugin that serves the Hook 'HookName'
    vmq_plugin:only(HookName :: atom(), HookArgs :: [any()]).

    View full-size slide

  24. erlio/vmq_plugin
    %% enabling an OTP Application Plugin
    vmq_plugin_mgr:enable_plugin(Application :: atom()).
    %% in Application.app.src
    {env, [
    {vmq_plugin_hooks, [
    {HookName, Module, Function, Arity, []}
    ...
    ]}
    ]}

    View full-size slide

  25. erlio/vmq_plugin
    Available Hooks:
    - auth_on_register/5
    - auth_on_publish/6
    - auth_on_subscribe/3
    - on_register/3
    - on_publish/6
    - on_subscribe/3
    - on_deliver/6 (planned)

    View full-size slide

  26. erlio/vmq_plugin
    Available Hooks:
    - msg_store_write_sync/2
    - msg_store_write_async/2
    - msg_store_delete_sync/1
    - msg_store_delete_async/1
    - msg_store_fold/2
    - msg_store_read/1

    View full-size slide

  27. erlio/vmq_plugin
    Checkout vmq_plugin and the Demo Plugin App
    https://github.com/erlio/vmq_plugin
    https://github.com/dergraf/vernemq_demo_plugin

    View full-size slide

  28. To Sum Up
    - MQTT is a great Protocol for various use
    cases (monitoring/metering/mobile/..)
    - VerneMQ brings scalability and powerful
    extensibility to the MQTT world.
    - Erl.io provides support and development
    services for VerneMQ

    View full-size slide

  29. http://verne.mq | http://erl.io
    @der_graf | @vernemq | @erlio_basel
    Merci Beaucoup!
    Je suis content de répondre a vos questions.

    View full-size slide

  30. References
    https://verne.mq
    http://mqtt.org
    https://github.com/erlio/vernemq
    https://github.com/erlio/vmq_plugin
    https://github.com/dergraf/vernemq_demo_plugin
    https://github.com/helium/plumtree
    http://asc.di.fct.unl.pt/~jleitao/pdf/srds07-leitao.pdf
    https://groups.google.com/forum/#!topic/mqtt/c1dlCdA5eOc

    View full-size slide