MQTT Best Practices

This talk will give a brief introduction to MQTT, what it is used for and shows how to apply best practices for message topics and payload design, as well as publishing and persisting with some examples.

This talk was given at the PHP USERGROUP DRESDEN e.V. on November 13, 2023

Transcript

1. ©2023 Public. All Rights Reserved. Best Practices PHP USERGROUP DRESDEN e.V. — Meetup V/2023 November 13, 2023 at portrino GmbH
2. ©2023 Public. All Rights Reserved. What is MQTT?
3. ©2023 Public. All Rights Reserved. First Google answer: MQTT stands for Message Queuing Telemetry Transport. It is a lightweight messaging protocol for use in cases where clients need a small code footprint and are connected to unreliable networks or networks with limited bandwidth resources.
4. ©2023 Public. All Rights Reserved. From the specification: MQTT is a Client Server publish/subscribe messaging transport protocol. It is light weight, open, simple, and designed to be easy to implement. These characteristics make it ideal for use in many situations, including constrained environments such as for communication in Machine to Machine (M2M) and Internet of Things (IoT) contexts where a small code footprint is required and/or network bandwidth is at a premium.
5. ©2023 Public. All Rights Reserved. https://docs.oasis-​ open.org/mqtt/mqtt/v5.0/mqtt-​ v5.0.html mqtt.org MQTT - The Standard for IoT Messaging A lightweight messaging protocol for small sensors and mobile devices, optimized for high-latency or unreliable networks, enabling a Connected World and the Internet of Things https://mqtt.org
6. ©2023 Public. All Rights Reserved. Source: https://mqtt.org Classic publish / subscribe
7. ©2023 Public. All Rights Reserved. Telemetry only?
8. ©2023 Public. All Rights Reserved. Well, no. In many IoT applications MQTT messages are used as the primary communication channel between apps and/or devices .
9. ©2023 Public. All Rights Reserved. Because... It is made for unreliable networks, but you can enforce message delivery by setting a certain quality of service (QoS). So why write another API?
10. ©2023 Public. All Rights Reserved. So... You can use MQTT messages also to query status on demand or send commands to applications or devices. Both require a request and response message.
11. ©2023 Public. All Rights Reserved. A Types of messages Telemetry messages Read-​ only or fan-​ in data of measurements transmitted from a device to be persisted and eventually processed by an application Emitted regularly by the device No acknowledgements (QoS=0), no response messages State messages State-​ Information data transmitted from a device or an application, so that any other application can process or act on that information Emitted regularly by device or requested by an application Acknowledgements (QoS=1), maybe request & response messages Command messages Instruction data transmitted from an application to instruct a target application to do something. Requested by an application, responded by the target application Acknowledgements (QoS=2), request & response messages D DB D A response request A A response request A
12. ©2023 Public. All Rights Reserved. What was QoS again?
13. ©2023 Public. All Rights Reserved. Quality of Service (QoS) in MQTT messaging is an agreement between sender and receiver on the guarantee of delivering a message. There are three levels of QoS: 0 - at most once 1 - at least once 2 - exactly once
14. ©2023 Public. All Rights Reserved. When a client is publishing to a broker, the client determines the QoS level for that message. When the broker sends the message to a subscribing client, the QoS set by the first client for that message is used again. So effectively the original publisher of any message determines the QoS of the message all the way to the final recipients.
15. ©2023 Public. All Rights Reserved. QoS Level 0 - at most once This is the simplest, lowest-​ overhead method of sending a message. The client simply publishes the message, and there is no acknowledgement by the broker.
16. ©2023 Public. All Rights Reserved. QoS Level 1 - at least once This method guarantees that the message will be transferred successfully to the broker. The broker sends an acknowledgement back to the sender, but in the event that that the acknowledgement is lost the sender won't realise the message has got through, so will send the message again. The client will re-​ send until it gets the broker's acknowledgement. This means that sending is guaranteed, although the message may reach the broker more than once.
17. ©2023 Public. All Rights Reserved. QoS Level 2 - exactly once This is the highest level of service, in which there is a sequence of four messages between the sender and the receiver, a kind of handshake to confirm that the main message has been sent and that the acknowledgement has been received. When the handshake has been completed, both sender and receiver are sure that the message was sent exactly once.
18. ©2023 Public. All Rights Reserved. MQTT Broker can retain one message per topic. If a client subscribes to a topic, it will receive the last published message from the broker immediately. This is very useful for state messages where apps act on state information from devices. One more thing: Retained messages
19. ©2023 Public. All Rights Reserved. Message topics
20. ©2023 Public. All Rights Reserved. From the specification: The following rules apply to Topic Names and Topic Filters: All Topic Names and Topic Filters MUST be at least one character long Topic Names and Topic Filters are case sensitive Topic Names and Topic Filters can include the space character A leading or trailing ‘/’ creates a distinct Topic Name or Topic Filter A Topic Name or Topic Filter consisting only of the ‘/’ character is valid Topic Names and Topic Filters MUST NOT include the null character (Unicode U+0000) [Unicode] Topic Names and Topic Filters are UTF-8 Encoded Strings; they MUST NOT encode to more than 65,535 bytes Message Topics
21. ©2023 Public. All Rights Reserved. OK, so... topics are arbitrary UTF-8 strings and something with slashes.
22. ©2023 Public. All Rights Reserved. Distinguish the types of message with a topic prefix Add routing information towards the device & the device/application name that produces the message's payload Add the type of data you transmit or request from the device or application Prefixes for types of messages could be: tlm for telemetry sta for state information cmd for commands 1. 2. 3. Message Topics Best Practices
23. ©2023 Public. All Rights Reserved. Telemetry topic example tlm/residence/vacation02/room/bath03/heater/heater01/temperature Type of message tlm = Telemetry Routing information Residence > Room > Heater Type of data / Measurement Temperature Name of device First Heater Temperature of the first heater in the third bathroom of my second vacation residence
24. ©2023 Public. All Rights Reserved. State topic examples sta/residence/vacation02/room/bath03/light/ceiling01/req/switched-​ on Type of message sta = State information Routing information Residence > Room > Light Type of data Request the switched-​ on status Name of device Ceiling light 1 Is the ceiling light switched on in my third bathroom of my second vacation residence? sta/mobile/iphone05/app/residence-​ commander/res/light/switched-​ on Type of message sta = State information Routing information Device / application Type of data Response to a switched-​ on request of a light Name of application Residence Commander Respond to the "Residence Commander" app on my fifth iPhone, please!
25. ©2023 Public. All Rights Reserved. State topic examples cmd/residence/vacation02/room/bath03/light/ceiling01/req/dim Type of message sta = State information Routing information Residence > Room > Light Type of data Instruction to dim the light Name of device Ceiling light 1 Please dim the ceiling light in my third bathroom of my second vacation residence. cmd/mobile/iphone05/app/residence-​ commander/res/light/dim Type of message sta = State information Routing information Device / application Type of data Response to a dim request of a light Name of application Residence Commander Respond to the "Residence Commander" app on my fifth iPhone, please!
26. ©2023 Public. All Rights Reserved. Advantages Messages of different types do not overlap on the same topic Messages of each type can be processed individually regarding their purpose Devices and applications have distinct hierarchical routing information for each message type Wildcard subscriptions are possible at every logical group of the topic Devices and applications can be added without conflicts Topics reflect the domain's naming, structure and capabilities Message Topics Best Practices
27. ©2023 Public. All Rights Reserved. Message payloads
28. ©2023 Public. All Rights Reserved. Keep the structure as simple as possible Transmit only data that is relevant for one specific data point Omit units, because the name of the measurement and the value should implicitly define the unit, respectively the processing application must know the units Add meta data only if necessary and if it does not fit into the message topic. Express clearly that it is meta data and don't encode multiple meta information in one key or value. Payload is expressed in JSON format and should have a JSON schema for validation Telemetry message payloads Example payload for a message with the topic tlm/residence/vacation02/room/bath03/heater/heater01/temperature { "timestamp": 1693406941326, "T": 21.5000000000002, "meta": { "device_serial": "65ebb617-​ b4a2-4c93" } } The data point consists of one T value. The device serial number (dsn) is meta information.
29. ©2023 Public. All Rights Reserved. Keep the structure as simple as possible The requester creates a unique request ID in order to match the response message in return. The requester tells the target topic to which the response should be published to. Payload is expressed in JSON format and should have a JSON schema for validation The responder must include the request ID in the response and transmit the response to the given response topic. The response payload contains a timestamp and the requested status Optionally meta data can be added to the response State message payloads Example payload for a status request message with the topic sta/residence/vacation02/room/bath03/light/ceiling01/req/switched-​ on { "req-​ id": "758b1b2c-4c5b-4a6a-8dfc-​ b4af7425e6fa", "res-​ to": "sta/mobile/iphone05/app/residence/..." } Example payload for a status response message with the topic sta/mobile/iphone05/app/residence-​ commander/res/light/switched-​ on { "req-​ id": "758b1b2c-4c5b-4a6a-8dfc-​ b4af7425e6fa", "timestamp": 1693406941326, "SwitchedOn": false, "meta": { "message": "No light bulb installed" } }
30. ©2023 Public. All Rights Reserved. Keep the structure as simple as possible The requester creates a unique request ID in order to match the response message in return. The requester tells the target topic to which the response should be published to. Payload is expressed in JSON format and should have a JSON schema for validation The responder must include the request ID in the response and transmit the response to the given response topic. The response payload contains a timestamp, the a success indicator and a response message. Optionally meta data can be added to the response. Command message payloads Example payload for a command message with the topic cmd/residence/vacation02/room/bath03/light/ceiling01/req/dim { "req-​ id": "758b1b2c-4c5b-4a6a-8dfc-​ b4af7425e6fa", "luminosity": 0.5, "res-​ to": "cmd/mobile/iphone05/app/residence/..." } Example payload for a status response message with the topic cmd/mobile/iphone05/app/residence-​ commander/res/light/dim { "req-​ id": "758b1b2c-4c5b-4a6a-8dfc-​ b4af7425e6fa", "timestamp": 1693406941326, "success": false, "message": "Light was already dimmed to 50% luminosity", "meta": { "error": "Too much power consumption, dimming down." } }
31. ©2023 Public. All Rights Reserved. Data points
32. ©2023 Public. All Rights Reserved. Data points for telemetry A telemetry message should contain only one data point and should be sent to one specific topic. A datapoint can consist of more than one value, for example geo-​ coordinates. Telemetry data points are always measured numbers or booleans expressed as numbers (0 or 1), but never strings.
33. ©2023 Public. All Rights Reserved. Examples { "timestamp": 1693406941326, "T": 21.5000000000002, "meta": { "device_serial": "65ebb617-​ b4a2-4c93" } } Single value data point { "timestamp": 1693406941326, "Lat": 52.581860, "Long": 13.527040 "meta": { "host": "portrino GmbH" } } Multi value data point (geo coordinates)
34. ©2023 Public. All Rights Reserved. Data persistence (example)
35. ©2023 Public. All Rights Reserved. Message persistance Mapping topics & messages to InfluxDB line protocol The topics and messages as described can be mapped to InfluxDB's line protocol directly using: Telegraf's standard MQTT consumer plugin with topic parsing capabilities https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer Telegraf's standard JSON input data format parser https://github.com/influxdata/telegraf/tree/master/plugins/parsers/json
36. ©2023 Public. All Rights Reserved. Message persistance Mapping topics & messages to InfluxDB line protocol Example payload for a telemetry message with the topic tlm/residence/vacation02/room/bath03/heater/heater01/temperature { "timestamp": 1693406941326, "T": 21.5000002, "meta": { "device_serial": "65ebb617-​ b4a2-4c93" } } + ## ... data_format = "json" json_strict = true json_query = "" tag_keys = [ "meta_*" ] json_string_fields = [] json_name_key = "" json_time_key = "timestamp" json_time_format = "unix_ms" json_timezone = "UTC" [[inputs.mqtt_consumer.topic_parsing]] topic = "tlm/residence/+/room/+/+/+/+" measurement = "_/_/_/_/_/_/_/measurement" tags = "_/_/residence/_/room/_/device/_" Example configuration for mqtt_consumer plugin (only the json format & topic parsing config is shown here) = temperature,residence="vacation02",room="bath03",device="heater01", meta_device_serial="65ebb617-​ b4a2-4c93" T=21.5000002 1693406941326 Generic mapping to InfluxDB line protocol measurement tag list fieldset timestamp There can be multiple topic parsing configurations in one plugin configuration
37. ©2023 Public. All Rights Reserved. Source: https://awesome.influxdata.com/docs/part-​ 1/introduction-​ to-​ influxdb/
38. ©2023 Public. All Rights Reserved. Sources MQTT meaning, first google answer https://www.paessler.com/it-​ explained/mqtt MQTT protocol specification v5.0 https://docs.oasis-​ open.org/mqtt/mqtt/v5.0/mqtt-​ v5.0.html Getting started with MQTT https://mqtt.org MQTT design best practices https://docs.aws.amazon.com/whitepapers/latest/designing-​ mqtt-​ topics-​ aws-​ iot-​ core/mqtt-​ design-​ best-​ practices.html MQTT Quality of Service (QoS) explained https://assetwolf.com/learn/mqtt-​ qos-​ understanding-​ quality-​ of-​ service JSON Schema Validation https://json-​ schema.org https://json-​ schema.hyperjump.io/ Telegraf's standard MQTT consumer plugin with topic parsing capabilities https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer Telegraf's standard JSON input data format parser https://github.com/influxdata/telegraf/tree/master/plugins/parsers/json
39. ©2023 Public. All Rights Reserved. MPOWR IT GmbH Enderstr. 94 01277 Dresden Deutschland Geschäftsführung: Patrick Pächnatz Holger Woltersdorf Web: https://mpowr.it E-​ Mail: hello@mpowr.it HRB 43777 Amtsgericht Dresden USt-​ ID: DE359347772 THANK YOU! Holger Woltersdorf CO-​ Founder & CEO