MQTT: A Pythonic introduction to the protocol for connected devices

Transcript

1. MQTT A Pythonic introduction to the protocol for connected devices Jonas Neubert PYCON US 2023 Photo credit: Ginkgo Bioworks
2. Photo credit: Ginkgo Bioworks
3. Photo credit: Ginkgo Bioworks
4. The IT vs OT Divide “OT” Operational Technology “IT” Information Technology
5. Process Data Process Parameters Sensor Readings Control Signals The IT vs OT Divide Field Devices Control Logic Business Logic “OT” Operational Technology “IT” Information Technology milliseconds to seconds per iteration interfaces with the physical world realtime & safety logic minutes to months per iteration interfaces to databases and APIs optimization & customization
7. MQTT Basics Lamps Demo More MQTT Features Micropython Demo Advanced MQTT Topics Table of Contents https: // jonasneubert.com / pycon2023 jn@jonasneubert.com slides: contact me:
8. Pub Sub Workcell 1 Workcell 2 Conveyor Robot Sensor Sensor Barcode Reader Switch Light Inventory Management Machine Learning Model ERP Historian (Time Series Database) Human Machine Interface 1 Human Machine Interface 2 cool kids lingo for: publish subscribe
9. Pub Sub Workcell 1 Workcell 2 Conveyor Robot Sensor Sensor Barcode Reader Switch Light Inventory Management Machine Learning Model ERP Historian (Time Series Database) Human Machine Interface 1 Human Machine Interface 2 Broker
10. Pub Sub Workcell 1 Workcell 2 Conveyor Robot Sensor Sensor Barcode Reader Switch Light Inventory Management Machine Learning Model ERP Historian (Time Series Database) Publish Subscribe Human Machine Interface 1 Human Machine Interface 2 Broker
11. Pub Sub Workcell 1 Workcell 2 Conveyor Robot Sensor Sensor Barcode Reader Switch Light Inventory Management Machine Learning Model ERP Historian (Time Series Database) Publish Subscribe Human Machine Interface 1 Human Machine Interface 2 Broker
12. Pub Sub Workcell 1 Workcell 2 Conveyor Robot Sensor Sensor Barcode Reader Switch Light Inventory Management Machine Learning Model ERP Historian (Time Series Database) Publish Subscribe Broker Human Machine Interface 1 Human Machine Interface 2
13. Peer-to-peer Workcell 1 Workcell 2 Conveyor Robot Sensor Sensor Barcode Reader Switch Light Inventory Management Machine Learning Model ERP Historian (Time Series Database) Human Machine Interface 1 Human Machine Interface 2
14. Peer-to-peer (aka: spaghetti) Workcell 1 Workcell 2 Conveyor Robot Sensor Sensor Barcode Reader Switch Light Inventory Management Machine Learning Model ERP Historian (Time Series Database) Human Machine Interface 1 Human Machine Interface 2
15. myhouse/livingroom/lights/sofalamp myhouse/+/lights/+ myhouse/livingroom/# Every message is published to a topic: my-arbitrary-topic-name myhouse/livingroom/lights/sofalamp site/area/line/unit/equipment/sensor/reading spBv1.2/<GroupID>/<EdgeNodeID>/<DeviceID>/<DataType> homeassistant/<ComponentType>/<NodeID>/<DeviceID>/config ISA-95-ish hierarchy Simple topic name Made-up hierarchy Sparkplug Home Assistant “all lights in all rooms” a specific topic “all devices in the living room” Subscriptions are for topic filters:
16. from paho.mqtt.client import Client client = Client() client.username_pw_set("pycon", "saltlake") client.connect("localhost", 1883) client.publish("livingroom/light/reading", "on") Example: paho-mqtt publisher
17. from paho.mqtt.client import Client def on_message(client, userdata, msg): print(f"{msg.topic}: \n{msg.payload}\n") client = Client(client_id="pycon-subscriber") client.on_message = on_message client.connect("localhost", 1883) client.subscribe("livingroom/light/#") client.loop_forever() Example: paho-mqtt subscriber
18. from paho.mqtt.client import Client import json reading_light = "livingroom/light/00:17:88:01:03:ce:55:5f-0b/set" sofa_lamp = "livingroom/light/00:17:88:01:01:1b:85:b9-0b/set" client = Client() client.connect("localhost", 1883) client.publish(reading_light, json.dumps({"on": False})) client.publish(sofa_lamp, json.dumps({"on": True})) client.publish(sofa_lamp, json.dumps({"bri": 100, "hue": 40000})) Example: paho-mqtt & hue2mqtt
19. MQTT History 1999 MQTT invented by Arlen Nipper and Andy Stanford-Clark for monitoring pipelines over satellite links 2010 v3.1 made available royalty-free 2010 Mosquitto broker released 2011 Facebook Messenger blog post 2013 v3.1 submitted as standard proposal to OASIS 2014 v3.1.1 published by OASIS 2019 v5 released Approximate Release Dates of selected other tools and protocols AMQP 2003 ActiveMQ 2004 RabbitMQ 2007 ZeroMQ 2009 NATS 2011 Kafka 2011 The Facebook Messenger blog post is here: https://engineering.fb.com/2011/08/12/android/building-facebook-messenger/
20. Sequence Diagram SUBSCRIBE "livingroom/lights/reading" Light Switch Broker Reading Light SUBACK "livingroom/lights/reading" PUBLISH "livingroom/lights/reading" "on" id=1 PUBLISH livingroom/lights/reading payload= "on" id =1 Light Switch Broker Reading Light 1.a
21. Sequence Diagram SUBSCRIBE "livingroom/lights/reading" Light Switch Broker Reading Light SUBACK "livingroom/lights/reading" PUBLISH "livingroom/lights/reading" "on" id=1 PUBLISH lvrm/lights/reading payload= "on" id =1 Light Switch Broker Reading Light 2.a CONNECT CONNACK CONNECT CONNACK DISCONNECT DISCONNECT
22. What if the subscriber shows up late? SUBSCRIBE "livingroom/lights/reading" "on" qos=1 Light Switch Broker Reading Light SUBACK "livingroom/lights/reading" PUBLISH "livingroom/lights/reading" "on" qos=1 id =1 Light Switch Broker Reading Light 3.a CONNECT CONNACK CONNECT CONNACK PUBACK id =1
23. Periodic Publishing (is not the answer) SUBSCRIBE"livingroom/lights/reading" Light Switch Broker Reading Light SUBACK "livingroom/lights/reading" PUBLISH "livingroom/lights/reading" "on" id =1 Light Switch Broker Reading Light 4.a CONNECT CONNACK CONNECT CONNACK PUBLISH "livingroom/lights/reading" "on" id =2 PUBLISH "livingroom/lights/reading" "on" id =3 PUBLISH "livingroom/lights/reading" "on" id =4 PUBLISH "livingroom/lights/reading" "on" id =4
24. Retained Messages (are the answer) SUBSCRIBE"livingroom/lights/reading" Light Switch Broker Reading Light SUBACK "livingroom/lights/reading" PUBLISH "livingroom/lights/reading" "on" id=1 retain=1 Light Switch Broker Reading Light 5.a CONNECT CONNACK CONNECT CONNACK PUBLISH "livingroom/lights/reading" "on" id=1 retain=1
25. Retained Messages SUBSCRIBE"livingroom/lights/reading" Light Switch Broker Reading Light SUBACK "livingroom/lights/reading" PUBLISH "livingroom/lights/reading" "on" id=1 retain=1 Light Switch Broker Reading Light 6.a CONNECT CONNACK CONNECT CONNACK PUBLISH "livingroom/lights/reading" "on" id=1 retain=1 ?
26. Keep Alive and Heartbeat SUBSCRIBE"livingroom/lights/reading" Light Switch Broker Reading Light SUBACK "livingroom/lights/reading" PUBLISH "livingroom/lights/reading" "on" id=1 retain=1 7.a CONNECT keepalive=60 CONNACK CONNECT CONNACK PUBLISH "livingroom/lights/reading" "on" id=1 retain=1 PINGREQ PINGRESP PUBLISH "livingroom/lights/reading" "off" id=2 retain=1 PINGREQ PINGRESP PINGREQ PINGRESP PINGREQ PINGRESP PINGREQ PINGRESP PINGREQ PINGRESP Light Switch Broker Reading Light PUBLISH "livingroom/lights/reading" "off" id=2 retain=1 PINGREQ and PINGRESP messages are only 2 bytes long.
27. Last Will and Testament SUBSCRIBE"livingroom/lights/reading" Light Switch Broker Reading Light SUBACK "livingroom/lights/reading" PUBLISH "livingroom/lights/reading" "on" id=1 retain=1 Light Switch Broker Reading Light 8.a CONNECT keepalive=60 willtopic=”...” willmsg=”off” CONNACK CONNECT CONNACK PUBLISH "livingroom/lights/reading" "on" id=1 retain=1 PINGREQ PINGRESP PINGREQ PINGRESP PUBLISH ”livingroom/lights/reading” "off" id=2 keepalive interval
28. Quality of Service (QoS) 1 SUBSCRIBE "livingroom/lights/reading" qos=1 Light Switch Broker Reading Light SUBACK "livingroom/lights/reading" Light Switch Broker Reading Light 9.a CONNECT keepalive=60 willtopic=”...” willmsg=”i’m dead” CONNACK CONNECT CONNACK PUBLISH "livingroom/lights/reading" "on" qos=1 id=1 PUBLISH ”livingroom/lights/reading” "on" qos=1 id=1 dup=1 PUBLISH ”livingroom/lights/reading” "on" qos=1 id=1 PUBLISH ”livingroom/lights/reading” "on" qos=1 id=1 dup=1 PUBACK id=1 PUBACK id=1 DISCONNECT DISCONNECT
29. Quality of Service (QoS) 2 SUBSCRIBE "livingroom/lights/reading" qos=1 Light Switch Broker Reading Light SUBACK "livingroom/lights/reading" PUBLISH "livingroom/lights/reading" "on" qos=2 id=1 10.a CONNECT CONNACK CONNECT CONNACK SUBSCRIBE "livingroom/lights/reading" qos=2 PUBLISH "livingroom/lights/reading" "on" id=1 Light Switch Broker Reading Light DISCONNECT PUBLISH "livingroom/lights/reading" "on" qos=2 id=1 dup=1 PUBLISH "livingroom/lights/reading" "on" qos=2 id=1 dup=1 SUBACK "livingroom/lights/reading" DISCONNECT PUBACK id=1 PUBRACK id=1 PUBREL id=1 PUBCOMP id=1 0 = at most once delivery 1 = at least once delivery 2 = exactly once delivery
30. Wrapper or own implementation? Client or broker? Event loop MQTT versions Persistent Sessions Retained Messages LWT QoS Github stars Project Status paho-mqtt Eclipse Foundation reference implementation client custom + callbacks 5.0, 3.1.1 ✔ ✔ ✔ 0, 1, 2 1.8k active asyncio-mqtt Asyncio wrapper for paho-mqtt client asyncio 5.0, 3.1.1 ✔ ✔ ✔ 0, 1, 2 253 active aiomqtt Minimal asyncio wrapper for paho-mqtt client asyncio 5.0, 3.1.1 ✔ ✔ ✔ 0, 1, 2 54 “limited” trio-paho-mqtt Trio wrapper for paho-mqtt client trio 5.0, 3.1.1 ✔ ✔ ✔ 0, 1, 2 11 dormant gmqtt From-scratch implementation client asyncio + callbacks 5.0, 3.1.1 ✘ ✔ ✔ 0, 1, 2 319 active HBMQTT Deprecated, From-scratch implementation both asyncio 3.1.1 ✘ ✔ ✘ 0, 1, 2 779 deprecated aMQTT “LTS for HBMQTT” both asyncio 3.1.1 ✘ ✔ ✘ 0, 1, 2 81 active mqttools From-scratch implementation, limited feature set both asyncio 5.0 ✘ ✔ ✘ 0 52 dormant
31. Micropython & CircuitPython MicroPython tl;dr: - Runs Python code on microcontrollers - Implements a subset of Python language - Adds new stdlib modules for doing microcontroller stuff - CircuitPython is a fork of Micropython that is maintained by Adafruit - Put your code into main.py and copy to device using Thonny editor (CircuitPython: code.py and drag and drop files) MicroPython Logo: © Damien P. George, and others, Adafruit Blinka Logo: © Phillip Torrone for Adafruit Industries MicroPython/ ├─ main.py ├─ lib/ │ ├─ umqtt/ │ │ ├─ robust.py │ │ ├─ simple.py
32. Micropython: MQTT Snake Raspberry Pi Pico W 1.14” LCD Module (Waveshare) 2x AA Battery
33. MQTT Snake Demo
34. MQTT version 5.0 Released in 2019. Backwards compatible. Adds new features, including: - Shared subscriptions (load balanced subscribers) - Topic aliases - Request-response(s) semantics - Flow Control - More reasons codes (similar to HTTP status codes) - Arbitrary “user properties” (similar to HTTP headers/fields) Support for MQTT v5 in Python tooling: ✓ paho-mqtt, gmqtt, mqttools ✗ aMQTT/hbmqtt, umqtt.simple, umqtt.robust, micropython-mqtt-async MQTT spec v5.0: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html Blog post series about MQTTv5 features: https://www.emqx.com/en/mqtt/mqtt5
35. Sparkplug B (aka Eclipse ) A content format definition for MQTT message payloads - Defines topic namespace - Defines payload format (using Google Protobufs) - A specific way to use retained messages and last will messages - Each message has these fields: Name, Alias, Time stamp, Data type, Value - Discoverability, self-describing - Edge Node vs Device Python packages: Eclipse Tahu, mqtt-spb-wrapper spBv1.2/<Group ID>/<Edge Node ID>/<Device ID>/<Data Type>
36. Home Assistant MQTT Integration Topic namespace for device discovery in Home Assistant: homeassistant/<Component Type>/<Node ID>/<Device ID>/config https://www.home-assistant.io/integrations/mqtt/
37. MQTT over x MQTT over Websocket For publishing/subscribing directly from a web browser. Adds headers to the MQTT message to also support the WSS protocol. MQTT over QUIC Uses QUIC instead of TCP for Layer 3, inspired by HTTP/3. Experimental feature in a few brokers. MQTT over Kafka Kafka MQTT Proxy is an addon service for your Kafka cluster. Allows publishing MQTT messages directly to Kafka without intermediary. MQTT over Zigbee, other IEEE 802.15.4, Bluetooth see next slide
38. MQTT-SN - Used with Zigbee and other IEEE 802.15.4, Bluetooth - Topic IDs (two bytes) as aliases for topic names - Supports sleeping client devices (broker queues messages) - QoS level -1 for session-less fire-and-forget - Last Will persists in session, and can be changed at any time - Gateway discovery mechanism for sensor networks MQTT-SN spec v1.2: https://www.oasis-open.org/committees/download.php/66091/MQTT-SN_spec_v1.2.pdf Version of MQTT for even-lower-power devices and on non-TCP-IP networks. An MQTT-SN gateway translates between MQTT-SN clients and an MQTT broker. Is this a typo in the spec? MQTT-SN was previously called MQTT-S (without “N”). That caused confusion because of the similarity to HTTP and HTTPS (where “S” means “secure”). Python tooling for MQTT-SN is sparse, no package recommendation.
39. If you liked this talk you might also like: Friday 11:30 – noon Into the Logisticverse: Improving Efficiency in Transportation Networks Uzoma Nicholas Muoh Friday 2:30 – 3:00pm Start thinking small: Next level Machine Learning with TinyML and Python Maria Jose Molina Contreras Friday 3:15 – 3:45pm Create interactive games using MicroPython and electronics Juliana Karoline de Sousa Friday 5:00 – 5:45pm Embedded Python Q&A Open Space For Q&A and playing with hardware. Hosted by Maria, Juliana Karoline, and Jonas. (Open door, come and go any time.) daily Adafruit CircuitPython Open Space expo hall MicroPython Badge at Auth0 booth What’s the URL for these slides again? https://jonasneubert.com/pycon2023 Watching this on Youtube? The best ways to contact me are jn@jonasneubert.com linkedin.com/in/jonasneubert Will there be a Q&A session? Yes!