MQTT: A Pythonic introduction to the protocol for connected devices

MQTT
A Pythonic introduction to the protocol for
connected devices
Jonas Neubert
PYCON US 2023
Photo credit: Ginkgo Bioworks

View Slide

Photo credit: Ginkgo Bioworks

View Slide

The IT vs OT Divide
“OT”
Operational Technology
“IT”
Information Technology

View Slide

Process Data
Process Parameters
Sensor Readings
Control Signals
The IT vs OT Divide
Field Devices Control Logic Business Logic
“OT”
Operational Technology
“IT”
Information Technology
milliseconds to seconds per iteration
interfaces with the physical world
realtime & safety logic
minutes to months per iteration
interfaces to databases and APIs
optimization & customization

View Slide

View Slide

MQTT Basics
Lamps Demo
More MQTT Features
Micropython Demo
Advanced MQTT Topics
Table of Contents
https: // jonasneubert.com / pycon2023
[email protected]
slides:
contact me:

View Slide

Pub Sub
Workcell 1
Workcell 2
Conveyor
Robot
Sensor
Sensor Barcode Reader Switch
Light
Inventory
Management
Machine
Learning
Model
ERP
Historian
(Time Series
Database)
Human Machine
Interface 1
Human Machine
Interface 2
cool kids lingo for:
publish subscribe

View Slide

Pub Sub
Workcell 1
Workcell 2
Conveyor
Robot
Sensor
Light
Inventory
Management
Machine
Learning
Model
ERP
Historian
(Time Series
Database)
Human Machine
Interface 1
Human Machine
Interface 2
Broker

View Slide

Pub Sub
Workcell 1
Workcell 2
Conveyor
Robot
Sensor
Light
Inventory
Management
Machine
Learning
Model
ERP
Historian
(Time Series
Database)
Publish
Subscribe
Human Machine
Interface 1
Human Machine
Interface 2
Broker

View Slide

Pub Sub
Workcell 1
Workcell 2
Conveyor
Robot
Sensor
Light
Inventory
Management
Machine
Learning
Model
ERP
Historian
(Time Series
Database)
Publish
Subscribe
Broker
Human Machine
Interface 1
Human Machine
Interface 2

View Slide

Peer-to-peer
Workcell 1
Workcell 2
Conveyor
Robot
Sensor
Light
Inventory
Management
Machine
Learning
Model
ERP
Historian
(Time Series
Database)
Human Machine
Interface 1
Human Machine
Interface 2

View Slide

Peer-to-peer (aka: spaghetti)
Workcell 1
Workcell 2
Conveyor
Robot
Sensor
Light
Inventory
Management
Machine
Learning
Model
ERP
Historian
(Time Series
Database)
Human Machine
Interface 1
Human Machine
Interface 2

View Slide

myhouse/livingroom/lights/sofalamp
myhouse/+/lights/+
myhouse/livingroom/#
Every message is published to a topic:
my-arbitrary-topic-name
myhouse/livingroom/lights/sofalamp
site/area/line/unit/equipment/sensor/reading
spBv1.2////
homeassistant////config
ISA-95-ish hierarchy
Simple topic name
Made-up hierarchy
Sparkplug
Home Assistant
“all lights in all rooms”
a speciﬁc topic
“all devices in the living room”
Subscriptions are for topic ﬁlters:

View Slide

from paho.mqtt.client import Client
client = Client()
client.username_pw_set("pycon", "saltlake")
client.connect("localhost", 1883)
client.publish("livingroom/light/reading", "on")
Example: paho-mqtt
publisher

View Slide

def on_message(client, userdata, msg):
print(f"{msg.topic}: \n{msg.payload}\n")
client = Client(client_id="pycon-subscriber")
client.on_message = on_message
client.subscribe("livingroom/light/#")
client.loop_forever()
Example: paho-mqtt
subscriber

View Slide

import json
reading_light = "livingroom/light/00:17:88:01:03:ce:55:5f-0b/set"
sofa_lamp = "livingroom/light/00:17:88:01:01:1b:85:b9-0b/set"
client = Client()
client.publish(reading_light, json.dumps({"on": False}))
client.publish(sofa_lamp, json.dumps({"on": True}))
client.publish(sofa_lamp, json.dumps({"bri": 100, "hue": 40000}))
Example: paho-mqtt
& hue2mqtt

View Slide

MQTT History
1999 MQTT invented by Arlen Nipper and Andy
Stanford-Clark for monitoring pipelines over satellite
links
2010 v3.1 made available royalty-free
2010 Mosquitto broker released
2011 Facebook Messenger blog post
2013 v3.1 submitted as standard proposal to OASIS
2014 v3.1.1 published by OASIS
2019 v5 released
Approximate Release Dates
of selected other tools and
protocols
AMQP 2003
ActiveMQ 2004
RabbitMQ 2007
ZeroMQ 2009
NATS 2011
Kafka 2011
The Facebook Messenger blog post is here: https://engineering.fb.com/2011/08/12/android/building-facebook-messenger/

View Slide

Sequence Diagram
SUBSCRIBE "livingroom/lights/reading"
Light Switch Broker Reading Light
SUBACK "livingroom/lights/reading"
PUBLISH "livingroom/lights/reading" "on" id=1
PUBLISH livingroom/lights/reading payload= "on" id =1
1.a

View Slide

Sequence Diagram
SUBSCRIBE "livingroom/lights/reading"
PUBLISH lvrm/lights/reading payload= "on" id =1
2.a
CONNECT
CONNACK
CONNECT
CONNACK
DISCONNECT
DISCONNECT

View Slide

What if the subscriber shows up late?
SUBSCRIBE
"livingroom/lights/reading" "on" qos=1
PUBLISH "livingroom/lights/reading" "on" qos=1 id =1
3.a
CONNECT
CONNACK
CONNECT
CONNACK
PUBACK id =1

View Slide

Periodic Publishing (is not the answer)
SUBSCRIBE"livingroom/lights/reading"
PUBLISH "livingroom/lights/reading" "on" id =1
4.a
CONNECT
CONNACK
CONNECT
CONNACK

View Slide

Retained Messages (are the answer)
PUBLISH "livingroom/lights/reading" "on" id=1 retain=1
5.a
CONNECT
CONNACK
CONNECT
CONNACK
PUBLISH
"livingroom/lights/reading" "on" id=1 retain=1

View Slide

Retained Messages
6.a
CONNECT
CONNACK
CONNECT
CONNACK
PUBLISH
?

View Slide

Keep Alive and Heartbeat
7.a
CONNECT keepalive=60
CONNACK
CONNECT
CONNACK
PINGREQ
PINGRESP
PUBLISH "livingroom/lights/reading" "off" id=2 retain=1
PINGREQ
PINGRESP
PINGREQ
PINGRESP
PINGREQ
PINGRESP
PINGREQ
PINGRESP
PINGREQ
PINGRESP
PUBLISH "livingroom/lights/reading" "off" id=2 retain=1
PINGREQ and PINGRESP
messages are only
2 bytes long.

View Slide

Last Will and Testament
8.a
CONNECT keepalive=60 willtopic=”...” willmsg=”off”
CONNACK
CONNECT
CONNACK
PUBLISH
PINGREQ
PINGRESP
PINGREQ
PINGRESP
PUBLISH
”livingroom/lights/reading” "off" id=2
keepalive interval

View Slide

Quality of Service (QoS) 1
SUBSCRIBE "livingroom/lights/reading" qos=1
9.a
CONNECT keepalive=60 willtopic=”...” willmsg=”i’m dead”
CONNACK
CONNECT
CONNACK
PUBLISH "livingroom/lights/reading" "on" qos=1 id=1
PUBLISH ”livingroom/lights/reading” "on" qos=1 id=1 dup=1
PUBLISH ”livingroom/lights/reading” "on" qos=1 id=1
PUBLISH ”livingroom/lights/reading” "on" qos=1 id=1 dup=1
PUBACK id=1
PUBACK id=1
DISCONNECT
DISCONNECT

View Slide

Quality of Service (QoS) 2
PUBLISH "livingroom/lights/reading" "on" qos=2 id=1
10.a
CONNECT
CONNACK
CONNECT
CONNACK
DISCONNECT
PUBLISH "livingroom/lights/reading" "on" qos=2 id=1 dup=1
PUBLISH "livingroom/lights/reading" "on" qos=2 id=1 dup=1
DISCONNECT
PUBACK id=1
PUBRACK id=1
PUBREL id=1
PUBCOMP id=1
0 = at most once delivery
1 = at least once delivery
2 = exactly once delivery

View Slide

Wrapper or own
implementation?
Client
or
broker?
Event
loop
MQTT
versions
Persistent
Sessions
Retained
Messages
LWT QoS Github
stars
Project
Status
paho-mqtt Eclipse Foundation
reference
implementation
client custom +
callbacks
5.0, 3.1.1 ✔ ✔ ✔ 0, 1, 2 1.8k active
asyncio-mqtt Asyncio wrapper
for paho-mqtt
client asyncio 5.0, 3.1.1 ✔ ✔ ✔ 0, 1, 2 253 active
aiomqtt Minimal asyncio
wrapper for
paho-mqtt
client asyncio 5.0, 3.1.1 ✔ ✔ ✔ 0, 1, 2 54 “limited”
trio-paho-mqtt Trio wrapper for
paho-mqtt
client trio 5.0, 3.1.1 ✔ ✔ ✔ 0, 1, 2 11 dormant
gmqtt From-scratch
implementation
client asyncio +
callbacks
5.0, 3.1.1 ✘ ✔ ✔ 0, 1, 2 319 active
HBMQTT Deprecated,
From-scratch
implementation
both asyncio 3.1.1 ✘ ✔ ✘ 0, 1, 2 779 deprecated
aMQTT “LTS for HBMQTT” both asyncio 3.1.1 ✘ ✔ ✘ 0, 1, 2 81 active
mqttools From-scratch
implementation,
limited feature set
both asyncio 5.0 ✘ ✔ ✘ 0 52 dormant

View Slide

Micropython & CircuitPython
MicroPython tl;dr:
- Runs Python code on microcontrollers
- Implements a subset of Python language
- Adds new stdlib modules for doing
microcontroller stuff
- CircuitPython is a fork of Micropython
that is maintained by Adafruit
- Put your code into main.py and copy to
device using Thonny editor
(CircuitPython: code.py and drag and drop ﬁles)
MicroPython Logo: © Damien P. George, and others, Adafruit Blinka Logo: © Phillip Torrone for Adafruit Industries
MicroPython/
├─ main.py
├─ lib/
│ ├─ umqtt/
│ │ ├─ robust.py
│ │ ├─ simple.py

View Slide

Micropython: MQTT Snake
Raspberry Pi
Pico W
1.14” LCD Module
(Waveshare)
2x AA Battery

View Slide

MQTT Snake Demo

View Slide

MQTT version 5.0
Released in 2019. Backwards compatible. Adds new features, including:
- Shared subscriptions (load balanced subscribers)
- Topic aliases
- Request-response(s) semantics
- Flow Control
- More reasons codes (similar to HTTP status codes)
- Arbitrary “user properties” (similar to HTTP headers/ﬁelds)
Support for MQTT v5 in Python tooling:
✓ paho-mqtt, gmqtt, mqttools
✗ aMQTT/hbmqtt, umqtt.simple, umqtt.robust, micropython-mqtt-async
MQTT spec v5.0: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html
Blog post series about MQTTv5 features: https://www.emqx.com/en/mqtt/mqtt5

View Slide

Sparkplug B (aka Eclipse )
A content format definition for MQTT message payloads
- Defines topic namespace
- Defines payload format (using Google Protobufs)
- A specific way to use retained messages and last will messages
- Each message has these fields: Name, Alias, Time stamp, Data type, Value
- Discoverability, self-describing
- Edge Node vs Device
Python packages: Eclipse Tahu, mqtt-spb-wrapper
spBv1.2////

View Slide

Home Assistant MQTT Integration
Topic namespace for device discovery in Home Assistant:
homeassistant////config
https://www.home-assistant.io/integrations/mqtt/

View Slide

MQTT over x
MQTT over Websocket
For publishing/subscribing directly from a web browser. Adds headers to the
MQTT message to also support the WSS protocol.
MQTT over QUIC
Uses QUIC instead of TCP for Layer 3, inspired by HTTP/3. Experimental feature
in a few brokers.
MQTT over Kafka
Kafka MQTT Proxy is an addon service for your Kafka cluster. Allows publishing
MQTT messages directly to Kafka without intermediary.
MQTT over Zigbee, other IEEE 802.15.4, Bluetooth
see next slide

View Slide

MQTT-SN
- Used with Zigbee and other IEEE 802.15.4, Bluetooth
- Topic IDs (two bytes) as aliases for topic names
- Supports sleeping client devices (broker queues messages)
- QoS level -1 for session-less ﬁre-and-forget
- Last Will persists in session, and can be changed at any time
- Gateway discovery mechanism
for sensor networks
MQTT-SN spec v1.2: https://www.oasis-open.org/committees/download.php/66091/MQTT-SN_spec_v1.2.pdf
Version of MQTT for even-lower-power
devices and on non-TCP-IP networks.
An MQTT-SN gateway translates between
MQTT-SN clients and an MQTT broker. Is this a typo in the spec? MQTT-SN was
previously called MQTT-S (without “N”).
That caused confusion because of the
similarity to HTTP and HTTPS (where
“S” means “secure”).
Python tooling for MQTT-SN is sparse, no package recommendation.

View Slide

If you liked this talk you might also like:
Friday
11:30 –
noon
Into the Logisticverse: Improving
Efﬁciency in Transportation Networks
Uzoma Nicholas Muoh
Friday
2:30 –
3:00pm
Start thinking small: Next level Machine
Learning with TinyML and Python
Maria Jose Molina Contreras
Friday
3:15 –
3:45pm
Create interactive games using
MicroPython and electronics
Juliana Karoline de Sousa
Friday
5:00 –
5:45pm
Embedded Python Q&A Open Space
For Q&A and playing with hardware. Hosted by
Maria, Juliana Karoline, and Jonas. (Open door,
come and go any time.)
daily Adafruit CircuitPython Open Space
expo hall MicroPython Badge at Auth0 booth
What’s the URL for these slides again?
https://jonasneubert.com/pycon2023
Watching this on Youtube?
The best ways to contact me are
[email protected]
linkedin.com/in/jonasneubert
Will there be a Q&A session?
Yes!

View Slide

MQTT: A Pythonic introduction to the protocol for connected devices

MQTT: A Pythonic introduction to the protocol for connected devices

Jonas Neubert

More Decks by Jonas Neubert

Other Decks in Technology

Featured

Transcript