Using NSQ in Python

The talk was performed at the PyCon Russia 2021 conference in collaboration with Oleg Ivashov.

Transcript

1. NSQ в Python Serge Matveenko Oleg Ivashov assaia.com
2. The Road to NSQ Serge Matveenko
3. Assaia ApronAI • Real-time • Computer Vision • Generate Insights • Optimize Operations • Reduce Accidents The future of airside operations
4. Assaia ApronAI
5. Basic Real-time Inference Video Stream Step 1 frames Step 2 DB View some data final data
6. Advanced Real-time Inference Video Stream Step 1 frames Step 2 DB View some data final data Side Step
7. Advanced Real-time Inference Video Stream Step 1 frames Step 2 DB View some data final data Side Step 1 Side Step N
8. Message Bus Application 1 Application N Application 2
9. Real-time Inference Video Stream Step 1 Step 2 DB View any data Side Step final data frames some data frames frames some data final data ?
10. Message Brokers • RabbitMQ ◦ Erlang ◦ AMQP • Apache Kafka ◦ Java ◦ Large memory footprint • NSQ ◦ Go ◦ Lightweight ◦ Easy to maintain ◦ Worth a try ☺
11. Real-time Inference Video Stream Step 1 Step 2 DB View any data Side Step final data frames some data frames frames some data final data NSQ
12. NSQ routing Application A topic A :: channel B NSQ Application B topic A Application C1 Application C2 topic A :: channel C
13. Serialization • JSON ◦ Text • BSON ◦ MongoDB oriented ◦ No RFC • Pickle ◦ Python specific • Protobuf ◦ Code generation ◦ Complex external systems interoperability • CBOR ◦ RFC ◦ Schemaless ◦ Has Schema (CDDL) RFC ◦ Worth a try ☺
14. NSQ Adoption • Simple binary message payload • Need to choose serialization • Maintain schema • Manage topics/channels structure • Manage channel overflow
15. “Talk is cheap. Show me the code.” — Linus Torvalds
16. The Journey with NSQ Oleg Ivashov
17. Journey plan • Choose a Python library • Support CBOR serialization • A way to maintain the scheme • We need a better buffer for NSQ
18. Journey plan • Choose a Python library • Support CBOR serialization • A way to maintain the scheme • We need a better buffer for NSQ
19. Choose a Python library
20. Choose a Python library • PyNSQ ◦ Official ◦ Tornado • Gnsq ◦ Gevent • Asycnsq ◦ asyncio • Ansq ◦ asyncio too
21. Choose a Python library: ansq vs asyncnsq = ansq
22. Choose a Python library: ansq github.com/list-family/ansq
23. ansq: how it works nsq = await open_connection() await nsq.pub(topic='books', message='War and Peace') await nsq.subscribe(topic='books', channel='student') async for message in nsq.messages(): print(f"Book updated: {message}") await message.fin()
24. Journey plan • Choose a Python library • Support CBOR serialization • A way to maintain the scheme • We need a better buffer for NSQ
25. Support CBOR serialization: cbor vs cbor2
26. Support CBOR serialization: cbor2 how-to >>> import cbor2 >>> data = cbor2.dumps(['py', 'con']) >>> data b'\x82bpyccon' >>> cbor2.loads(data) ['py', 'con']
27. Journey plan • Choose a Python library • Support CBOR serialization • A way to maintain the scheme • We need a better buffer for NSQ
28. A way to maintain the scheme • Around 20 different schemes • 11 different applications from video stream to view • Data types • Soft scheme upgrade
29. A way to maintain the scheme: attrs github.com/python-attrs/attrs
30. A way to maintain the scheme: attrs >>> import attr >>> @attr.s(auto_attribs=True) ... class Book: ... title: str = attr.ib( ... converter=str.capitalize, ... validator=is_str, ... default='' ... )
31. A way to maintain the scheme: attrs >>> book = Book(title='War and Peace') >>> payload = attr.asdict(book) >>> payload {'title': 'War and peace'} >>> Book(**payload) Book(title='War and peace')
32. A way to maintain the scheme: nested objects https://www.youtube.com/channel/UCTU_DiNpWGlpffdB1mvTBmw
33. A way to maintain the scheme: nested objects >>> class Box(BaseSchema): ... x0: float; y0: float; ... ... >>> class Prediction(BaseSchema): ... box: Box ... value: float ...
34. A way to maintain the scheme: nested objects https://www.youtube.com/channel/UCTU_DiNpWGlpffdB1mvTBmw
35. A way to maintain the scheme: nested objects >>> object = Prediction(bbox=BBox(value=1)) >>> payload = attr.asdict(object) >>> payload {'bbox': {'value': 1}} >>> Prediction(**payload) Prediction(bbox={'value': 1})
36. A way to maintain the scheme: cattrs >>> import cattr >>> payload = cattr.unstructure(message) >>> payload {'bbox': {'value': 1}} >>> cattr.structure(payload, Prediction) Prediction(bbox=BBox(value=1))
37. Journey plan • Choose a Python library • Support CBOR serialization • A way to maintain the scheme • We need a better buffer for NSQ
38. We need a better buffer for NSQ
39. We need a better buffer for NSQ
40. demo.assaia.com
41. We need a better buffer for NSQ
42. We need a better buffer for NSQ
43. We need a better buffer for NSQ
44. We need a better buffer for NSQ
45. We need a better buffer for NSQ
46. NSQ can’t deal with backpressure • “Backpressure explained — the resisted flow of data through software” Jay Phelps, VP of Engineering @ Outsmartly https://medium.com/@jayphelps/backpressure-explained-the-flow-of-data- through-software-2350b3e77ce7 • “I'm not feeling the async pressure” Armin Ronacher, Director Of Engineering @ Sentry https://lucumr.pocoo.org/2020/1/1/async-pressure/
47. NSQ can’t deal with backpressure: solutions • Ephemeral channels • Application-side buffer
48. NSQ and backpressure: ephemeral channels await nsq.subscribe( topic='book', channel='student#ephemeral' ) async for message in nsq.messages(): await message.fin() # Channel is empty await nsq.subscribe( topic='book', channel='student#ephemeral' )
49. NSQ and backpressure: application-side buffer class Buffer(collections.deque): ... async def write_buffer(): async for message in reader: buffer.put_nowait(message) loop.create_task(write_buffer())
50. NSQ in Python • NSQ just works • ansq library • CBOR for serialization • attrs + cattrs for schema • … some customization may be required
51. NSQ в Python Serge Matveenko fosstodon.org/@lig Oleg Ivashov linkedin.com/in/ivashov Serge Matveenko Oleg Ivashov Assaia assaia.com Questions?