Tweet
Tweet

Slide 1

Slide 1 text

NSQ в Python Serge Matveenko Oleg Ivashov assaia.com

Slide 2

Slide 2 text

The Road to NSQ Serge Matveenko

Slide 3

Slide 3 text

Assaia ApronAI ● Real-time ● Computer Vision ● Generate Insights ● Optimize Operations ● Reduce Accidents The future of airside operations

Slide 4

Slide 4 text

Assaia ApronAI

Slide 5

Slide 5 text

Basic Real-time Inference Video Stream Step 1 frames Step 2 DB View some data final data

Slide 6

Slide 6 text

Advanced Real-time Inference Video Stream Step 1 frames Step 2 DB View some data final data Side Step

Slide 7

Slide 7 text

Advanced Real-time Inference Video Stream Step 1 frames Step 2 DB View some data final data Side Step 1 Side Step N

Slide 8

Slide 8 text

Message Bus Application 1 Application N Application 2

Slide 9

Slide 9 text

Real-time Inference Video Stream Step 1 Step 2 DB View any data Side Step final data frames some data frames frames some data final data ?

Slide 10

Slide 10 text

Message Brokers ● RabbitMQ ○ Erlang ○ AMQP ● Apache Kafka ○ Java ○ Large memory footprint ● NSQ ○ Go ○ Lightweight ○ Easy to maintain ○ Worth a try ☺

Slide 11

Slide 11 text

Real-time Inference Video Stream Step 1 Step 2 DB View any data Side Step final data frames some data frames frames some data final data NSQ

Slide 12

Slide 12 text

NSQ routing Application A topic A :: channel B NSQ Application B topic A Application C1 Application C2 topic A :: channel C

Slide 13

Slide 13 text

Serialization ● JSON ○ Text ● BSON ○ MongoDB oriented ○ No RFC ● Pickle ○ Python specific ● Protobuf ○ Code generation ○ Complex external systems interoperability ● CBOR ○ RFC ○ Schemaless ○ Has Schema (CDDL) RFC ○ Worth a try ☺

Slide 14

Slide 14 text

NSQ Adoption ● Simple binary message payload ● Need to choose serialization ● Maintain schema ● Manage topics/channels structure ● Manage channel overflow

Slide 15

Slide 15 text

“Talk is cheap. Show me the code.” — Linus Torvalds

Slide 16

Slide 16 text

The Journey with NSQ Oleg Ivashov

Slide 17

Slide 17 text

Journey plan ● Choose a Python library ● Support CBOR serialization ● A way to maintain the scheme ● We need a better buffer for NSQ

Slide 18

Slide 18 text

Journey plan ● Choose a Python library ● Support CBOR serialization ● A way to maintain the scheme ● We need a better buffer for NSQ

Slide 19

Slide 19 text

Choose a Python library

Slide 20

Slide 20 text

Choose a Python library ● PyNSQ ○ Official ○ Tornado ● Gnsq ○ Gevent ● Asycnsq ○ asyncio ● Ansq ○ asyncio too

Slide 21

Slide 21 text

Choose a Python library: ansq vs asyncnsq = ansq

Slide 22

Slide 22 text

Choose a Python library: ansq github.com/list-family/ansq

Slide 23

Slide 23 text

ansq: how it works nsq = await open_connection() await nsq.pub(topic='books', message='War and Peace') await nsq.subscribe(topic='books', channel='student') async for message in nsq.messages(): print(f"Book updated: {message}") await message.fin()

Slide 24

Slide 24 text

Journey plan ● Choose a Python library ● Support CBOR serialization ● A way to maintain the scheme ● We need a better buffer for NSQ

Slide 25

Slide 25 text

Support CBOR serialization: cbor vs cbor2

Slide 26

Slide 26 text

Support CBOR serialization: cbor2 how-to >>> import cbor2 >>> data = cbor2.dumps(['py', 'con']) >>> data b'\x82bpyccon' >>> cbor2.loads(data) ['py', 'con']

Slide 27

Slide 27 text

Journey plan ● Choose a Python library ● Support CBOR serialization ● A way to maintain the scheme ● We need a better buffer for NSQ

Slide 28

Slide 28 text

A way to maintain the scheme ● Around 20 different schemes ● 11 different applications from video stream to view ● Data types ● Soft scheme upgrade

Slide 29

Slide 29 text

A way to maintain the scheme: attrs github.com/python-attrs/attrs

Slide 30

Slide 30 text

A way to maintain the scheme: attrs >>> import attr >>> @attr.s(auto_attribs=True) ... class Book: ... title: str = attr.ib( ... converter=str.capitalize, ... validator=is_str, ... default='' ... )

Slide 31

Slide 31 text

A way to maintain the scheme: attrs >>> book = Book(title='War and Peace') >>> payload = attr.asdict(book) >>> payload {'title': 'War and peace'} >>> Book(**payload) Book(title='War and peace')

Slide 32

Slide 32 text

A way to maintain the scheme: nested objects https://www.youtube.com/channel/UCTU_DiNpWGlpffdB1mvTBmw

Slide 33

Slide 33 text

A way to maintain the scheme: nested objects >>> class Box(BaseSchema): ... x0: float; y0: float; ... ... >>> class Prediction(BaseSchema): ... box: Box ... value: float ...

Slide 34

Slide 34 text

A way to maintain the scheme: nested objects https://www.youtube.com/channel/UCTU_DiNpWGlpffdB1mvTBmw

Slide 35

Slide 35 text

A way to maintain the scheme: nested objects >>> object = Prediction(bbox=BBox(value=1)) >>> payload = attr.asdict(object) >>> payload {'bbox': {'value': 1}} >>> Prediction(**payload) Prediction(bbox={'value': 1})

Slide 36

Slide 36 text

A way to maintain the scheme: cattrs >>> import cattr >>> payload = cattr.unstructure(message) >>> payload {'bbox': {'value': 1}} >>> cattr.structure(payload, Prediction) Prediction(bbox=BBox(value=1))

Slide 37

Slide 37 text

Journey plan ● Choose a Python library ● Support CBOR serialization ● A way to maintain the scheme ● We need a better buffer for NSQ

Slide 38

Slide 38 text

We need a better buffer for NSQ

Slide 39

Slide 39 text

We need a better buffer for NSQ

Slide 40

Slide 40 text

demo.assaia.com

Slide 41

Slide 41 text

We need a better buffer for NSQ

Slide 42

Slide 42 text

We need a better buffer for NSQ

Slide 43

Slide 43 text

We need a better buffer for NSQ

Slide 44

Slide 44 text

We need a better buffer for NSQ

Slide 45

Slide 45 text

We need a better buffer for NSQ

Slide 46

Slide 46 text

NSQ can’t deal with backpressure ● “Backpressure explained — the resisted flow of data through software” Jay Phelps, VP of Engineering @ Outsmartly https://medium.com/@jayphelps/backpressure-explained-the-flow-of-data- through-software-2350b3e77ce7 ● “I'm not feeling the async pressure” Armin Ronacher, Director Of Engineering @ Sentry https://lucumr.pocoo.org/2020/1/1/async-pressure/

Slide 47

Slide 47 text

NSQ can’t deal with backpressure: solutions ● Ephemeral channels ● Application-side buffer

Slide 48

Slide 48 text

NSQ and backpressure: ephemeral channels await nsq.subscribe( topic='book', channel='student#ephemeral' ) async for message in nsq.messages(): await message.fin() # Channel is empty await nsq.subscribe( topic='book', channel='student#ephemeral' )

Slide 49

Slide 49 text

NSQ and backpressure: application-side buffer class Buffer(collections.deque): ... async def write_buffer(): async for message in reader: buffer.put_nowait(message) loop.create_task(write_buffer())

Slide 50

Slide 50 text

NSQ in Python ● NSQ just works ● ansq library ● CBOR for serialization ● attrs + cattrs for schema ● … some customization may be required

Slide 51

Slide 51 text

NSQ в Python Serge Matveenko fosstodon.org/@lig Oleg Ivashov linkedin.com/in/ivashov Serge Matveenko Oleg Ivashov Assaia assaia.com Questions?