Zaqar Messaging for Microservices and IoT

OpenStack Summit Austin 2016

Transcript

1. ZAQAR MESSAGING FOR MICROSERVICES AND IoT Ryan Brown Victoria Martinez de la Cruz April 27th, 2016
2. WHAT IS ZAQAR?
3. 3 TL;DR Messaging and Notifications for OpenStack To produce an OpenStack messaging service that affords a variety of distributed application patterns in an efficient, scalable and highly-available manner, and to create and maintain associated Python libraries and documentation. Mission statement
4. 4 Main goal ...and in OpenStack itself!* It aims to connect applications running on the cloud... *for certain scenarios… don’t get too excited
5. 5 What Zaqar is not - It’s not a replacement RabbitMQ or QPID - It’s not a queue service - It’s not an email service
6. 6 Multitenant Main features Component based Messaging patterns
7. 7 Notifications Main features At-least-once delivery guarantee Horizontally scalable
8. 8 Messaging for the cloud Self-healing applications Intercloud communication Data processing Some use cases
9. 9 Zaqar architecture Client 1 Client 2 Client 3 Client N Transport Protocol API Storage
10. 10 Storage backends - Redis - Ephemeral by default - High-throughput - In-memory - MongoDB - HA - Persistent - Disk-backed
11. 11 Storage MongoDB SQLAlchemy Redis Transport WSGI Juno Kilo Zaqar architecture Storage backends and transmission mechanisms supported over time Liberty Storage MongoDB SQLAlchemy (mgmt only) Redis Transport WSGI Websocket Msgpack Mitaka Storage MongoDB SQLAlchemy (mgmt only) Redis Transport WSGI Websocket (beta) Storage MongoDB SQLAlchemy (mgmt only) Redis Transport WSGI Websocket
12. ZAQAR’S FUTURE
13. 13 Notifications Within OpenStack - Polling - Between services - To users - Not a replacement for oslo.messaging - Standardize on message format - oslo.versionedobjects
14. 14 Dead letter queue - A place for failed or undeliverable messages - Planned for Newton
15. 15 Subscription Confirmation - Confirm with webhook recipient that subscription is valid
16. MESSAGES LIFECYCLE
17. 17 Work Queue Sender - Send job - Listen on finished queue Worker - Listen on job queue - Receive job Sender Zaqar Worker {“job”: 123} {“job”: 123}
18. 18 Work Queue Sender - Send job - Listen on finished queue Worker - Listen on job queue - Receive job - Claim message Sender Zaqar Worker Claim job 123
19. 19 Work Queue Sender - Send job - Listen on finished queue Worker - Listen on job queue - Receive job - Claim message - Delete message Sender Zaqar Worker Delete job 123
20. 20 Expiration and Lifetimes - Time-To-Live settings - Subscriptions - Claims - Messages - Stale data can be worse than no data
21. MESSAGE PATTERNS
22. 22 Client Notifications Websockets, email, and webhooks - Web browser notifications - Horizon! - Email for people and legacy systems - Webhooks for cross-service collaboration
23. 23 Messaging for Services Shares lots with work queueing - Push or pull - Long-lived messages mean systems can go offline without losing commands
24. 24 Stack CREATE Instance CREATE Heat Notifications Integrate Heat with more workflows Actions Register w/ load balancer Add to inventory system Register metric metadata Zaqar event Listener Can have many listeners
25. 25 Using Heat Notifications In environment.yml event_sinks: - type: zaqar-queue target: myqueue ttl: 1200
26. 26 Stack CREATE Create some stuff Heat Hooks Wait before/after resource actions Actions Manual verification Legacy systems Hook event Finish extra work Wait for external action Continue CREATE
27. ZAQAR FOR IoT
28. 28 Signed URLs Narrow, specific credentials - Per action, per queue - Write-only or read-only - TTLs too!
29. 29 Sensor data Many, many writers - Short TTL - High volume - Missed messages are fine
30. 30 Example IoT Application Climate control on campus - Multiple buildings - Control systems in every office - Company directive to “go green”
31. 31 Mobile app Zaqar Server Climate Control Thermostat Web interface Websocket connection HTTP API connection Commands Air Conditioner 1 Send metrics Receive commands Air Conditioner 2
32. 32 Mobile app Zaqar Server Changes Thermostat Web interface Set temp to 22C Air Conditioner 1 Air Conditioner 2 Notifications
33. 33 Mobile app Zaqar Server Changes Thermostat Web interface Air Conditioner 1 Air Conditioner 2 23.4C 24C
34. Q&A @ryan_sb @vkmc