Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introducing Openstack Marconi

Introducing Openstack Marconi

This covers the Openstack Marconi service: Queuing as a Service. Answers the how, what, and why of Marconi, dives into the API and the architecture, and discusses the future of the project.

Alejandro Cabrera

July 17, 2013
Tweet

More Decks by Alejandro Cabrera

Other Decks in Programming

Transcript

  1. Message Queues as a Service: Introducing Marconi Alejandro Cabrera ([email protected])

    July 17, 2013 Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  2. from pprint import pprint if __name__ == ’__main__’: meta =

    { ’author’ : ’Alejandro Cabrera’, ’email’ : ’[email protected]’, ’objective’ : ’Introduce Marconi!’ } pprint(meta) Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  3. About Me Software developer @ Rackspace for almost a year

    Likes: Python, Linux, open-source, automation, quality, cordiality Dislikes: Proprietary, manual processes, hostility Github: cabrera Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  4. Overview Marconi: What, Why, How? Marconi and You: Configuring Alejandro

    Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  5. Overview Marconi: What, Why, How? Marconi and You: Configuring Growing

    Marconi: Development and the Future Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  6. Marconi: Queuing for the Cloud Marconi is: Queuing as a

    service: queues, messages, claims Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  7. Marconi: Queuing for the Cloud Marconi is: Queuing as a

    service: queues, messages, claims An HTTP-driven, mostly-RESTful API Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  8. Marconi: Queuing for the Cloud Marconi is: Queuing as a

    service: queues, messages, claims An HTTP-driven, mostly-RESTful API An Openstack hopeful Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  9. Marconi: Queuing for the Cloud Marconi is: Queuing as a

    service: queues, messages, claims An HTTP-driven, mostly-RESTful API An Openstack hopeful Open-source! Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  10. Marconi: Queuing for the Cloud Marconi is: Queuing as a

    service: queues, messages, claims An HTTP-driven, mostly-RESTful API An Openstack hopeful Open-source! Marconi is not: Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  11. Marconi: Queuing for the Cloud Marconi is: Queuing as a

    service: queues, messages, claims An HTTP-driven, mostly-RESTful API An Openstack hopeful Open-source! Marconi is not: A replacement for X (celery, zmq, rabbitmq) Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  12. Marconi: Queuing for the Cloud Marconi is: Queuing as a

    service: queues, messages, claims An HTTP-driven, mostly-RESTful API An Openstack hopeful Open-source! Marconi is not: A replacement for X (celery, zmq, rabbitmq) A message broker Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  13. Marconi: What’s In a Name? Guglielmo Marconi An Italian inventor

    often credited with the invention of radio Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  14. Marconi: What’s In a Name? Guglielmo Marconi An Italian inventor

    often credited with the invention of radio Marconi (the service) is all about managing messages Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  15. Why Marconi? There was a need for a messsaging platform

    that is: Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  16. Why Marconi? There was a need for a messsaging platform

    that is: Open Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  17. Why Marconi? There was a need for a messsaging platform

    that is: Open Easy to use Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  18. Why Marconi? There was a need for a messsaging platform

    that is: Open Easy to use Scalable Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  19. Why Marconi? There was a need for a messsaging platform

    that is: Open Easy to use Scalable Extensible Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  20. Why Marconi? There was a need for a messsaging platform

    that is: Open Easy to use Scalable Extensible Marconi aims to fill those needs Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  21. Why Marconi Instead of. . . ? Amazon SQS? Alejandro

    Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  22. Why Marconi Instead of. . . ? Amazon SQS? SQS

    is closed - you can’t run your own instance Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  23. Why Marconi Instead of. . . ? Amazon SQS? SQS

    is closed - you can’t run your own instance Celery? Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  24. Why Marconi Instead of. . . ? Amazon SQS? SQS

    is closed - you can’t run your own instance Celery? Celery is a task-based service Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  25. Why Marconi Instead of. . . ? Amazon SQS? SQS

    is closed - you can’t run your own instance Celery? Celery is a task-based service It depends on a broker to handle the queuing aspects Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  26. Why Marconi Instead of. . . ? Amazon SQS? SQS

    is closed - you can’t run your own instance Celery? Celery is a task-based service It depends on a broker to handle the queuing aspects RabbitMQ, qpid, ZMQ? Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  27. Why Marconi Instead of. . . ? Amazon SQS? SQS

    is closed - you can’t run your own instance Celery? Celery is a task-based service It depends on a broker to handle the queuing aspects RabbitMQ, qpid, ZMQ? You have to scale them yourself Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  28. How Does it Work? Queues: create them! Messages: post them

    to queues (FIFO guaranteed) Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  29. How Does it Work? Queues: create them! Messages: post them

    to queues (FIFO guaranteed) Claims: workers claim messages Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  30. How Does it Work? Queues: create them! Messages: post them

    to queues (FIFO guaranteed) Claims: workers claim messages That’s all there is to it. Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  31. That’s All? Yes. It’s up to you to build higher-level

    patterns: Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  32. That’s All? Yes. It’s up to you to build higher-level

    patterns: Response/request Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  33. That’s All? Yes. It’s up to you to build higher-level

    patterns: Response/request Publish/subscribe Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  34. That’s All? Yes. It’s up to you to build higher-level

    patterns: Response/request Publish/subscribe Master/slave Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  35. That’s All? Yes. It’s up to you to build higher-level

    patterns: Response/request Publish/subscribe Master/slave Producer(s)/Consumer(s) Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  36. Marconi: Configuration Marconi doesn’t try to limit you You can

    configure. . . : Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  37. Marconi: Configuration Marconi doesn’t try to limit you You can

    configure. . . : Logging details Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  38. Marconi: Configuration Marconi doesn’t try to limit you You can

    configure. . . : Logging details Transport drivers: wsgi, zmq Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  39. Marconi: Configuration Marconi doesn’t try to limit you You can

    configure. . . : Logging details Transport drivers: wsgi, zmq As well as transport-specific configuration! Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  40. Marconi: Configuration Marconi doesn’t try to limit you You can

    configure. . . : Logging details Transport drivers: wsgi, zmq As well as transport-specific configuration! Storage drivers Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  41. Marconi: Configuration Marconi doesn’t try to limit you You can

    configure. . . : Logging details Transport drivers: wsgi, zmq As well as transport-specific configuration! Storage drivers Messages: size (default: 256KB), batch size Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  42. Marconi: Configuration Marconi doesn’t try to limit you You can

    configure. . . : Logging details Transport drivers: wsgi, zmq As well as transport-specific configuration! Storage drivers Messages: size (default: 256KB), batch size Queues: name length Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  43. Marconi: Configuration Marconi doesn’t try to limit you You can

    configure. . . : Logging details Transport drivers: wsgi, zmq As well as transport-specific configuration! Storage drivers Messages: size (default: 256KB), batch size Queues: name length Philosophy: configure all the things! Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  44. Marconi: Can I Use it Now? Yes, with a caveat

    Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  45. Marconi: Can I Use it Now? Yes, with a caveat

    It’s pre-alpha status Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  46. Marconi: Can I Use it Now? Yes, with a caveat

    It’s pre-alpha status (Don’t run this in production!) Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  47. Marconi API: From the Top GET /v1 GET /v1/health Alejandro

    Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  48. Marconi API: Handling Queues PUT /v1/queues/{queue} GET /v1/queues{?marker,limit,detailed} PUT /v1/queues/{queue}

    GET /v1/queues/{queue} GET /v1/queues/{queue}/stats DELETE /v1/queues/{queue} Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  49. Marconi API: Messages! GET /v1/queues/{queue}/messages{?marker,limit,echo} GET /v1/queues/{queue}/messages/{message_id}{?claim_id} GET /v1/queues/{queue}/messages{?ids,claim_id} POST

    /v1/queues/{queue}/messages DELETE /v1/queues/{queue}/messages/{message_id}{?claim_id} Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  50. Marconi API: Claims Processing POST /v1/queues/{queue}/claims{?limit} GET /v1/queues/{queue}/claims/{claim_id} PATCH /v1/queues/{queue}/claims/{claim_id}

    DELETE /v1/queues/{queue}/claims/{claim_id} Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  51. Going Deeper: Internal Architecture Highly modular - driver-driven architecture Storage:

    sqlite, mongodb Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  52. Going Deeper: Internal Architecture Highly modular - driver-driven architecture Storage:

    sqlite, mongodb Transport: wsgi (zmq in the works) Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  53. Going Deeper: Internal Architecture Highly modular - driver-driven architecture Storage:

    sqlite, mongodb Transport: wsgi (zmq in the works) Authentication: keystone, nova, endpoint (client-side) Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  54. Going Deeper: Internal Architecture Highly modular - driver-driven architecture Storage:

    sqlite, mongodb Transport: wsgi (zmq in the works) Authentication: keystone, nova, endpoint (client-side) The idea is to be highly configurable Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  55. Going Deeper: Internal Architecture Highly modular - driver-driven architecture Storage:

    sqlite, mongodb Transport: wsgi (zmq in the works) Authentication: keystone, nova, endpoint (client-side) The idea is to be highly configurable You know what you need - you make the trade-offs Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  56. A Look at the Structure: Big Picture marconi |-- bootstrap.py

    |-- cmd # garbabe collector, standalone server process |-- common # shared utilities |-- __init__.py |-- openstack # openstack common code |-- storage # storage drivers |-- tests # test helpers |-- transport # transport drivers ‘-- version.py Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  57. A Look at the Structure: Storage storage/ |-- base.py |--

    exceptions.py |-- __init__.py |-- mongodb ‘-- sqlite Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  58. A Look at the Structure: Storage (MongoDB) storage/ |-- base.py

    |-- exceptions.py |-- __init__.py |-- mongodb | |-- claims.py | |-- controllers.py | |-- driver.py | |-- __init__.py | |-- messages.py | |-- options.py | |-- queues.py | ‘-- utils.py ‘-- sqlite Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  59. A Look at the Structure: Storage (sqlite) storage/ |-- base.py

    |-- exceptions.py |-- __init__.py |-- mongodb ‘-- sqlite # development driver |-- claims.py |-- controllers.py |-- driver.py |-- __init__.py |-- messages.py |-- queues.py ‘-- utils.py Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  60. A Look at the Structure: Transports transport/ |-- auth.py |--

    base.py |-- helpers.py |-- __init__.py |-- wsgi ‘-- zmq Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  61. A Look at the Structure: Transports (wsgi) transport/ |-- auth.py

    |-- base.py |-- helpers.py |-- __init__.py |-- wsgi | |-- app.py | |-- claims.py | |-- driver.py | |-- exceptions.py | |-- health.py | |-- helpers.py | |-- __init__.py | |-- messages.py | |-- queues.py | ‘-- stats.py ‘-- zmq Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  62. A Look at the Structure: Transports (zmq) transport/ |-- auth.py

    |-- base.py |-- helpers.py |-- __init__.py |-- wsgi ‘-- zmq ‘-- __init__.py Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  63. A Look at the Structure: Transports (zmq) transport/ |-- auth.py

    |-- base.py |-- helpers.py |-- __init__.py |-- wsgi ‘-- zmq ‘-- __init__.py # you can help make this happen! Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  64. Going Client-Side: Introducing python-marconiclient! You have a server, but now

    you need to work with it Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  65. Going Client-Side: Introducing python-marconiclient! You have a server, but now

    you need to work with it python-marconiclient is. . . : Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  66. Going Client-Side: Introducing python-marconiclient! You have a server, but now

    you need to work with it python-marconiclient is. . . : A Python API for communicating with a marconi server Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  67. Going Client-Side: Introducing python-marconiclient! You have a server, but now

    you need to work with it python-marconiclient is. . . : A Python API for communicating with a marconi server A CLI that accomplishes the same Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  68. Going Client-Side: Introducing python-marconiclient! You have a server, but now

    you need to work with it python-marconiclient is. . . : A Python API for communicating with a marconi server A CLI that accomplishes the same A reference implementation for alternate marconi clients Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  69. So. . . What Does it Look Like? >>> client

    = Client(conn, async=False) >>> client.queues(marker=..., limit=10, detailed=False) <generator object <genexpr> ar 0x7fd3ef1ed730> >>> client.create_queue(name=’darn_good_queue’, metadata=json.dumps({’game’: ’d2’ } ) ) <MarconiQueue [wot]> >>> client.home <HomeDoc ...> # affects all operations for objects # acquired from the client >>> client.async False Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  70. Client: Handling Queues >>> queue = next(q for q in

    client.queues if q.name == ’darn_good_queue’) >>> queue.name u’darn_good_queue’ >>> queue.stats <QueueStats ...> >>> queue.get_messages() <generator object <genexpr> ar 0x7fd3ef1ed742> >>> queue.get_messages(ids=[’50b68a50d6f5b8c8a7c62b’, ’50b68a50d6f5b8c8a7c62b’ ], claim=’a28ee94e-6cb4-11e2-b4’, limit=1) <generator object <genexpr> ar 0x7fd3ef1ed742> >>> queue.post_messages(messages=...) >>> queue.delete() Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  71. Client: Handling Queues (Metadata) >>> queue.metadata {’a’: 1, ’b’: 2}

    >>> queue.set_metadata({’a’: 2, ...}) Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  72. Client: Message Handling >>> message = next(queue.messages(...)) >>> message.age 90

    >>> message.ttl 120 >>> message.id u’9y22h21w2h’ >>> message.body {u’action’: u’win’} >>> message.delete() >>> message.status <EnumValue: Message.Free [value=1]> Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  73. Client: Claims >>> queue.claim(limit=10) <generator object <genexpr> ar 0x7fd3ef1ed742> >>>

    claim = queue.claim(limit=10) # identical to queue message fetch API, # with claim id pre-filled >>> claim.messages(...) <generator object <genexpr> ar 0x7fd3ef1ed742> >>> msg = next(claim.messages(...)) >>> msg.status <EnumValue: Message.Claimed [value=2]> Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  74. Client: Claims >>> claim.id ... >>> claim.ttl 90 >>> claim.age

    120 >>> claim.patch(ttl=..., grace=...) >>> claim.release() Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  75. Marconi & Openstack & You Currently: pre-incubation We have the

    form, we have contributors (5-6) Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  76. Marconi & Openstack & You Currently: pre-incubation We have the

    form, we have contributors (5-6) (We need more conributors!) Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  77. Marconi & Openstack & You Currently: pre-incubation We have the

    form, we have contributors (5-6) (We need more conributors!) We keep our code: Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  78. Marconi & Openstack & You Currently: pre-incubation We have the

    form, we have contributors (5-6) (We need more conributors!) We keep our code: Here: server Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  79. Marconi & Openstack & You Currently: pre-incubation We have the

    form, we have contributors (5-6) (We need more conributors!) We keep our code: Here: server And here: client Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  80. Marconi & Openstack & You Currently: pre-incubation We have the

    form, we have contributors (5-6) (We need more conributors!) We keep our code: Here: server And here: client Join us on IRC: #openstack-marconi @ freenode Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  81. The Future Get incubated into Openstack Release the reference client

    Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  82. The Future Get incubated into Openstack Release the reference client

    Implement more: Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  83. The Future Get incubated into Openstack Release the reference client

    Implement more: Transport drivers Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  84. The Future Get incubated into Openstack Release the reference client

    Implement more: Transport drivers Storage drivers Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi
  85. The Future Get incubated into Openstack Release the reference client

    Implement more: Transport drivers Storage drivers Grow! Alejandro Cabrera ([email protected]) Message Queues as a Service: Introducing Marconi