Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ITT 2018 - Dejan Bosanac - Scalable Cloud Messaging

ITT 2018 - Dejan Bosanac - Scalable Cloud Messaging

For successful implementation of distributed systems, flexible and scalable messaging layer is one of the most important components. Setting a static messaging infrastructure and provisioning it manually doesn’t fit well with the cloud-native development model most organisations are adopting lately.

enmasse.io provides an open source solution for deploying your own messaging infrastructure in the cloud. It’s based on proven standards and technologies, like AMQP 1.0 and Kubernetes. Additionally, the platform provides a wide range of features expected of the cloud ready messaging solution. Features range from scalability and elasticity to multi-tenancy, simple management and monitoring.

This session will cover the EnMasse project in details. It will start by covering messaging concepts and how they fit the cloud environment. Then we’ll continue with the platform architecture and see how various features are implemented. Finally, we’ll discuss deployment and monitoring options necessary to run a platform like this in the production.

990b89ca5f918a94ef6523d399eda9a4?s=128

Istanbul Tech Talks

April 17, 2018
Tweet

Transcript

  1. Messaging for the cloud Building a scalable messaging service on

    Kubernetes and OpenShift Dejan Bosanac Senior Software Engineer, Red Hat @dejanb 1
  2. 2 Outline •  What is messaging? •  How to scale

    messaging in general •  EnMasse messaging cloud platform
  3. 3 •  Sending messages ◦  Internally in distributed systems ◦ 

    Externally between systems •  Communication at the application level •  Messages go from sender/producer to receiver/consumer ◦  Asynchronously ◦  Time decoupling What is messaging?
  4. 4 Basic features Request-response Publish-subscribe Competing-consumers

  5. 5 Basic features At most once At least once Exactly

    once X X
  6. 6 •  Multi Protocol Broker ◦  AMQP 1.0, MQTT, STOMP,

    OpenWire, Artemis Core ◦  JMS 2.0 (API) •  https://activemq.apache.org/artemis/ •  Started as HornetQ JBoss project in 2009 •  In 2014 donated to Apache ActiveMQ (Sub project ActiveMQ Artemis) ActiveMQ Artemis
  7. 7 Brokered messaging Store and forward Producer Broker Consumer Send

    message Accepted Send message Accepted
  8. 8 •  Great performance due to ◦  Reactive Architecture ◦ 

    Efficient message journal •  High-availability: ◦  Shared storage ◦  Replicated journal ActiveMQ Artemis
  9. 9 •  International Standard (ISO/IEC ISO 19464) •  Binary Protocol

    •  Rich feature set: ◦  conversation multiplexing ◦  advanced flow control ◦  Type system ◦  QoS Guarantees •  Symmetrical message exchange ◦  No Broker required AMQP 1.0
  10. 10 Direct Messaging Router Producer Router Consumer Send message Accepted

    Send message Accepted
  11. 11 •  Lightweight AMQP 1.0 message router written in C

    •  http://qpid.apache.org/components/dispatch-router/ •  Provides flexible and scalable interconnect between AMQP endpoints •  Redundant paths •  Supports waypoints Qpid Dispatch Router
  12. 12 •  Store and Forward ◦  Queue ◦  Topic • 

    Direct ◦  Anycast ◦  Multicast (Broadcast) Addressing semantics
  13. 13 Basic idea R R R P C B B

  14. 14 Messaging in the cloud

  15. 16 •  Open source cloud messaging running on Kubernetes and

    OpenShift •  enmasse.io •  github.com/enmasseproject/enmasse EnMasse Messaging-as-a-Service
  16. 17 •  Support for store-and-forward and direct messaging mechanisms • 

    Multiple communication patterns: request-response, publish-subscribe and competing consumers •  Scale and elasticity of message brokers •  AMQP 1.0 and MQTT support •  Simple setup, management and monitoring •  Multitenancy: manage multiple independent instances •  Deploy on-premise or in the cloud EnMasse Features
  17. 18 Overview

  18. OpenShift console 19

  19. 20 Address Space { "apiVersion": "v1/enmasse", "kind": "AddressSpace", "metadata": {

    "name": "myspace" }, "spec": { "type": "standard", "plan": "unlimited-standard" } }
  20. 21 Address Space Plan "addressPlans": [ "small-queue", "small-anycast" ],"resources": [

    { "name": "router", "min": "0.0", "max": "2.0" }, { "name": "broker", "min": "0.0", "max": "3.0" }, { "name": "aggregate", "min": "0.0", "max": "5.0" } ]
  21. 22 Address Plan "addressType": "queue", "metadata": { "name": "small-queue" },

    "requiredResources": [ { "name": "router", "credit": 0.2 }, { "name": "broker", "credit": 0.3 } ]
  22. 23 Address { "apiVersion": "enmasse.io/v1", "kind": "AddressList", "items": [ {

    "spec": { "address": "myqueue", "type": "queue", "plan": "small-queue" } } ] }
  23. Messaging console 24

  24. 25 Monitoring

  25. MQTT support 26 •  MQTT gateway ◦  Handles connections with

    remote MQTT clients ◦  Bridges MQTT - AMQP protocols •  MQTT lwt ◦  Provides the “will testament” feature ◦  In charge to recover & send the “will” if client dies •  Doesn’t support ◦  QoS2 ◦  Retained messages
  26. 27 Architecture

  27. 28 Brokered space

  28. Connecting 29 •  Use TLS with SNI on port 443

    Clients •  Qpid - http://qpid.apache.org/ ◦  JMS 2.0 client ◦  Python, C, C++, Ruby •  JavaScript RHEA client - https://github.com/amqp/rhea •  Vert.x Proton - https://github.com/vert-x3/vertx-proton
  29. Resources 30 •  EnMasse - http://enmasse.io •  ActiveMQ Artemis -

    https://activemq.apache.org/artemis/ •  Qpid Dispatch Router - http://qpid.apache.org/components/dispatch-router/
  30. Thank you ! Questions ? @dejanb