Architecture

The underlying messaging framework of the WSO2 Message Broker is powered by Andes, one of the distributed message brokering systems compatible with the leading Advanced Message Queuing Protocol (AMQP)(0-91)). In addition, WSO2 Message Broker is also compatible with the Message Queueing and Telemetry Transport (MQTT) 3.1.

This section describes the architecture at the following levels.

Component Architecture

The following diagram depicts the component-based architecture of the WSO2 Message Broker.

Transports
The WSO2 Message Broker supports the AMQP transport and the MQTT transport.

Inbound Disruptor

All incoming events are inserted into the inbound disruptor ring where many handlers work in parallel. Data is stored and deleted through the persistence layer.

Outbound Disruptor
This reads messages from the database concurrently, and passes them to the transport for delivery.

Slot Management
A queue can be divided into several slots. A slot is a chunk of messages which can be owned by one node at a time.  The slot manager generates distributes slots between slot delivery workers based on the requirement. A publisher returns the last message ID to the slot manager after every 1000 messages. 

Data Stores
These are used to save any information related to messaging such as AMQP exchanges, message content etc. Information relating to topic context and authentication are saved in the registry. Other information is saved in the message store.

Andes Kernel
This contains the WSO2 specific implementation, which is used when handling different messaging protocols.

Slot-based architecture

The slot-based message delivery system is designed to enable global queues to be shared among the nodes in an MB cluster.

A queue is mapped to a row in a message store and it can be divided into many slots. A slot is a chunk of messages in the row that can be owned by one MB node at a given time. 

The slot manager communicates with both publishers and subscribers and acts as the coordinator for distributing slots among the nodes. A slot assignment map is maintained to track the slots that are assigned to nodes at a given time. The activities of a slot manager can be illustrated as follows.

Communicating with publishers
Each publisher belongs to a queue/topic and its messages are published in the row of the message store mapped to this queue. A publisher returns its last message ID to the slot manager node in the MB cluster after every 1000 messages or after a timeout. The slot manager updates the Hazelcast Distributed Map with these IDs and uses them to generate slots when it receives requests for slots from slot delivery worker nodes.

Assigning slots
When a client subscribes to a queue/topic, a slot delivery worker requests for a slot. Then the slot manager first looks for returned slots (i.e. slots that were previously assigned to another subscriber node which has left the cluster) and assigns one of them if any are available. If there are no such slots, an empty slot (i.e. a slot with no messages currently published in it) is generated and assigned to the slot delivery worker.

Deleting a slot
Once a subscriber node has sent all the messages it has read from a slot and received acknowledgements, it sends a request to the slot manager to delete the slot. The slot manager removes the relevant entry from the slot assignment map once it receives the request to delete.

Reassigning slot when the last subscriber leaves
If a subscriber node to which a slot is assigned leaves the cluster, the slot manager reassigns the slot to another subscriber node to free the slot pool. 

Delivering messages to subscribers
A slot delivery worker reads all the messages published in a slot assigned to it and passes them to the message flusher, which delivers them to subscribers who have subscribed to the relevant queue/topic in a Round Robin manner. Messages that were not delivered due to a delivery failure and messages rejected by the subscriber are buffered queue-wise in the Message Flusher.