You can troubleshoot and trace possible errors that can occur with WSO2 Message Broker (WSO2 MB) in a given environment by using the methods given below.
...
Class | Description | |
|---|---|---|
| Inbound |
| All inbound events (e.g. such as message arrival, subscription add/close events etc.) are handled through this class. |
| The incoming message goes through this processor first, where its message ID and destination data are populated to ensure the message order closest to the message arrival time. | |
| This processor will take the message content chunks, convert them to the andes core chunk size and delegate the rest of the work to the | |
| This processor will write the message metadata and content chunks to the storage database using a batch approach. | |
| Upon saving the message to storage, this handler is triggered to notify a message received event, or to notify a message acknowledged event from the consumer side. | |
| This event is used to communicate the transaction commit/rollback events from the publisher side. | |
| Outbound |
| This processor is used to deliver the message to one/all of the active subscriptions (based on the message destination). |
| This class is used to handover the message to the consumer after reading from the internal message buffer ( | |
| There are multiple SlotDeliveryWorkers slot delivery workers managed by the | |
| This is where the coordinator logic resides within WSO2 MB. All slots are managed and distributed through this class across the cluster. | |
| AMQP |
| A channel is used for delivering and accepting messages to/from the broker. Each AMQP consumer/publisher has its own unique channel with a channel ID. |
| This is used as the bridge between the Qpid messaging events and Andes events. | |
| MQTT |
| This is used as the bridge between the moquette messaging events and Andes events. |
| This handles all events coming through the moquette disruptor (e.g. such as subscriber-connect, pub-acks etc.) and connects to the | |
| This class acts as an interface before storing MQTT messages to the message store, validating the message format, in addition to handling events such as consumer/publisher creation and closing in terms of the message store. | |
| This class handles the lifecycle of MQTT subscriptions and also takes part in routing a given message to matching subscribers. |
...
This is an MB-specific logging implementation for tracing a message through its inbound event until it is delivered to the consumer application. This implementation has minimal impact on the performance of the broker functionality. To enable message tracing in WSO2 MB:
- Open the
log4j.propertiesfile stored in the<MB_HOME>/repository/conffolder. Uncomment the following:
Code Block #log4j.logger.org.wso2.andes.tools.utils.MessageTracer=TRACE,CARBON_TRACE_LOGFILE
Once message tracing is enabled, you can start the server and execute a grep command with the relevant message ID you want to trace. This will give print all the logs related to your message ID on your terminal.
...
You can easily create a a heap dump and thread dump using the CarbonDump tool that is shipped with your product. This These will also provide information about the product version and any patch inconsistencies.
...
| Table | Description |
|---|---|
| Each slot, the assigned node ID and the current status are maintained here. |
| Whenever a node communicates a possible slot range to the coordinator, the node will decide on the appropriate message ID range to be included in the slot and update this table with the last |
| This table contains the last published message ID for each node in order to calculate the global safe zone (minimum messageID from all nodes) that is required for deleting slots upon completion. |
| Whenever a slot is given by the coordinator to an MB node for processing, its |
Within this contextWith the above information, you can infer the following validations in the database at any given time:
There should not be any slots in the
MB_SLOTtable if theMB_METADATAtable is empty. This is an eventual guarantee. Even if there are slots queued for deletion, this rule must still be satisfied after some time.There should be no records in the
MB_METADATAtable if theMB_CONTENTtable is empty (one-to-one relationship).- Given the minimum message ID in the
MB_NODE_TO_LAST_PUBLISHED_IDtable, all slots within theMB_SLOTtable with the “assigned” status (state = 2) and the endMessageID less than the minimum published ID should be deleted (or at-least be cleared after some time).
Retrieving logs from the JMS client
You can simply monitor the logs from the JMS clients connecting to WSO2 MB by enabling the following startup property on the clients:
| Code Block |
|---|
-Damqj.protocol.logging.level=true |
Monitoring JAVA metrics
The metrics dashboard of WSO2 MB provides general JVM metrics as well as MB-specific metrics to help you identify how the broker is running in a loaded/relaxed environment. This functionality will give you information such as the unexpected increases of delivery channels, latencies of database reads/writes etc., which will help you identify possible errors in the system. See the documentation on metrics for instructions on how to configure and use the metrics dashboard.
Identifying common warnings/logs
The following table details some of the most common warning messages/logs that can be encountered when working with WSO2 MB. You will also find here the possible causes and solutions for such warnings/logs.
Warning | Cause | Solutions / Approach |
|---|---|---|
[WARN] Invalid message state transition from <state1> | This means that the message lifecycle has deviated from itsthe expected execution path (e. g.Example: getting SENT -> SCHEDULED_TO_SEND ). | First realize , distinguish the expected execution order through class using the “org.wso2.andes.kernel.MessageStatus”[11] class and then try to pinpoint identify the exact phase point at which the issue occurs. (High occurred. There is a high probability of race conditions.) |
[WARN] Invalid State transition from <stateA> suggested : <stateB> Slot ID : MyQueue|22309482... | This means that the slot lifecycle (used for delivery) has deviated from itsthe expected execution path (e. g.Example: CREATED -> DELETED ). | First realize , distinguish the expected execution order through class using the “org.wso2.andes.kernel.SlotState” [12]class and then try to pinpoint identify the exact phase point at which the issue occurs. (High occurred. There is a high probability of race conditions.) |
[WARN] Error when trying to read property <property>. Switching to default value : <defaultValue> | This can happen if the MB configuration file ( | Make Be sure that the correct version of the |
[INFO] Local subscription ADDED [TestQueue]ID=635@NODE/10.100.5.115:4000/T=1456518837302/D=true/X=false/O=null/E= | This log is printed whenever a new queue/topic subscription is added to the cluster. Lifecycle The lifecycle of a subscription is ADDED -> DELETED. In case of durable topic subscriptions, a subscription may be disconnected before being deleted (. If a subscription is disconnected, its the messages will still persist until the subscriber is deleted). *Please note Note that this log is not printed if a durable subscription is disconnected and added for the second time after the cluster startup (Since it’s only a state update in terms of durable topics)starts. | Info Level log. |
Channel created (ID: 21765) {org.wso2.andes.kernel.AndesChannel} | This log is printed every time a publish/subscribe channel is established with between the client and the broker node. An Andes channel is mapped (one-to-one) to an | Info level log. |