Failover

High availability is an important aspect associated with any server implementation that ensures a certain degree of operational continuity when unplanned downtime events impact parts of a system. JMS 1.1 does not provide this; instead, most of the time it is vendor specific.

The idea behind failover is stopping a single-point-of-failure in a system. As the broker is the middle man storing and forwarding the messages, if that server goes down, the entire message flow of the system will go down no matter what other servers and functions are involved. In order to make a robust messaging system, it is mandatory to have a failover mechanism.

In order to achieve this, a few instances of message broker servers are set up and running in the system, while the system (generally) uses one broker. If that broker goes down, it automatically switches to the second broker and continues messaging. If the second broker fails, it will try the next one and so on. Thus, the whole system will not have any downtime.

This page covers the following topics:

Failover with WSO2 Message Broker and WSO2 ESB

Follow the instructions below to set up a failover mechanism using MB and ESB:

Setting up MB instances in a single machine

Download the latest version of MB and extract it into a folder.
Make an exact copy of MB in three different locations and rename them as MB1, MB2 and MB3. Those will be the 3 separate MB instances.

Navigate to MB1/repository/conf/carbon.xml and define an offset of 1.

<Ports>
<!-- Ports offset. This entry will set the value of the ports defined below to
 the define value + Offset.
 e.g. Offset=2 and HTTPS port=9443 will set the effective HTTPS port to 9445
 -->
 <Offset>1</Offset>

Navigate to MB2/repository/conf/carbon.xml and define an offset of 2.
Navigate to MB3/repository/conf/carbon.xml and define an offset of 3.
The 3 MB instances are ready. Start each instance of MB by running one of the following commands:
- <MB_HOME>/bin/wso2server.sh (on Linux)
- <MB_HOME>/bin/wso2server.bat (on Windows)
The MB instances will start on ports 5673, 5674 and 5675 respectively.

In the above setup, 3 separate MB servers are started along with dedicated Cassandra servers for each. Those Cassandra servers will run in ports 9160, 9161 and 9162 respectively.

Setting up MB instances in different machines

Download the latest version of MB and extract it into a folder.
Make an exact copy of that folder in three separate machines. Let's assume MB1 is in the machine with IP1, MB2 in the machine with IP2 and MB3 in the machine with IP3.
Start all the above instances by running one of the following commands:
- <MB_HOME>/bin/wso2server.sh (on Linux)
- <MB_HOME>/bin/wso2server.bat (on Windows)
All severs will start with port offset 0, which is 5672.

Setting up a WSO2 ESB instance

Start ESB in the default port (port offset 0). This is possible if you used the single machine setup above, as those servers were started with different port offsets. If you are using different machines, use another machine to start WSO2 ESB. Instructions can be found in Configure with WSO2 Message Broker.

It is not possible to start multiple WSO2 products with their default configurations simultaneously in the same environment. Since all WSO2 products use the same port in their default configuration, there will be port conflicts.

There is only a single difference to enable failover across the three brokers we have setup. That is when specifying the following in the ESB_HOME/repository/conf/jndi.properties file:

connectionfactory.ConnectionFactory

connectionfactory.QueueConnectionFactory

connectionfactory.TopicConnectionFactory

For example, if you hope to use a single machine MB setup as described above,

connectionfactory.QueueConnectionFactory = amqp://admin:admin@clientID/carbon?failover='roundrobin'&cyclecount='2'&brokerlist='tcp://localhost:5673?retries='5'&connectdelay='30000';tcp://localhost:5674?retries='5'&connectdelay='30000';tcp://localhost:5675?retries='5'&connectdelay='30000''
 
connectionfactory.TopicConnectionFactory = amqp://admin:admin@clientID/carbon?failover='roundrobin'&cyclecount='2'&brokerlist='tcp://localhost:5673?retries='5'&connectdelay='30000';tcp://localhost:5674?retries='5'&connectdelay='30000';tcp://localhost:5675?retries='5'&connectdelay='30000''

If you hope to use the broker setup made across several machines as described above,

connectionfactory.QueueConnectionFactory = amqp://admin:admin@clientID/carbon?failover='roundrobin'&cyclecount='2'&brokerlist='tcp://IP1:5672?retries='5'&connectdelay='30000';tcp://IP2:5672?retries='5'&connectdelay='30000';tcp://IP3:5672?retries='5'&connectdelay='30000''
 
connectionfactory.TopicConnectionFactory = amqp://admin:admin@clientID/carbon?failover='roundrobin'&cyclecount='2'&brokerlist='tcp://IP1:5672?retries='5'&connectdelay='30000';tcp://IP2:5672?retries='5'&connectdelay='30000';tcp://IP3:5672?retries='5'&connectdelay='30000''

The parameters used above are described in detail below:

Brokerlist option

brokerlist='<broker url>[;<broker url>]'

The broker list defines the various brokers that can be used for this connection. A minimum of one broker URL is required. Additional URLs are semi-colon(';') delimited.

Broker URL format

<transport>://<host>[:<port>][?<option>='<value>'[&<option>='<value>']]

Option	Default	Description
retries	1	The number of times to retry connection to this Broker
ssl	false	Use ssl on the connection
connecttimeout	30000	How long in (milliseconds) to wait for the connection to succeed
connectdelay	none	How long in (milliseconds) to wait before attempting to reconnect

Brokerlist failover option

failover='<method>[?<options>]'

Method	Description
singlebroker	This will only use the first broker in the list.
roundrobin	This method tries each broker in turn.
nofailover	[New in 0.5] This method disables all retry and failover logic.

The current default is to use the singlebroker method when only one broker is present, and the roundrobin method with multiple brokers. The method value in the URL may also be any valid class on the classpath that implements the FailoverMethod interface.

Options

Option	Default	Description
cyclecount	1	The number of times to loop through the list of available brokers before failure.

Testing the setup

When sending messages to a queue (you can configure SOAP UI to send messages continuously one after the other) using the setup described above, kill MB1. You will notice that failover happens at ESB to the second broker i.e MB2. The logs at the ESB side will indicate that failover occurred.

If there were any previous messages at MB1, they will not be delivered.

Usually, the solution for that (and many other problems) is broker clustering. For information on the advantages of clustering and WSO2 Message Broker's support on clustering, see Clustering Message Broker.