Clustered Deployment

In a production environment, WSO2 Message Broker (WSO2 MB) can be clustered and configured with an external Database Management System (DBMS) of your choice. 

Advantages of clustering WSO2 MB

Clustering will give you a very scalable product. This is useful because adding more WSO2 MB nodes to your cluster enables you to publish your messages or do subscriptions in a load balanced way. With clustering, your product can be scaled up to meet high messaging demands and if there are performance issues, the product can be scaled down according to the requirement. Therefore, WSO2 Message Broker can deliver high-performance results as opposed to many commercial and conventional Message Brokers that have very low performance when the "size of a message" becomes too large.

Recommended deployment pattern

The following is the common deployment model referred to in the configurations. This is the recommended deployment pattern for WSO2 Message Broker and depicts the minimum number of broker nodes necessary to achieve high availability and high efficiency in the cluster.

Figure 1: Message Broker Nodes Configured With An External DBMS Cluster.

In this deployment model, assume that all these nodes are on different hosts. There are two WSO2 Message Broker nodes and any number of nodes that you prefer for the external DBMS (this can even be a single node).

Although this example specifies two WSO2 Message Broker nodes as a minimum requirement for optimal performance, you can have more Message Broker nodes depending on your requirement.

Use the following instructions to configure the two Message Broker nodes and connect them to an external DBMS.

Setting up the DBMS

The following steps describe how to download and install MySQL Server, create the databases, configure the datasources, and configure WSO2 MB to connect to them.

1. Download and install MySQL Server.

2. Download the MySQL JDBC driver.

3. Unzip the downloaded MySQL driver zipped archive, and copy the MySQL JDBC driver JAR (mysql-connector-java-x.x.xx-bin.jar) into the <PRODUCT_HOME>/repository/components/lib directory.

4. Define the host name for configuring permissions for the new database by opening the /etc/hosts file and adding the following line:
   

   <MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com

   You would do this step only if your database is on a separate server (not on your local machine).

5. Enter the following command in a terminal/command window, where username is the username you want to use to access the databases:
   mysql -u username -p
6. When prompted, specify the password that will be used to access the databases with the username you specified.

7. Create the databases using the following commands, where <PRODUCT_HOME> is the path to any of the product instances you installed, and username and password are the same as those you specified in the previous steps:

   About using MySQL in different operating systems

   For users of Microsoft Windows, when creating the database in MySQL, it is important to specify the character set as latin1. Failure to do this may result in an error (error code: 1709) when starting your cluster. This error occurs in certain versions of MySQL (5.6.x) and is related to the UTF-8 encoding. MySQL originally used the latin1 character set by default, which stored characters in a 2-byte sequence. However, in recent versions, MySQL defaults to UTF-8 to be friendlier to international users. Hence, you must use latin1 as the character set as indicated below in the database creation commands to avoid this problem. Note that this may result in issues with non-latin characters (like Hebrew, Japanese, etc.). The following is how your database creation command should look.

   mysql> create database <DATABASE_NAME> character set latin1;

   For users of other operating systems, the standard database creation commands will suffice. For these operating systems, the following is how your database creation command should look.

   mysql> create database <DATABASE_NAME>;

   mysql> create database wso2_mb;
   mysql> use wso2_mb;
   mysql> source <MB_HOME>/dbscripts/mb-store/mysql-mb.sql;
   mysql> grant all on wso2_mb.* TO username@localhost identified by "password";
   
   
   mysql> create database userdb;
   mysql> use userdb;
   mysql> source <MB_HOME>/dbscripts/mysql.sql;
   mysql> grant all on userdb.* TO username@localhost identified by "password";
   
   
   mysql> create database regdb;
   mysql> use regdb;
   mysql> source <MB_HOME>/dbscripts/mysql.sql;
   mysql> grant all on regdb.* TO username@localhost identified by "password";

   Note: Ensure that MySQL is configured so that all nodes can connect to it.

Configuring the WSO2 MB nodes in the cluster

Once you have set up the DBMS for your cluster as explained previously, you can update the configuration files in each node of the WSO2 MB cluster as explained below.

Configuring the broker.xml

1. Open the <MB_HOME>/repository/conf/broker.xml file. This is the root configuration file of WSO2 MB. The changes made to this file must be done in all the WSO2 MB nodes.

2. Do thrift-related configurations. Here you must configure the thriftServerHost value to point to the IP address of the MB server node.

   See the following table for details on these configurations.

   Configuration	Description
   coordination	This configuration is related to MB thrift communications.
   nodeID	The node ID of each member in a cluster must be unique. In a clustered deployment, an ID is assigned to each MB node via the cluster node identifier. The node ID specified by this parameter can be used to override the cluster node identifier for this MB node. If the node ID is set to 'default', the node ID will be retrieved using the Hazelcast member ID that is specified in the axis2.xml file as shown below. <parameter name="localMemberHost">192.168.0.102</parameter> Note that when this parameter is set to 'default', the node ID is made up of the 'localMemberHost' and the 'localMemberPort' (defined in the axis2.xml) as shown below. The 'localMemberHost' will be the hostname of the server if the hostname can be mapped with the provided IP address. Otherwise, the 'localMemberHost' will always be the provided IP address. NODE:localMemberHost:localMemberPort
   thriftServerHost	This is a sub-element of the <coordination> tag. WSO2 MB uses Apache Thrift for communications relating to message delivery. Therefore, an Apache Thrift server is started in each MB node in a clustered deployment. This element should point to the IP address of the Apache Thrift server. This should point to the IP address of the MB node that hosts the thrift server. The default value for this is localhost. For example, if you are configuring a Message Broker node hosted in 192.168.0.102 as the thrift server, this value should be 192.168.0.102.
   thriftServerPort	This is another sub-element of the <coordination> tag. This should point to the port of the thrift server in MB. The default port is 7611. It is recommended to use this port for all broker nodes in your cluster.
   thriftSOTimeout	This is used to handle half-open TCP connections between the broker nodes in a cluster. In such situations, the socket may need to have a timeout value to invalidate the connection (in milliseconds). A timeout of zero is interpreted as an infinite timeout. Be sure to set this value to 180000 milliseconds as shown below in order to handle a high number of half-open TCP connections. <thriftSOTimeout>180000</thriftSOTimeout>

Configuring the qpid-config.xml file

Enable heartbeat messaging for each of the broker nodes by following the steps given below. This is required for handling TCP connections between the broker nodes and client systems (publishers and subscribers).

1. Open the qpid-config.xml file that is stored in the <MB_HOME>/repository/conf/advanced/ directory.

2. Set the heartbeat delay as shown below. The recommended heartbeat delay is 30 seconds.

   <heartbeat>
       <delay>30</delay>
       <timeoutFactor>2.0</timeoutFactor>
   </heartbeat>

   The elements in the above configuration are explained below.

   delay	The <delay> element specifies the time interval between heartbeat messages. This is the time that the broker or client will wait to receive a response from the other party. If the response is not received within this time, a heartbeat message is triggered.
   timeoutFactor	The number of heartbeat messages the broker will send to the client before terminating the connection. That is, if the timeoutFactor is 2, the broker will send heartbeat messages every 30 seconds twice, and if a response is not received from the client, the connection will be terminated.

Configuring the axis2.xml

This section provides all the configurations related to enabling and correctly configuring clustering.

1. Open the <MB_HOME>/repository/conf/axis2/axis2.xml file. The changes made to this file must be done in both broker nodes.

2. Enable Hazelcast clustering by doing the following configuration.
   <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

3. Specify the membership scheme that you plan to use for clustering. See Clustering Overview for more information on membership schemes.
   <parameter name="membershipScheme">wka</parameter>

4. Configure the localMemberHost and the localMemberPort of the server to point to the IP address of the host where the MB server resides. This has to be done for each server.

5. When using the “wka” membership scheme, each member of the cluster should be configured with the information about other cluster members. In this case, the other broker node must be defined here. For example, if the broker node you are configuring is on 192.168.0.102, you must configure the other broker node, which is hosted on 192.168.0.103, as indicated below.

Configuring user-mgt.xml

1. Open the <MB_HOME>/repository/conf/user-mgt.xml file on each MB server.

2. Update the dataSource property in user-mgt.xml to the jndiConfig name used in master-datasources.xml for user store database configuration.
   

   <Property name="dataSource">jdbc/WSO2UserStoreDB</Property>

Configuring master-datasources.xml

In this configuration file, you must configure the datasources to point to the databases that you configured earlier.

1. Open the <MB_HOME>/repository/conf/datasources/master-datasources.xml file. The changes made to this file must be done in both broker nodes.

2. Remove the H2-based WSO2_MB_STORE_DB configuration. This is the default configuration. To do this, remove or comment out the following code snippet.
   

   <datasource>
           	<name>WSO2_MB_STORE_DB</name>
           	<description>The datasource used for message broker database</description>
           	<jndiConfig>
               	<name>WSO2MBStoreDB</name>
           	</jndiConfig>
           	<definition type="RDBMS">
               	<configuration>
                   	<url>jdbc:h2:repository/database/WSO2MB_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url>
                   	<username>wso2carbon</username>
                   	<password>wso2carbon</password>
                   	<driverClassName>org.h2.Driver</driverClassName>
                   	<maxActive>50</maxActive>
                   	<maxWait>60000</maxWait>
                   	<testOnBorrow>true</testOnBorrow>
                   	<validationQuery>SELECT 1</validationQuery>
                   	<validationInterval>30000</validationInterval>
                   	<defaultAutoCommit>false</defaultAutoCommit>
               	</configuration>
           	</definition>
   </datasource>

3. Uncomment or add the following MySQL-based WSO2_MB_STORE_DB configuration.

   <datasource>
          <name>WSO2_MB_STORE_DB</name>
          <jndiConfig>
              <name>WSO2MBStoreDB</name>
          </jndiConfig>
          <definition type="RDBMS">
              <configuration>
                  <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                  <url>jdbc:mysql://localhost/wso2_mb</url>
                  <username>root</username>
                  <password>root</password>
                  <maxActive>50</maxActive>
                  <maxWait>60000</maxWait>
                  <minIdle>5</minIdle>
                  <testOnBorrow>true</testOnBorrow>
                  <validationQuery>SELECT 1</validationQuery>
                  <validationInterval>30000</validationInterval>
                  <defaultAutoCommit>false</defaultAutoCommit>
               </configuration>
           </definition>
   </datasource> 

4. Add the user store database configuration.

   <datasource>
            <name>WSO2_USER_STORE_DB</name>
            <jndiConfig>
                <name>jdbc/WSO2UserStoreDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <url>jdbc:mysql://192.168.0.104/userdb</url>
                    <username>root</username>
                    <password>root</password>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <minIdle>5</minIdle>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                   <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
   </datasource>

5. Add the registry database configuration.

   <datasource>
       <name>WSO2_GOV_DB</name>
           <description>The datasource used for registry and user manager</description>
           <jndiConfig>
           <name>jdbc/WSO2GovDB</name>
       </jndiConfig>
       <definition type="RDBMS">
           <configuration>
               <url>jdbc:mysql://192.168.0.104/regdb</url>
               <username>root</username>
               <password>root</password>
               <driverClassName>com.mysql.jdbc.Driver</driverClassName>
               <maxActive>80</maxActive>
               <maxWait>60000</maxWait>
               <minIdle>5</minIdle>
               <testOnBorrow>true</testOnBorrow>
               <validationQuery>SELECT 1</validationQuery>
               <validationInterval>30000</validationInterval>
           </configuration>
       </definition>
   </datasource>

Configuring registry.xml

In this configuration file, you must correctly mount the registry database.

1. Open the <MB_HOME>/repository/conf/registry.xml file. The changes made to this file must be done in all the WSO2 MB nodes. 

2. Add the following configurations to mount the registry and do not replace any existing configurations.
   

   <dbConfig name="wso2govregistry">
   	<dataSource>jdbc/WSO2GovDB</dataSource>
   </dbConfig>
    
   <remoteInstance url="https://localhost:9443/registry">
       <id>govdb</id>
       <dbConfig>wso2govregistry</dbConfig>
       <readOnly>false</readOnly>
       <enableCache>true</enableCache>
       <registryRoot>/</registryRoot>
   </remoteInstance>
    
   <mount path="/_system/governance" overwrite="true">
       <instanceId>govdb</instanceId>
       <targetPath>/_system/governance</targetPath>
   </mount>

   When mounting configurations, make note of the following.

   • The dataSource you specify under the <dbConfig name="sharedregistry"> tag must match the jndiConfig name you specified in the master-datasources.xml file.
   • The registry mountpath is used to identify the type of registry. For example, ”/_system/config” refers to configuration registry, and "/_system/governance" refers to the governance registry.
   • The dbconfig entry enables you to identify the datasource you configured in the master-datasources.xml file. We use the unique name sharedregistry to refer to that datasource entry. 
   • The remoteInstance section refers to an external registry mount. We can specify the read-only/read-write nature of this instance as well as caching configurations and the registry root location. In the case of a Message Broker node, the readOnly property must be set to false as we require all broker nodes to have both read and write permissions. The remoteInstance url is used internally in Governance Registry for notification functionality. For all other non-Governance Registry server instances, you can leave this property as it is. If your MB cluster is not configured with a WSO2 Governance Registry, you do not need to change this URL entry for new values. 
   • You must define a unique name “id” for each remote instance, which is then referred to from mount configurations. In the above example, the unique ID for the remote instance is govdb. 

   • In each of the mounting configurations, we specify the actual mount path and target mount path. The targetPath can be any meaningful name. In this instance, it is /_system/mbNodes.

Configuring cluster coordination

WSO2 MB 3.2.0 introduces cluster coordination through an RDBMS. This means that the coordination between the nodes in a cluster can be managed through an RDBMS, just as message persistence. Therefore, the RDBMS that is connected to the MB nodes in the cluster is, by default, used for message persistence as well as cluster coordination. Shown below is the configuration in the broker.xml file (stored in the <MB_HOME>/repository/conf/ directory), which enables RDBMS-based cluster coordination. If required, you can disable the following configuration, which will allow the hazelcast engine to manage cluster coordination. However, note that you need to configure the cluster to handle network partitioning when hazelcast-based cluster coordination is used. 

<rdbmsBasedCoordination enabled="true">
    <!-- Heartbeat interval used in the RDBMS base coordination algorithm in milliseconds -->
    <heartbeatInterval>5000</heartbeatInterval>
    <!-- Time to wait before informing others about coordinator change in milliseconds. This value should be
            larger than a database read time including network latency and should be less than heartbeatInterval -->
    <coordinatorEntryCreationWaitTime>3000</coordinatorEntryCreationWaitTime>
    <!-- Time interval used to poll database for membership related events in milliseconds. -->
    <eventPollingInterval>4000</eventPollingInterval>
</rdbmsBasedCoordination>
<!-- Enabling this will make the cluster notifications such as Queue changes(additions and deletions),
        Subscription changes, etc. sent within the cluster be synchronized using RDBMS. If set to false, Hazelcast
        will be used for this purpose.-->
<RDBMSBasedClusterEventSynchronization enabled="true">
    <!--Specifies the interval at which, the cluster events will be read from the database. Needs to be
            declared in milliseconds. Setting this to a very low value could downgrade the performance where as
            setting this to a large value could increase the time taken for a cluster event to be synchronized in
            all the nodes in a cluster.-->
    <eventSyncInterval>1000</eventSyncInterval>
</RDBMSBasedClusterEventSynchronization>

Handling network partitioning in Hazelcast-based clustering

Network partitioning in the network used for cluster coordination can sometimes disrupt the cluster. WSO2 MB 3.2.0 introduces a new configuration to handle network partitioning that may occur when cluster coordination is managed by the hazelcast engine (as opposed to the RDBMS as explained above).

For example, consider a cluster of three WSO2 MB nodes that uses two separate networks for cluster coordination and message persistence. The network with the RDBMS will persist messages, subscriptions and queues etc., and the cluster network communicates information about subscriptions, queue additions, and to decide on the coordinator of the cluster. If one of the nodes in the cluster disconnects from the network used for cluster coordination, that node will separate from the cluster and function as a separate node/cluster (separate partition) as illustrated in the diagram below. That is, the three-node cluster will now be working as two separate clusters. However, the disconnected node will still be up and running, and message persistence will continue uninterrupted through the message persistence network. This will cause inconsistencies in the system because message persistence and cluster coordination will not be synchronized. In such a situation, it is necessary to stop the disconnected node from accepting messages.

The above situation can be prevented in MB 3.2.0 by configuring the minimum node count of the cluster. For example, if the cluster size is 5 and we have configured the minimum node count to 3 nodes, during a network partition (or when nodes crash or shut down) any partition that has 3 or more nodes will keep functioning, while the other partitions will stop processing messages.

Follow the steps given below to enable this feature.

1. Open the <MB_HOME>/repository/conf/broker.xml file and set the <networkPartitionsDetection> element shown below to enabled="true". Note that this configuration is disabled by default as shown below.
   

   <?xml version="1.0"?>
   <networkPartitionsDetection enabled="false">
       <minimumClusterSize>1</minimumClusterSize>
   </networkPartitionsDetection>

2. Update the <minimumClusterSize> property in the above configuration to specify the minimum node count the cluster should maintain in order to operate. Note that if this value should be at least two for cluster coordination to work. If the number of nodes in the cluster becomes less that configured value, the cluster will not accept any incoming traffic. That is, all subscriptions will be disconnected.

Starting the servers

Start the MB servers by executing the <MB_HOME>/bin/wso2server.sh file.
$ ./wso2server.sh