This site contains the documentation that is relevant to older WSO2 product versions and offerings.
For the latest WSO2 documentation, visit https://wso2.com/documentation/.

Clustering and Deployment

Owned by Former user (Deleted)
Last updated: Jul 25, 2018
12 min read

In a production environment, WSO2 Message Broker can be clustered and configured with an external Database Management System (DBMS) of your choice. 

Deployment architecture

The following is the common deployment model referred to in the configurations. This is the recommended deployment pattern for WSO2 Message Broker and depicts the minimum number of broker nodes necessary to achieve high availability and high efficiency in the cluster.

Figure 1: Message Broker Nodes Configured With An External DBMS Cluster.

In this deployment model, assume that all these nodes are on different hosts. There are two WSO2 Message Broker nodes and any number of nodes that you prefer for the external DBMS (this can even be a single node).

ServerIP Address
MB Server 01192.168.0.102
MB Server 02192.168.0.103
RDBMS192.168.0.104

Although this example specifies two WSO2 Message Broker nodes as a minimum requirement for optimal performance, you can have more Message Broker nodes depending on your requirement.

Time synchronization across all nodes

It is vital to time synchronize all the nodes across the cluster to make the Message Broker nodes function properly.

Ensure all required ports are open

Ensure that required ports are open in the firewall or not bound to any other service. The list of ports used by WSO2 Message Broker can be found in Default Ports of WSO2 Products.

Advantages of clustering WSO2 Message Broker

The advantage that can be gained when clustering the Message Broker is that you can have a very scalable product. This is useful as adding more Message Broker nodes to your cluster enables you to publish your messages or do subscriptions in a load balanced way. The Message Broker can also be scaled up to meet high messaging demands and if you are unhappy with the performance, MB can be scaled to meet your requirements. 

Due to the above reason, WSO2 Message Broker can deliver high-performance results as opposed to many commercial and conventional Message Brokers that have very low performance when the "size of a message" becomes too large.

Follow the instructions below to configure the cluster.

Configuring clustering for WSO2 Message Broker

Use the instructions available in the following sections to configure the two Message Broker nodes for clustering and connecting them to an external DBMS.

Installing and configuring the database

The following steps describe how to download and install MySQL Server, create the databases, configure the data sources, and configure the Message Broker to connect to them.

Note: MySQL is used as an example here, but you can use any database for this and change the configurations accordingly. See Configuring the DBMS for Storage for more options.

  1. Download and install MySQL Server.

  2. Download the MySQL JDBC driver.

  3. Unzip the downloaded MySQL driver zipped archive, and copy the MySQL JDBC driver JAR (mysql-connector-java-x.x.xx-bin.jar) into the <PRODUCT_HOME>/repository/components/lib directories of both the WSO2 MB nodes.

  4. Define the host name for configuring permissions for the new database by opening the /etc/hosts file and adding the following line:
    <MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com

    You would do this step only if your database is not on your local machine and on a separate server.

  5. Enter the following command in a terminal/command window, where username is the username you want to use to access the databases:
    mysql -u username -p
  6. When prompted, specify the password that will be used to access the databases with the username you specified.

  7. Create the databases using the following commands, where <PRODUCT_HOME> is the path to any of the product instances you installed, and username and password are the same as those you specified in the previous steps:

    About using MySQL in different operating systems

    For users of Microsoft Windows, when creating the database in MySQL, it is important to specify the character set as latin1. Failure to do this may result in an error (error code: 1709) when starting your cluster. This error occurs in certain versions of MySQL (5.6.x) and is related to the UTF-8 encoding. MySQL originally used the latin1 character set by default, which stored characters in a 2-byte sequence. However, in recent versions, MySQL defaults to UTF-8 to be friendlier to international users. Hence, you must use latin1 as the character set as indicated below in the database creation commands to avoid this problem. Note that this may result in issues with non-latin characters (like Hebrew, Japanese, etc.). The following is how your database creation command should look.

    mysql> create database <DATABASE_NAME> character set latin1;

    For users of other operating systems, the standard database creation commands will suffice. For these operating systems, the following is how your database creation command should look.

    mysql> create database <DATABASE_NAME>;
    mysql> create database wso2_mb;
mysql> use wso2_mb;
mysql> source <MB_HOME>/dbscripts/mb-store/mysql-mb.sql;
mysql> grant all on wso2_mb.* TO username@localhost identified by "password";

mysql> create database userdb;
mysql> use userdb;
mysql> source <MB_HOME>/dbscripts/mysql.sql;
mysql> grant all on userdb.* TO username@localhost identified by "password";

mysql> create database regdb;
mysql> use regdb;
mysql> source <MB_HOME>/dbscripts/mysql.sql;
mysql> grant all on regdb.* TO username@localhost identified by "password";

    Note: Ensure that MySQL is configured so that all nodes can connect to it.

Configuring broker.xml

  1. Open the <MB_HOME>/repository/conf/broker.xml file. This is the root configuration file of Message Broker. The changes made to this file must be done in all the WSO2 Message Broker nodes.

  2. Do thrift-related configurations. Here you must configure the thriftServerHost value to point to the IP address of the MB server node.

    See the following table for details on these configurations.

    ConfigurationDescription
    coordination

    This configuration is related to MB thrift communications.

    nodeIDIn a clustered deployment, an ID is assigned to each MB node via the cluster node identifier. This element can be used to override the cluster node identifier for this MB node. If the value for this element is left as default, the default node ID is generated using the IP and a universally unique identifier (UUID). The node ID of each member in a cluster must be unique.
    thriftServerHost

    This is a sub-element of the <coordination> tag. WSO2 MB uses Apache Thrift for communications relating to message delivery. Therefore, an Apache Thrift server is started in each MB node in a clustered deployment. This element should point to the IP address of the Apache Thrift server. This should point to the IP address of the MB node that hosts the thrift server. The default value for this is localhost. For example, if you are configuring a Message Broker node hosted in 192.168.0.102 as the thrift server, this value should be 192.168.0.102.

    thriftServerPort

    This is another sub-element of the <coordination> tag. This should point to the port of the thrift server in MB. The default port is 7611. It is recommended to use this port for all broker nodes in your cluster.

Configuring the qpid-config.xml file

Enable heartbeat messaging for each of the broker nodes by following the steps given below. This is required for handling TCP connections between the broker nodes and client systems (publishers and subscribers).

What is heartbeat messaging?

When a client is connected to the broker, both the broker and the client should be able to detect problem situations where the TCP connection is half open or where the connecting client/broker is unresponsive. This can be achieved by enabling heartbeat messaging between the broker and the client.

When this configuration is enabled, both the broker and the connecting client will be able to check if the connection is active, and if the connecting system (broker or client) is active by periodically sending heartbeat messages to each other. 

For example, you can set the heartbeat delay time to 30 seconds. This means that if the broker (or the client) does not receive the expected response from the other system within 30 seconds, it will send a heartbeat message to check if the connection is inactive or if the other system is inactive. If the connection is inactive or is half-open, there will be a clear indication of connection failure. However, if the connection is active but the other system is unresponsive, the broker or the client will continue to send heartbeat messages (every 30 seconds). By default, the broker and clients are configured to terminate (timeout) the connection after sending two heartbeat messages; however, this timeout configuration can be changed.

Note: In a scenario where the subscriber client is sending the heartbeat message to the broker node, if the connection is found to be broken or if the broker node is inactive, the connection will failover to another broker node in the cluster.

  1. Open the qpid-config.xml file that is stored in the <MB_HOME>/repository/conf/advanced/ directory.

  2. Set the heartbeat delay as shown below. The recommended heartbeat delay is 30 seconds.

    <heartbeat>
    <delay>30</delay>
    <timeoutFactor>2.0</timeoutFactor>
</heartbeat>

    The elements in the above configuration are explained below.

    delayThe <delay> element specifies the time interval between heartbeat messages. This is the time that the broker or client will wait to receive a response from the other party. If the response is not received within this time, a heartbeat message is triggered.
    timeoutFactorThe number of heartbeat messages the broker will send to the client before terminating the connection. That is, if the timeoutFactor is 2, the broker will send heartbeat messages every 30 seconds twice, and if a response is not received from the client, the connection will be terminated.

Configuring axis2.xml

This section provides all the configurations related to enabling and correctly configuring clustering.

  1. Open the <MB_HOME>/repository/conf/axis2/axis2.xml file. The changes made to this file must be done in both broker nodes.

  2. Enable Hazelcast clustering by doing the following configuration.
    <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

  3. Specify the membership scheme that you plan to use for clustering. See Clustering Overview for more information on membership schemes.
    <parameter name="membershipScheme">wka</parameter>

  4. Configure the localMemberHost and the localMemberPort of the server to point to the IP address of the host where the MB server resides in. This has to be done for each server.

  5. When using the “wka” membership scheme, each member of the cluster should be configured with the information about other cluster members. In this case, the other broker node must be defined here. For example, if the broker node you are configuring is on 192.168.0.102, you must configure the other broker node, which is hosted on 192.168.0.103, as indicated below.

Configuring user-mgt.xml

  1. Open the <MB_HOME>/repository/conf/user-mgt.xml file on each MB server.

  2. Update the dataSource property in user-mgt.xml to the jndiConfig name used in master-datasources.xml for user store database configuration.

    <Property name="dataSource">jdbc/WSO2UserStoreDB</Property>

Configuring master-datasources.xml

In this configuration file, you must configure the datasources to point to the databases that you configured earlier.

Note: MySQL is used as an example here, but you can use any database for this and change the configurations accordingly. See Configuring the DBMS for Storage for more options.

 Also, make sure to replace the username and password with the username and password used by your database.

  1. Open the <MB_HOME>/repository/conf/datasources/master-datasources.xml file. The changes made to this file must be done in both broker nodes.

  2. Remove the H2-based WSO2_MB_STORE_DB configuration. This is the default configuration. To do this, remove or comment out the following code snippet.

    <datasource>
        	<name>WSO2_MB_STORE_DB</name>
        	<description>The datasource used for message broker database</description>
        	<jndiConfig>
            	<name>WSO2MBStoreDB</name>
        	</jndiConfig>
        	<definition type="RDBMS">
            	<configuration>
                	<url>jdbc:h2:repository/database/WSO2MB_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url>
                	<username>wso2carbon</username>
                	<password>wso2carbon</password>
                	<driverClassName>org.h2.Driver</driverClassName>
                	<maxActive>50</maxActive>
                	<maxWait>60000</maxWait>
                	<testOnBorrow>true</testOnBorrow>
                	<validationQuery>SELECT 1</validationQuery>
                	<validationInterval>30000</validationInterval>
                	<defaultAutoCommit>false</defaultAutoCommit>
            	</configuration>
        	</definition>
</datasource>

  3. Uncomment or add the following MySQL-based WSO2_MB_STORE_DB configuration.

    <datasource>
       <name>WSO2_MB_STORE_DB</name>
       <jndiConfig>
           <name>WSO2MBStoreDB</name>
       </jndiConfig>
       <definition type="RDBMS">
           <configuration>
               <driverClassName>com.mysql.jdbc.Driver</driverClassName>
               <url>jdbc:mysql://localhost/wso2_mb</url>
               <username>root</username>
               <password>root</password>
               <maxActive>50</maxActive>
               <maxWait>60000</maxWait>
               <minIdle>5</minIdle>
               <testOnBorrow>true</testOnBorrow>
               <validationQuery>SELECT 1</validationQuery>
               <validationInterval>30000</validationInterval>
               <defaultAutoCommit>false</defaultAutoCommit>
            </configuration>
        </definition>
</datasource>

  4. Add the user store database configuration.

    <datasource>
         <name>WSO2_USER_STORE_DB</name>
         <jndiConfig>
             <name>jdbc/WSO2UserStoreDB</name>
         </jndiConfig>
         <definition type="RDBMS">
             <configuration>
                 <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                 <url>jdbc:mysql://192.168.0.104/userdb</url>
                 <username>root</username>
                 <password>root</password>
                 <maxActive>50</maxActive>
                 <maxWait>60000</maxWait>
                 <minIdle>5</minIdle>
                 <testOnBorrow>true</testOnBorrow>
                 <validationQuery>SELECT 1</validationQuery>
                <validationInterval>30000</validationInterval>
             </configuration>
         </definition>
</datasource>

  5. Add the registry database configuration.

    <datasource>
    <name>WSO2_GOV_DB</name>
        <description>The datasource used for registry and user manager</description>
        <jndiConfig>
        <name>jdbc/WSO2GovDB</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <url>jdbc:mysql://192.168.0.104/regdb</url>
            <username>root</username>
            <password>root</password>
            <driverClassName>com.mysql.jdbc.Driver</driverClassName>
            <maxActive>80</maxActive>
            <maxWait>60000</maxWait>
            <minIdle>5</minIdle>
            <testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
        </configuration>
    </definition>
</datasource>

Configuring registry.xml

In this configuration file, you must correctly mount the registry database.

  1. Open the <MB_HOME>/repository/conf/registry.xml file. The changes made to this file must be done in all the WSO2 Message Broker nodes. 

  2. Add the following configurations to mount the registry and do not replace any existing configurations.

    <dbConfig name="wso2govregistry">
	<dataSource>jdbc/WSO2GovDB</dataSource>
</dbConfig>
 
<remoteInstance url="https://localhost:9443/registry">
    <id>govdb</id>
    <dbConfig>wso2govregistry</dbConfig>
    <readOnly>false</readOnly>
    <enableCache>true</enableCache>
    <registryRoot>/</registryRoot>
</remoteInstance>
 
<mount path="/_system/governance" overwrite="true">
    <instanceId>govdb</instanceId>
    <targetPath>/_system/governance</targetPath>
</mount>

    When doing mounting configurations, make note of the following.

    • The dataSource you specify under the <dbConfig name="sharedregistry"> tag must match the jndiConfig name you specified in the master-datasources.xml file.
    • The registry mountpath is used to identify the type of registry. For example, ”/_system/config” refers to configuration registry, and "/_system/governance" refers to the governance registry.
    • The dbconfig entry enables you to identify the datasource you configured in the master-datasources.xml file. We use the unique name sharedregistry to refer to that datasource entry. 
    • The remoteInstance section refers to an external registry mount. We can specify the read-only/read-write nature of this instance as well as caching configurations and the registry root location. In the case of a Message Broker node, the readOnly property must be set to false as we require all broker nodes to have both read and write permissions. The remoteInstance url is used internally in Governance Registry for notification functionality. For all other non-Governance Registry server instances, you can leave this property as it is. If your MB cluster is not configured with a WSO2 Governance Registry, you do not need to change this URL entry for new values. 
    • You must define a unique name “id” for each remote instance, which is then referred to from mount configurations. In the above example, the unique ID for the remote instance is govdb. 

    • In each of the mounting configurations, we specify the actual mount path and target mount path. The targetPath can be any meaningful name. In this instance, it is /_system/mbNodes.

Starting the servers

  • Start the MB servers by executing the <MB_HOME>/bin/wso2server.sh file.
    $ ./wso2server.sh