Page Comparison

The following sections give you information and instructions on how to cluster the message broker profile of WSO2 EI.  

...

1. Configure the datasourcesto point to the REGISTRY_LOCAL1, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as follows in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file as follows:

   Tip
   Replace the username, password, and database URL of your MySQL environment accordingly.

   Code Block
   language xml
   <datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration"> <providers> <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider> </providers> <datasources> <datasource> <name>REGISTRY_LOCAL1</name> <description>The datasource used for registry- local</description> <jndiConfig> <name>jdbc/WSO2CarbonDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL1?autoReconnect=true</url> <username>regadmin</username> <password>regadmin</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource> <datasource> <name>REGISTRY_DB</name> <description>The datasource used for registry- config/governance</description> <jndiConfig> <name>jdbc/WSO2RegistryDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url> <username>regadmin</username> <password>regadmin</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource> <datasource> <name>WSO2_USER_DB</name> <description>The datasource used for registry and user manager</description> <jndiConfig> <name>jdbc/WSO2UMDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url> <username>regadmin</username> <password>regadmin</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource> </datasources> </datasources-configuration>

   Tip

   Repeat this configuration on the second WSO2 EI node to configure the datasourcesto point to the REGISTRY_LOCAL2, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as follows: (Change the username, password, and database URL as needed for your environment): Code Block language xml <datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration"> <providers> <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider> </providers> <datasources> <datasource> <name>REGISTRY_LOCAL2</name> <description>The datasource used for registry- local</description> <jndiConfig> <name>jdbc/WSO2CarbonDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL2?autoReconnect=true</url> <username>regadmin</username> <password>regadmin</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource> <datasource> <name>REGISTRY_DB</name> <description>The datasource used for registry- config/governance</description> <jndiConfig> <name>jdbc/WSO2RegistryDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url> <username>regadmin</username> <password>regadmin</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource> <datasource> <name>WSO2_USER_DB</name> <description>The datasource used for registry and user manager</description> <jndiConfig> <name>jdbc/WSO2UMDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url> <username>regadmin</username> <password>regadmin</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource> </datasources> </datasources-configuration>

2. Add the following configuration in the <EI_HOME>/wso2/broker/conf/user-mgt.xml file to configure the user stores. 

   Tip
   Enter the datasource information for the user store that you configured in the  <EI_HOME>/wso2/broker/conf/datasources /master- datasources . xml file. You can change the admin username and password as well. However, you should do this before starting the server.

   Code Block
   language xml
   <Configuration> <AddAdmin>true</AddAdmin> <AdminRole>admin</AdminRole> <AdminUser> <UserName>admin</UserName> <Password>admin</Password> </AdminUser> <EveryOneRoleName>everyone</EveryOneRoleName> <Property name="dataSource">jdbc/WSO2UMDB</Property> </Configuration>

3. Remove or uncomment the default H2-based WSO2_MB_STORE_DB following configuration in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file. 

   Code Block
   language xml
   <datasource> <name>WSO2_MB_STORE_DB</name> <description>The datasource used for message broker database</description> <jndiConfig> <name>WSO2MBStoreDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:h2:repository/database/WSO2MB_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url> <username>wso2carbon</username> <password>wso2carbon</password> <driverClassName>org.h2.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> <defaultAutoCommit>false</defaultAutoCommit> </configuration> </definition> </datasource>

4. Uncomment or add the following WSO2_MB_STORE_DB configuration in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file based on your DBMS type.

   Tip
   Update the JDBC URL to correctly point to your database and enter the username and password for your database user with the proper permissions.

   Localtabgroup

   Localtab title Configuring MSSQL Code Block language xml <datasource> <name>WSO2_MB_STORE_DB</name> <jndiConfig> <name>jdbc/WSO2MBStoreDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <defaultAutoCommit>false</defaultAutoCommit> <dataSourceClassName>com.microsoft.sqlserver.jdbc.SQLServerXADataSource</dataSourceClassName> <dataSourceProps> <property name="URL">jdbc:sqlserver://127.0.0.1\SQLExpress</property> <property name="databaseName">wso2mb</property> <property name="user">sa</property> <property name="password">mssql</property> </dataSourceProps> </configuration> </definition> </datasource> Localtab title Configuring MySQL Code Block language xml <datasource> <name>WSO2_MB_STORE_DB</name> <jndiConfig> <name>WSO2MBStoreDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <url>jdbc:mysql://localhost/wso2_mb</url> <username>root</username> <password>root</password> <maxActive>50</maxActive> <maxWait>60000</maxWait> <minIdle>5</minIdle> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> <defaultAutoCommit>false</defaultAutoCommit> </configuration> </definition> </datasource> Localtab title Configuring Oracle Code Block language xml  <datasource> <name>WSO2_MB_STORE_DB</name> <jndiConfig> <name>WSO2MBStoreDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <defaultAutoCommit>false</defaultAutoCommit> <driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName> <url>jdbc:oracle:thin:@localhost:1521/orcl</url> <maxActive>100</maxActive> <maxWait>60000</maxWait> <minIdle>5</minIdle> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> <username>scott</username> <password>tiger</password> </configuration> </definition> </datasource>

5. Uncomment or add the following configuration in the <EI_HOME>/wso2/broker/conf/broker.xml file.  (Enter the message store based on your database type and Andes context store).

   Localtabgroup

   Localtab title Configuring MSSQL Code Block ... <persistence> <messageStore class="org.wso2.andes.store.rdbms.RDBMSMessageStoreImpl"> <propertyname="dataSource">jdbc/MSSQLWSO2MBStoreDB</property> </messageStore> <contextStore class="org.wso2.andes.store.rdbms.RDBMSAndesContextStoreImpl"> <propertyname="dataSource">jdbc/MSSQLWSO2MBStoreDB</property> </contextStore> ... </persistence> Localtab title Configuring MySQL Code Block ... <persistence> <messageStore class="org.wso2.andes.store.rdbms.RDBMSMessageStoreImpl"> <propertyname="dataSource">jdbc/MySQLMessageStore</property> </messageStore> <contextStore class="org.wso2.andes.store.rdbms.RDBMSAndesContextStoreImpl"> <propertyname="dataSource">jdbc/MySQLMessageStore</property> </contextStore> ... </persistence> Localtab title Configuring Oracle Code Block ... <persistence> <messageStore class="org.wso2.andes.store.rdbms.RDBMSMessageStoreImpl"> <propertyname="dataSource">WSO2MBStoreDB</property> </messageStore> <contextStore class="org.wso2.andes.store.rdbms.RDBMSAndesContextStoreImpl"> <propertyname="dataSource">WSO2MBStoreDB</property> </contextStore> ... </persistence>

6. Edit the <EI_HOME>/wso2/broker/conf/axis2/axis2.xml file as follows to set up the cluster configurations.

   • Enable clustering for this node as follows: <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
   • Set the membership scheme to "wka" to enable the well-known address registration method. (This node sends cluster initiation messages to the WKA members): 
     <parameter name="membershipScheme">wka</parameter>
   • Specify the name of the cluster this node will join as folows: <parameter name="domain">wso2.ei.domain</parameter>

   • Specify the host to communicate cluster messages as follows: <parameter name="localMemberHost">xxx.xxx.xxx.xx2</parameter>

   • Specify the port to communicate cluster messages as follows: <parameter name="localMemberPort">4100</parameter>

     Info
     This port number is not affected by the port offset value specified in the  <EI_HOME>/wso2/broker/conf/carbon. xml file. If this port number is already assigned to another server, the clustering framework automatically increments this port number. However, if there are two servers running on the same machine, ensure that a unique port is set for each server.

   • Specify the well-known members as follows: (The port value for the WKA node must be the same value as it's localMemberPort (in this case it is 4100).

     Info

     You can also use IP address ranges for the hostname (e.g.,   192.168.1.2-10). However, you can define a range only for the last portion of the IP address. Smaller the range, the faster the time it takes to discover members since each node has to scan a lesser number of potential members. The best practice is to add all the members (including itself) in all the nodes to avoid any conflicts in configurations.

     Code Block
     language xml
     <members> <member> <hostName>xxx.xxx.xxx.xx1</hostName> <port>4100</port> </member> <member> <hostName>xxx.xxx.xxx.xx2</hostName> <port>4100</port> </member> </members>

7. Add the following thrift-related configurations in the <EI_HOME>/wso2/broker/conf/broker.xml file.

   Tip
   This file is the root configuration file of brokering. Do the changes you do to this file in all the broker profile nodes. Configure the thriftServerHost value to point to the IP address of the message broker profile node.

   Code Block
   language xml
   <coordination> <nodeID>default</nodeID> <thriftServerHost>192.168.0.103</thriftServerHost> <thriftServerPort>7611</thriftServerPort> <thriftSOTimeout>0</thriftSOTimeout> </coordination>

   Follwingaredetails on these configurations.

   Configuration	Description
   coordination	This configuration is related to brokering thrift communications.
   nodeID	In a clustered deployment, an ID is assigned to each brokering node via the cluster node identifier. This element can be used to override the cluster node identifier for this brokering node. If the value for this element is left as default, the default node ID is generated using the IP and a universally unique identifier (UUID). The node ID of each member in a cluster must be unique.
   thriftServerHost	This is a sub-element of the <coordination> tag. WSO2 EI uses Apache Thrift for communications relating to message delivery. Therefore, an Apache Thrift server is started in each brokering node in a clustered deployment. This element should point to the IP address of the Apache Thrift server. This should point to the IP address of the brokering node that hosts the thrift server. The default value for this is localhost. For example, if you are configuring a brokering node hosted in 192.168.0.102 as the thrift server, this value should be 192.168.0.102.
   thriftServerPort	This is another sub-element of the <coordination> tag. This should point to the port of the thrift server in the message broker profile. Info title What is the effective thrift server port ? The broker.xml file indicates 7611 as the default thrift server port. However, the default thrift server port that is effective in the MB profile of WSO2 EI is 7614 (7611 + 3). This is because, by default, the server port of the MB profile of WSO2 EI is offset by 3 in the <EI_HOME>/wso2/broker/conf/carbon.xml file. It is important to note that the effective thrift server port depends upon the default server port offset in the carbon.xml file as well as the thriftServerPort value in the broker.xml file. Be sure to keep the correct thrift server port open in your system to avoid errors. It is recommended to use the same thrift server port for all broker nodes in your cluster.
   thriftSOTimeout	This is used to handle half-open TCP connections between the broker nodes in a cluster. In such situations, the socket may need to have a timeout value to invalidate the connection (in milliseconds). A timeout of zero is interpreted as an infinite timeout. Be sure to to set this value to to 180000 milliseconds as shown below in order to handle a high number of half-open TCP connections. Code Block <thriftSOTimeout>180000</thriftSOTimeout>

8. Enable heartbeat messaging for each of the broker nodes by following the steps given below. This is required for handling TCP connections between the broker nodes and client systems (publishers and subscribers).

   Info
   title What is heartbeat messaging?
   When a client is connected to the broker, both the broker and the client should be able to detect problem situations where the TCP connection is half open or where the connecting client/broker is unresponsive. This can be achieved by enabling heartbeat messaging between the broker and the client. The heartbeat messaging configuration allows both the broker and the client to verify whether the connection is inactive or whether the connecting system (broker or client) is inactive by periodically sending messages to the each other. For example, you can set the heartbeat delay time to 30 seconds. This means that if the broker (or the client) does not receive the expected response from the other system within 30 seconds, it will send a heartbeat message to check if the connection is inactive or if the other system is inactive. If the connection is inactive or is half-open, there will be a clear indication of connection failure. However, if the connection is active but the other system is unresponsive, the broker or the client will continue to send heartbeat messages (every 30 seconds). By default, the broker and clients are configured to terminate (timeout) the connection after sending two heartbeat messages; however, this time out configuration can be changed. Note: In a scenario where the subscriber client is sending the heartbeat message to the broker node, if the connection is found to be broken or if the broker node is inactive, the connection will failover to another broker node in the cluster.

   1. Open the qpid-config.xml file that is stored in the <EI_HOME>/wso2/broker/conf/advanced/ directory.

   2. Set the heartbeat delay as shown below. The recommended heartbeat delay is 30 seconds.

      Code Block
      <heartbeat> <delay>30</delay> <timeoutFactor>2.0</timeoutFactor> </heartbeat>

      The elements in the above configuration are explained below.

      delay	The <delay> element specifies the time interval between heartbeat messages. This is the time that the broker or client will wait to receive a response from the other party. If the response is not received within this time, a heartbeat message is triggered.
      timeoutFactor	The number of heartbeat messages the broker will send to the client before terminating the connection. That is, if the timeoutFactor is 2, the broker will send heartbeat messages every 30 seconds twice, and if a response is not received from the client, the connection will be terminated.

...

The Messgae Broker profile in WSO2 EI introduces cluster coordination through an RDBMS. This means that the coordination between the nodes in a cluster can be managed through an RDBMS, just as message persistence. Therefore, the RDBMS that is connected to the MB nodes in the cluster is, by default, used for message persistence as well as cluster coordination. Shown below is the  

Use the following configuration in the broker.xml file xml file (stored in the <EI_HOME>/wso2/broker/conf/ directory) , which enables to enable RDBMS-based cluster coordination. If required, you can disable the following configuration, which will allow the hazelcast engine to manage cluster coordination. However, note that you need to configure the cluster to handle network partitioning when hazelcast-based cluster coordination is used. 

<rdbmsBasedCoordination enabled="true">
    <!-- Heartbeat interval used in the RDBMS base coordination algorithm in milliseconds -->
    <heartbeatInterval>5000</heartbeatInterval>
    <!-- Time to wait before informing others about coordinator change in milliseconds. This value should be
            larger than a database read time including network latency and should be less than heartbeatInterval -->
    <coordinatorEntryCreationWaitTime>3000</coordinatorEntryCreationWaitTime>
    <!-- Time interval used to poll database for membership related events in milliseconds. -->
    <eventPollingInterval>4000</eventPollingInterval>
</rdbmsBasedCoordination>
<!-- Enabling this will make the cluster notifications such as Queue changes(additions and deletions),
        Subscription changes, etc. sent within the cluster be synchronized using RDBMS. If set to false, Hazelcast
        will be used for this purpose.-->
<RDBMSBasedClusterEventSynchronization enabled="true">
    <!--Specifies the interval at which, the cluster events will be read from the database. Needs to be
            declared in milliseconds. Setting this to a very low value could downgrade the performance where as
            setting this to a large value could increase the time taken for a cluster event to be synchronized in
            all the nodes in a cluster.-->
    <eventSyncInterval>1000</eventSyncInterval>
</RDBMSBasedClusterEventSynchronization>

...