Versions Compared

Old Version 29

changes.mady.by.user Former user

Saved on

compared with

New Version Current

changes.mady.by.user Former user

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The following sections give you information and instructions on how to cluster the message broker profile of WSO2 EI.  

Table of Contents
maxLevel4
minLevel3

The clustering deployment pattern

This pattern has two WSO2 EI nodes that include the Message Broker profile to serve service requests with high availability and scalability. The following image depicts the sample pattern of this clustering deployment scenario will follow.

Tip

This pattern uses two nodes as well-known members. It is always recommended to have all nodes of the cluster as well-known members.

Image RemovedImage Added

Warning

When configuring your WSO2 products for clustering to host them in your production environment, it is necessary using Hazelcast, you need to use a specific IP address in your configurations and not localhost or host names in your configurations.

Creating the databases

All profiles of WSO2 EI uses a database to store information such as user management details and registry data. All nodes in the cluster must use one central database for config and governance registry mounts. You can create the following databases and associated datasources.

...

WSO2_USER_DB

...

In addition to the databases listed above, the Message Broker profile of WSO2 EI requires the following broker-specific data:

...

MB_DB

...

Warning

It is recommended to use an industry-standard RDBMS such as Oracle, PostgreSQL, MySQL, MS SQL, etc. for most enterprise testing and production environments. However, you can also use the embedded H2 database only for the REGISTRY_LOCAL1 and REGISTRY_LOCAL2.

Follow the steps given below to create the databases required for the Message Broker profile of WSO2 EI. 

...

Download and install the MySQL Server.

...

Download the MySQL JDBC driver.

...

Download and unzip the WSO2 EI binary distribution. 

Tip

Throughout this guide, <EI_HOME> refers to the extracted directory of the WSO2 EI product distribution.

...

Add the following line to the /etc/hosts file to define the hostname for configuring permissions for the new database: <MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com

Note

Do this step only if your database is not on your local machine and on a separate server.

...

Specify the password to access the databases with the username you specified when prompted.

...

Create the databases using the following commands:

Code Block
languagesql
mysql> create database WSO2_USER_DB;
mysql> use WSO2_USER_DB;
mysql> source <EI_HOME>/dbscripts/mysql.sql;
mysql> grant all on WSO2_USER_DB.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";

mysql> create database REGISTRY_DB;
mysql> use REGISTRY_DB;
mysql> source <EI_HOME>/dbscripts/mysql.sql;
mysql> grant all on REGISTRY_DB.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";

mysql> create database REGISTRY_LOCAL1;
mysql> use REGISTRY_LOCAL1;
mysql> source <EI_HOME>/dbscripts/mysql.sql;
mysql> grant all on REGISTRY_LOCAL1.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";
 
mysql> create database REGISTRY_LOCAL2;
mysql> use REGISTRY_LOCAL2;
mysql> source <EI_HOME>/dbscripts/mysql.sql;
mysql> grant all on REGISTRY_LOCAL2.* TO regadmin@"carbondb.mysql-wso2.com" identified 
by "regadmin";
Info
titleAbout using MySQL in different operating systems

For users of Microsoft Windows, when creating the database in MySQL, it is important to specify the character set as latin1. Failure to do this may result in an error (error code: 1709) when starting your cluster. This error occurs in certain versions of MySQL (5.6.x) and is related to the UTF-8 encoding. MySQL originally used the latin1 character set by default, which stored characters in a 2-byte sequence. However, in recent versions, MySQL defaults to UTF-8 to be friendlier to international users. Hence, you must use latin1 as the character set as indicated below in the database creation commands to avoid this problem. Note that this may result in issues with non-latin characters (like Hebrew, Japanese, etc.). The following is how your database creation command should look.

mysql> create database <DATABASE_NAME> character set latin1;

For users of other operating systems, the standard database creation commands will suffice. For these operating systems, the following is how your database creation command should look.

mysql> create database <DATABASE_NAME>;

Create and configure the MB_DB database, which is specific to the message broker profile using the following commands:

Tip

<EI_HOME> , username and password are the same as those you specified in the previous steps.

...

languagesql

...

.

Tip
  • In this guide, the IP of the first Message Broker (MB) node is referred to as https://xxx.xxx.xxx.xx1 and the IP of the second MB node is referred to as https://xxx.xxx.xxx.xx2.

  • If you want to test out clustering in a development environment using the same server, you need to port offset one of your MB nodes.
    Specify the port offset value in the <EI_HOME>/wso2/broker/conf/carbon.xml file.

    Note

    This is not recommended for production environments. Change all ports used in your configurations based on the offset value if you are setting a port offset for the specific node.

    For more information on the default ports of WSO2 Enterprise Integrator, see the WSO2 administration guide.

    Expand
    titleClick here for more information on configuring the port offset.

    When you run multiple products/clusters or multiple instances of the same product on the same server or virtual machines (VMs), change their default ports with an offset value to avoid port conflicts. An offset defines the number by which all ports in the runtime (e.g., HTTP(S) ports) are increased. For example, if the default HTTP port is 9763 and the offset is 1, the effective HTTP port will change to 9764. For each additional product instance, set the port offset to a unique value. The offset of the default ports is zero.

    The port value will automatically increase as shown in the Port Value column in the following table, allowing five WSO2 product instances or servers to run on the same machine.

    WSO2 product instance

    Port Offset

    Port Value

    WSO2 server 1

    0

    9443

    WSO2 server 2

    1

    9444

    WSO2 server 3

    2

    9445

    WSO2 server 4

    3

    9446

    WSO2 server 5

    4

    9447

    Example:

    Code Block
    languagexml
    <Ports>
	...
	<Offset>5</Offset>
	...
</Ports>

Creating the databases

All profiles of WSO2 EI uses a database to store information such as user management details and registry data. All nodes in the cluster must use one central database for config and governance registry mounts. You can create the following databases and associated datasources.

Database NameDescription
WSO2_USER_DB
JDBC user store and authorization manager.

REGISTRY_DB

Shared database for config and governance registry mounts in the product's nodes.
REGISTRY_LOCAL1Local registry space in Node 1.
REGISTRY_LOCAL2Local registry space in Node 2.

In addition to the databases listed above, the MB profile of WSO2 EI requires the following broker-specific data:

Database NameDescription
MB_DB
Stores instance data that are specific to the message broker profile.
Warning

It is recommended to use an industry-standard RDBMS such as Oracle, PostgreSQL, MySQL, MS SQL, etc. for most enterprise testing and production environments. However, you can also use the embedded H2 database only for the REGISTRY_LOCAL1 and REGISTRY_LOCAL2.

Follow the steps given below to create the databases required for the MB profile of WSO2 EI. 

  1. Download and install the MySQL Server.

  2. Download the MySQL JDBC driver.

  3. Download and unzip the WSO2 EI binary distribution. Be sure to select the correct product version. 

    Tip

    Throughout this guide, <EI_HOME> refers to the extracted directory of the WSO2 EI product distribution.

  4. Unzip the downloaded MySQL driver, and copy the MySQL JDBC driver JAR (mysql-connector-java-x.x.xx-bin.jar) to the <EI_HOME>/lib/ directory of both the WSO2 EI nodes.

  5. Add the following line to the /etc/hosts file to define the hostname for configuring permissions for the new database: <MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com

    Note

    Do this step only if your database is not on your local machine and on a separate server.

  6. Execute the following command in a terminal/command window, where username is the username you want to use to access the databases: mysql -u username -p

  7. When prompted, specify the password to access the databases.

  8. Create the databases using the following commands:

    Code Block
    languagesql
    mysql> create database WSO2_USER_DB;
mysql> use WSO2_USER_DB;
mysql> source <EI_HOME>/dbscripts/mysql.sql;
mysql> grant all on WSO2_USER_DB.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";

mysql> create database REGISTRY_DB;
mysql> use REGISTRY_DB;
mysql> source <EI_HOME>/dbscripts/mysql.sql;
mysql> grant all on REGISTRY_DB.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";

mysql> create database REGISTRY_LOCAL1;
mysql> use REGISTRY_LOCAL1;
mysql> source <EI_HOME>/dbscripts/mysql.sql;
mysql> grant all on REGISTRY_LOCAL1.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";
 
mysql> create database REGISTRY_LOCAL2;
mysql> use REGISTRY_LOCAL2;
mysql> source <EI_HOME>/dbscripts/mysql.sql;
mysql> grant all on wso2REGISTRY_mbLOCAL2.* TO username@localhostregadmin@"carbondb.mysql-wso2.com" identified 
by "passwordregadmin";

Mounting the registry

Add the following configurations to the <EI_HOME>/wso2/broker/conf/registry.xml file of each WSO2 EI node to configure the shared registry database and mounting details. This ensures that the shared registry for governance and configurations (i.e., the REGISTRY_DB database) mounts on both WSO2 EI nodes. 

...

Note the following when adding these configurations:

...

The registry mount path denotes the type of registry. For example, ”/_system/config” refers to configuration Registry, and "/_system/governance" refers to the governance registry.

...

  1.  Info
    titleAbout using MySQL in different operating systems
    For users of Microsoft Windows, when creating the database in MySQL, it is important to specify the character set as latin1. Failure to do this may result in an error (error code: 1709) when starting your cluster. This error occurs in certain versions of MySQL (5.6.x) and is related to the UTF-8 encoding. MySQL originally used the latin1 character set by default, which stored characters in a 2-byte sequence. However, in recent versions, MySQL defaults to UTF-8 to be friendlier to international users. Hence, you must use latin1 as the character set as indicated below in the database creation commands to avoid this problem. Note that this may result in issues with non-latin characters (like Hebrew, Japanese, etc.). The following is how your database creation command should look.
    mysql> create database <DATABASE_NAME> character set latin1;
    For users of other operating systems, the standard database creation commands will suffice. For these operating systems, the following is how your database creation command should look.
    mysql> create database <DATABASE_NAME>;
  2. Create and configure the MB_DB database, which is specific to the MB profile:
     Tip
    <EI_HOME> , username, and password are the same as those you specified in the previous steps.
     Code Block
    languagesql
    mysql> create database wso2_mb;
mysql> use wso2_mb;
mysql> source <EI_HOME>/wso2/broker/
...
  1. dbscripts/
...
  1. mb-store/
...
  1. mysql-
...
Also, specify the cache ID in the <remoteInstance> section. This enables caching to function properly in the clustered environment.
 Note
Cache ID is the same as the JDBC connection URL of the registry database. This value is the Cache ID of the remote instance. It should be in the format of $database_username@$database_url, where $database_username is the username of the remote instance database and $database_url is the remote instance database URL. This cacheID denotes the enabled cache. In this case, the database it should connect to is REGISTRY_DB, which is the database shared across all the nodes. You can find that in the mounting configurations of the same datasource that is being used.
...
Define a unique name in the <id> tag for each remote instance. This is then referred to from mount configurations. In the above example, the unique ID for the remote instance is "instanceId".
...
Specify the actual mount path and target mount path in each of the mounting configurations. The target path can be any meaningful name. In this instance, it is "/_system/mbconfig".
 Code Block
languagexml
<dbConfig name="sharedregistry">
<dataSource>jdbc/WSO2RegistryDB</dataSource>
</dbConfig> 

<remoteInstance url="https://localhost:9443/registry">
    <id>instanceid</id>
    <dbConfig>sharedregistry</dbConfig>
    <readOnly>false</readOnly>
    <enableCache>true</enableCache>
    <registryRoot>/</registryRoot>
<cacheId>regadmin@jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</cacheId>
</remoteInstance>
 
<mount path="/_system/config" overwrite="true">
    <instanceId>instanceid</instanceId>
    <targetPath>/_system/mbconfig</targetPath>
</mount>
 
<mount path="/_system/governance" overwrite="true">
    <instanceId>instanceid</instanceId>
    <targetPath>/_system/governance</targetPath>
</mount>
Configuring a message broker profile node
 Warning
It is vital to synchronize time across all the Broker nodes in the cluster to make them function properly.
Do the following configurations for all nodes of your cluster.
  1. Configure the datasourcestopoint to the REGISTRY_LOCAL1, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as follows in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file as follows:
     Tip
    Replace the username, password, and database URL of your MySQL environment accordingly.
     Code Block
    languagexml
    <datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration"> 
     <providers> 
        <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider> 
    </providers> 
    <datasources> 
        <datasource> 
            <name>REGISTRY_LOCAL1</name> 
            <description>The datasource used for registry- local</description> 
            <jndiConfig> 
                <name>jdbc/WSO2CarbonDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL1?autoReconnect=true</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
        <datasource> 
            <name>REGISTRY_DB</name> 
            <description>The datasource used for registry- config/governance</description>mb.sql;
mysql> grant all on wso2_mb.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";
Mounting the registry
Add the following configurations to the <EI_HOME>/wso2/broker/conf/registry.xml file of each WSO2 EI node to configure the shared registry database and mounting details. This ensures that the shared registry for governance and configurations (i.e., the REGISTRY_DB database) mounts on both WSO2 EI nodes. 
 Note
Note the following when adding these configurations:
  • The existing dbConfig called wso2registry must not be removed.
  • The datasource you specify in the <dbConfig name="sharedregistry"> tag must match the JNDI Config name you specify in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file.
  • The registry mount path denotes the type of registry. For example, ”/_system/config” refers to configuration Registry, and "/_system/governance" refers to the governance registry.
  • The <dbconfig> entry enables you to identify the datasource you configured in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file. The unique name "shared registry" refers to that datasource entry.
  • The <remoteInstance> section refers to an external registry mount. Specify the read-only/read-write nature of this instance, caching configurations and the registry root location in this section.
  • Also, specify the cache ID in the <remoteInstance> section. This enables caching to function properly in the clustered environment.
     Info
    Cache ID is the same as the JDBC connection URL of the registry database. This value is the Cache ID of the remote instance. It should be in the format of $database_username@$database_url, where $database_username is the username of the remote instance database and $database_url is the remote instance database URL. This cache ID denotes the enabled cache. In this case, the database it should connect to is REGISTRY_DB, which is the database shared across all the nodes. You can find that in the mounting configurations of the same datasource that is being used.
  • Define a unique name in the <id> tag for each remote instance. This is then referred to from mount configurations. In the above example, the unique ID for the remote instance is "instanceId".
  • Specify the actual mount path and target mount path in each of the mounting configurations. The target path can be any meaningful name. In this instance, it is "/_system/mbconfig".
 Code Block
languagexml
<dbConfig name="sharedregistry">
<dataSource>jdbc/WSO2RegistryDB</dataSource>
</dbConfig> 

<remoteInstance url="https://localhost:9443/registry">
    <id>instanceid</id>
    <dbConfig>sharedregistry</dbConfig>
    <readOnly>false</readOnly>
    <enableCache>true</enableCache>
    <registryRoot>/</registryRoot>
<cacheId>regadmin@jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</cacheId>
</remoteInstance>
 
<mount path="/_system/config" overwrite="true">
    <instanceId>instanceid</instanceId>
    <targetPath>/_system/mbconfig</targetPath>
</mount>
 
<mount path="/_system/governance" overwrite="true">
    <instanceId>instanceid</instanceId>
    <targetPath>/_system/governance</targetPath>
</mount>
Configuring a message broker profile node
 Warning
titleTime synchronising the broker nodes
 Expand
The RDBMS identifies the latest record based on timestamps so a newer timestamp always gets preference. Therefore, it is vital to synchronize time across all the MB nodes in the cluster to make them function properly. Given below are instructions for time synchronizing on Linux servers. Note that you need administrator permissions to do this.
  1. Install the ntpdate package by executing the following commands.
    • For RedHat Enterprise Linux, CentOS, and Oracle Linux:
       Code Block
      $sudo apt-get install ntpdate;
    • For Debian and Ubuntu:
       Code Block
      $ntpdate pool.ntp.org
  2. Execute the command given below to create a script named timesync.sh in the /home/user directory.
     Code Block
    sudo ntpdate pool.ntp.org >>  /tmp/timeSyncLog.log
    This command will sync the time of the local machine with the time in the pool.ntp.org time server cluster. The  pool.ntp.org  project is a big virtual cluster of time servers providing reliable easy-to-use NTP services for millions of clients. It's the default "time server" for most of the major Linux distributions, and many networked appliances. The logs of the time synchronization will be routed to the /tmp/timeSyncLog.log file.
  3. Now, you need to schedule a cron job. Every computer calculates time locally according to a clock. This may cause a considerable time gap between servers in a clustered setup. Therefore, to make sure that the time on each of the servers are synchronized, we need to run the above script from time to time in every server machine. All servers will sync to the time received by "pool.ntp.org". 
    1. First, execute the following command:
       Code Block
      $ crontab -e
    2. Give the cron expression. For example, if you want time synchronization every minute, use the following cron expression:
       Code Block
      * * * * * sh /home/user/timesync.sh
Do the following configurations for all nodes of your cluster.
  1. Configure the datasources to point to the REGISTRY_LOCAL1, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as follows in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file as follows:
     Tip
    Replace the username, password, and database URL of your MySQL environment accordingly.
     Localtabgroup
     Localtab
    activetrue
    titleMB Node 1
    For MB node 1, configure the datasources to point to the REGISTRY_LOCAL1, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as follows:
     Code Block
    languagexml
    <datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration"> 
     <providers> 
        <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider> 
    </providers> 
    <datasources> 
        <datasource> 
            <name>REGISTRY_LOCAL1</name> 
            <description>The datasource used for registry- local</description> 
            <jndiConfig> 
                <name>jdbc/WSO2CarbonDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL1?autoReconnect=true</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
  
    <jndiConfig>
                      
    <name>jdbc/WSO2RegistryDB</name>
    <maxActive>50</maxActive> 
                    
    </jndiConfig>
    <maxWait>60000</maxWait> 
                    
    <definition type="RDBMS">
    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 
    <configuration>
    1</validationQuery> 
                    
    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url>
    <validationInterval>30000</validationInterval> 
                </configuration> 
          
    <username>regadmin</username>
      </definition> 
        </datasource> 
        <datasource> 
  
    <password>regadmin</password>
              <name>REGISTRY_DB</name> 
            <description>The 
     <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
 
    datasource used for registry- config/governance</description> 
            <jndiConfig> 
   
    <maxActive>50</maxActive>
                 <name>jdbc/WSO2RegistryDB</name> 
       
    <maxWait>60000</maxWait>
         </jndiConfig> 
            <definition type="RDBMS"> 

    <testOnBorrow>true</testOnBorrow>
                    <configuration> 
    
    <validationQuery>SELECT
     
    1</validationQuery>
                   <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url> 
     
    <validationInterval>30000</validationInterval>
                   
       </configuration> 
    <username>regadmin</username> 
           
    </definition>
             
     <
    <password>regadmin</
    datasource>
    password> 
         
    <datasource>
               
       <name>WSO2_USER_DB</name>
    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
             
    <description>The
     
    datasource
     
    used
     
    for
     
    registry
     
    and
     
    user
     
    manager<
    <maxActive>50</
    description>
    maxActive> 
            
    <jndiConfig>
            <maxWait>60000</maxWait> 
        
    <name>jdbc/WSO2UMDB</name>
                
      <
    <testOnBorrow>true</
    jndiConfig>
    testOnBorrow> 
            
    <definition
     
    type="RDBMS">
           <validationQuery>SELECT 1</validationQuery> 
        
    <configuration>
                <validationInterval>30000</validationInterval> 
   
         <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url>
                 </configuration> 
            
    <username>regadmin<
    </
    username>
    definition> 
        </datasource> 
         <datasource> 
    <password>regadmin</password>
    
            <name>WSO2_USER_DB</name> 
       
    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
         <description>The datasource used for registry and user manager</description> 
        
    <maxActive>50</maxActive>
        <jndiConfig> 
                
    <maxWait>60000</maxWait>
    <name>jdbc/WSO2UMDB</name> 
            </jndiConfig> 
           
    <testOnBorrow>true</testOnBorrow>
     <definition type="RDBMS"> 
                <configuration> 
  
    <validationQuery>SELECT
     
    1</validationQuery>
                     <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url> 
   
    <validationInterval>30000</validationInterval>
                     
     </configuration> 
            </definition>
    <username>regadmin</username> 
          
    </datasource>
         
    </datasources> 
</datasources-configuration>
     Tip
    Repeat this configuration on the second WSO2 EI node to configure the datasourcestopoint to the REGISTRY_LOCAL2, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as follows: (Change the username, password, and database URL as needed for your environment):
     Code Block
    languagexml
    <datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration">
         <password>regadmin</password> 
           
    <providers>
             
    <provider>org
    <driverClassName>com.
    wso2
    mysql.
    carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider>
    jdbc.Driver</driverClassName> 
               
    </providers>
         
    <datasources>
    <maxActive>50</maxActive> 
       
    <datasource>
                 
    <name>REGISTRY_LOCAL2<
    <maxWait>60000</
    name>
    maxWait> 
           
    <description>The
     
    datasource
     
    used
     
    for
     
    registry-
     
    local</description>
        <testOnBorrow>true</testOnBorrow> 
       
    <jndiConfig>
                 <validationQuery>SELECT 1</validationQuery> 
 
    <name>jdbc/WSO2CarbonDB</name>
                 
    </jndiConfig>
          <validationInterval>30000</validationInterval> 
     
    <definition
     
    type="RDBMS">
              </configuration> 
     
    <configuration>
           </definition> 
        </datasource> 
   
    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL2?autoReconnect=true</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
    </datasources> 
</datasources-configuration>
     Code Block
    languagexml
    <Configuration>
     Localtab
    titleMB Node 2
    For node 2, configure the datasources to point to the REGISTRY_LOCAL2,  WSO2_REGISTRY_DB, and WSO2_USER_DB  databases as shown below. Change the username, password, and database URL as needed for your environment.
     Code Block
    languagexml
    <datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration"> 
     <providers> 
        
        <driverClassName>com.mysql.jdbc.Driver</driverClassName>
    <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider> 
    </providers> 
    <datasources> 
       
    <maxActive>50</maxActive>
     <datasource> 
            
          <maxWait>60000</maxWait>
    <name>REGISTRY_LOCAL2</name> 
            <description>The datasource 
      
    used for registry- 
    <testOnBorrow>true<
    local</
    testOnBorrow>
    description> 
            <jndiConfig> 
     
    <validationQuery>SELECT
     
    1</validationQuery>
              <name>jdbc/WSO2CarbonDB</name> 
         
    <validationInterval>30000</validationInterval>
       </jndiConfig> 
            
    </configuration>

    <definition type="RDBMS"> 
          
    </definition>
          <configuration> 
 
    </datasource>
             
    <datasource>
              
       <name>REGISTRY_DB</name>
    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL1?autoReconnect=true</url> 
            
    <description>The
     
    datasource
     
    used
     
    for
     
    registry-
     
    config/governance</description>
       <username>regadmin</username> 
        
    <jndiConfig>
                <password>regadmin</password> 
   
    <name>jdbc/WSO2RegistryDB</name>
                 
    </jndiConfig>
        <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
       
    <definition
     
    type="RDBMS">
                <maxActive>50</maxActive> 
   
    <configuration>
                     <maxWait>60000</maxWait> 
  
    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url>
                      <testOnBorrow>true</testOnBorrow> 
 
    <username>regadmin</username>
                       <validationQuery>SELECT 1</validationQuery> 
    <password>regadmin</password>
    
                    
    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
    <validationInterval>30000</validationInterval> 
                </configuration> 
 
    <maxActive>50</maxActive>
               </definition> 
        
    <maxWait>60000<
    </
    maxWait>
    datasource> 
        <datasource> 
         
    <testOnBorrow>true</testOnBorrow>
       <name>REGISTRY_DB</name> 
            <description>The datasource used 
     <validationQuery>SELECT 1</validationQuery>
     
    for registry- config/governance</description> 
            <jndiConfig> 
    <validationInterval>30000</validationInterval>
    
                
    </configuration>
    <name>jdbc/WSO2RegistryDB</name> 
            </
    definition>
    jndiConfig> 
       
    </datasource>
         <definition type="RDBMS"> 
  
    <datasource>
                 
    <name>WSO2_USER_DB</name>
     <configuration> 
          
    <description>The
     
    datasource
     
    used
     
    for
     
    registry
     
    and
     
    user
     
    manager</description>
       <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url> 
        
    <jndiConfig>
                <username>regadmin</username> 
   
    <name>jdbc/WSO2UMDB</name>
                 
    </jndiConfig>
        <password>regadmin</password> 
       
    <definition
     
    type="RDBMS">
                <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
   
    <configuration>
                     <maxActive>50</maxActive> 
  
    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url>
                      <maxWait>60000</maxWait> 
 
    <username>regadmin</username>
                       <testOnBorrow>true</testOnBorrow> 

    <password>regadmin</password>
                        
     <driverClassName>com.mysql.jdbc.Driver</driverClassName>
    <validationQuery>SELECT 1</validationQuery> 
                    
    <maxActive>50<
    <validationInterval>30000</
    maxActive>
    validationInterval> 
                </configuration> 
 
    <maxWait>60000</maxWait>
               </definition> 
        
    <testOnBorrow>true<
    </
    testOnBorrow>
    datasource> 
         <datasource> 
          
    <validationQuery>SELECT 1</validationQuery>
      <name>WSO2_USER_DB</name> 
            <description>The datasource used for registry and user 
    <validationInterval>30000<
    manager</
    validationInterval>
    description> 
            <jndiConfig> 
   
    </configuration>
                 
    </definition>
    <name>jdbc/WSO2UMDB</name> 
        
    </datasource>
        </
    datasources>
</datasources-configuration>
    Add the following configuration in the <EI_HOME>/wso2/broker/conf/user-mgt.xml file to configure the user stores. 
     Tip
    Enter the datasource information for the user store that you configured in the  <EI_HOME>/wso2/broker/conf/ datasources /master- datasources . xml file. You can change the admin username and password as well. However, you should do this before starting the server.
    jndiConfig> 
         
    <AddAdmin>true</AddAdmin>
       <definition 
     <AdminRole>admin</AdminRole>
    type="RDBMS"> 
    
    <AdminUser>
             
    <UserName>admin</UserName>
       <configuration> 
    
    <Password>admin</Password>
         
    </AdminUser>
         
    <EveryOneRoleName>everyone</EveryOneRoleName>
         
    <Property name="dataSource">jdbc/WSO2UMDB</Property>
</Configuration>
    Remove or comment out the default H2-based WSO2_MB_STORE_DB following configuration in the  <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file. 
     Code Block
    languagexml
    <datasource>
     <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url> 
         
    <name>WSO2_MB_STORE_DB</name>
         
    <description>The
     
    datasource
     
    used
     
    for
     
    message
     
    broker
     
    database</description>
    <username>regadmin</username> 
           
    <jndiConfig>
             
    <name>WSO2MBStoreDB</name>
    <password>regadmin</password> 
              
    </jndiConfig>
         
    <definition type="RDBMS">
     <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
       
    <configuration>
                 
    <url>jdbc:h2:repository/database/WSO2MB_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url>
    <maxActive>50</maxActive> 
             
    <username>wso2carbon</username>
           <maxWait>60000</maxWait> 
    
    <password>wso2carbon</password>
                 
    <driverClassName>org.h2.Driver</driverClassName>
       <testOnBorrow>true</testOnBorrow> 
        
    <maxActive>50</maxActive>
                <validationQuery>SELECT 
    <maxWait>60000<
    1</
    maxWait>
    validationQuery> 
           
    <testOnBorrow>true</testOnBorrow>
             <validationInterval>30000</validationInterval> 
  
    <validationQuery>SELECT
     
    1</validationQuery>
                 
    <validationInterval>30000<
    </
    validationInterval>
    configuration> 
            
    <defaultAutoCommit>false<
    </
    defaultAutoCommit>
    definition> 
        </
    configuration>
    datasource> 
   </
    definition>
    datasources> 
</
    datasource>
    datasources-configuration>
  2. Remove or comment out the default H2-based WSO2_MB_STORE_DB and WSO2_CARBON_DB datasource configurations from the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file. 
  3. Uncomment or add the following the following WSO2_MB_STORE_DB configuration in the  configuration in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file based  file based on your DBMS type.
     Tip
    Update the JDBC URL to correctly point to your database and enter the username and password for your database user with the proper permissions.
     Localtabgroup
    <datasource>
    <name>WSO2_MB_STORE_DB</name>
     Localtab
    titleConfiguring MSSQL
     Code Block
    languagexml
     Code Block
    languagexml
    <datasource>
    <name>WSO2_MB_STORE_DB</name>
    <jndiConfig>
        <name>jdbc/WSO2MBStoreDB</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <defaultAutoCommit>false</defaultAutoCommit>
            <dataSourceClassName>com.microsoft.sqlserver.jdbc.SQLServerXADataSource</dataSourceClassName>
            <dataSourceProps>
       <jndiConfig>         <name>jdbc/WSO2MBStoreDB</name><property name="URL">jdbc:sqlserver://127.0.0.1\SQLExpress</property>
           </jndiConfig>     <definition<property typename="RDBMS">databaseName">wso2mb</property>
           <configuration>     <property name="user">sa</property>
      <defaultAutoCommit>false</defaultAutoCommit>          <property name="password">mssql</property>
 <dataSourceClassName>com.microsoft.sqlserver.jdbc.SQLServerXADataSource</dataSourceClassName>           </dataSourceProps>
 <dataSourceProps>       </configuration>
    </definition>
</datasource>
     Localtab
    titleConfiguring MySQL
     Note
    If your MySQL username or password is not root, make sure to update the <username> and <password> properties.
     Code Block
    languagexml
    <datasource>
   <property name="URL">jdbc:sqlserver://127.0.0.1\SQLExpress</property> <name>WSO2_MB_STORE_DB</name>
    <jndiConfig>
        <name>WSO2MBStoreDB</name>
  <property name="databaseName">wso2mb</property>  </jndiConfig>
    <definition type="RDBMS">
        <configuration>
 <property name="user">sa</property>          <driverClassName>com.mysql.jdbc.Driver</driverClassName>
      <property name="password">mssql</property>     <url>jdbc:mysql://localhost/wso2_mb</url>
       </dataSourceProps>     <username>regadmin</username>
   </configuration>     </definition> </datasource>
     Localtab
    titleConfiguring MySQL
     Code Block
    languagexml
    <datasource>   <password>regadmin</password>
 <name>WSO2_MB_STORE_DB</name>     <jndiConfig>      <maxActive>50</maxActive>
  <name>WSO2MBStoreDB</name>        </jndiConfig>  <maxWait>60000</maxWait>
  <definition type="RDBMS">         <configuration><minIdle>5</minIdle>
            <driverClassName>com.mysql.jdbc.Driver</driverClassName><testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT <url>jdbc:mysql://localhost/wso2_mb</url>1</validationQuery>
            <username>root<<validationInterval>30000</username>validationInterval>
            <password>root<<defaultAutoCommit>false</password>defaultAutoCommit>
        </configuration>
    <maxActive>50<</maxActive>
definition>
</datasource>
     Localtab
    titleConfiguring Oracle
     Code Block
    languagexml
     <datasource>
    <name>WSO2_MB_STORE_DB</name>
     <jndiConfig>
<maxWait>60000</maxWait>        <name>WSO2MBStoreDB</name>
    <minIdle>5<</minIdle>jndiConfig>
    <definition type="RDBMS">
       <testOnBorrow>true</testOnBorrow> <configuration>
           <validationQuery>SELECT 1<<defaultAutoCommit>false</validationQuery>defaultAutoCommit>
            <validationInterval>30000</validationInterval><driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
            <defaultAutoCommit>false</defaultAutoCommit><url>jdbc:oracle:thin:@localhost:1521/orcl</url>
        </configuration>    <maxActive>100</maxActive>
</definition> </datasource>
     Localtab
    titleConfiguring Oracle
     Code Block
    languagexml
     <datasource>     <name>WSO2_MB_STORE_DB</name>     <jndiConfig> <maxWait>60000</maxWait>
       <name>WSO2MBStoreDB</name>     <<minIdle>5</jndiConfig>
 minIdle>
  <definition type="RDBMS">         <configuration><testOnBorrow>true</testOnBorrow>
            <defaultAutoCommit>false<<validationQuery>SELECT 1</defaultAutoCommit>validationQuery>
            <driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName><validationInterval>30000</validationInterval>
            <url>jdbc:oracle:thin:@localhost:1521/orcl</url><username>scott</username>
            <maxActive>100<<password>tiger</maxActive>password>
            <maxWait>60000</maxWait>
            <minIdle>5</minIdle>
            <testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
            <username>scott</username>
   </configuration>
    </definition>
</datasource>
  4. Add the following configuration in the <EI_HOME>/wso2/broker/conf/user-mgt.xml file of both nodes to configure the user stores. 
     Tip
    Enter the datasource information for the user store that you configured in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file. You can change the admin username and password as well. However, you should do this before starting the server.
     Code Block
    languagexml
    <Configuration>
    <AddAdmin>true</AddAdmin>
    <AdminRole>admin</AdminRole>
    <AdminUser>
        <UserName>admin</UserName>
        
    <password>tiger<
    <Password>admin</
    password>
    Password>
    </AdminUser>
   
    <
     <EveryOneRoleName>everyone</
    configuration>
    EveryOneRoleName>
    
     </definition>
    <Property name="dataSource">jdbc/WSO2UMDB</Property>
</
    datasource>
    Configuration>
  5. Uncomment or add the following configuration in in the <EI_HOME>/wso2/broker/conf/broker.xml file.  ( Enter the message store based on your database type and Andes context store).
     Localtabgroup
     Localtab
    titleConfiguring MSSQL
     Code Block
    ...
<persistence>
  <messageStore class="org.wso2.andes.store.rdbms.RDBMSMessageStoreImpl">
        <propertyname="dataSource">jdbc/MSSQLWSO2MBStoreDB<>WSO2MBStoreDB</property>
  </messageStore>
  <contextStore class="org.wso2.andes.store.rdbms.RDBMSAndesContextStoreImpl">
        <propertyname="dataSource">jdbc>WSO2MBStoreDB</MSSQLWSO2MBStoreDB</property>
  </contextStore>
...
</persistence>
     Localtab
    titleConfiguring MySQL
     Code Block
    ...
<persistence>
  <messageStore class="org.wso2.andes.store.rdbms.RDBMSMessageStoreImpl">
        <propertyname="dataSource">jdbc>WSO2MBStoreDB</MySQLMessageStore</property>
  </messageStore>
  <contextStore class="org.wso2.andes.store.rdbms.RDBMSAndesContextStoreImpl">
        <propertyname="dataSource">jdbc>WSO2MBStoreDB</MySQLMessageStore</property>
  </contextStore>
...
</persistence>
     Localtab
    titleConfiguring Oracle
     Code Block
    ...
<persistence>
  <messageStore class="org...
<persistence>wso2.andes.store.rdbms.RDBMSMessageStoreImpl">
        <propertyname="dataSource">WSO2MBStoreDB</property>
  </messageStore>
  <messageStore<contextStore class="org.wso2.andes.store.rdbms.RDBMSMessageStoreImplRDBMSAndesContextStoreImpl">
        <propertyname="dataSource">WSO2MBStoreDB</property>
  </messageStore>contextStore>
...
</persistence>
<contextStore class="org.wso2.andes.store.rdbms.RDBMSAndesContextStoreImpl">
  

  6. Add the following thrift-related configurations in the <EI_HOME>/wso2/broker/conf/broker.xml file.
     Tip
    This file is the root configuration file of brokering. Do the changes you do to this file in all the broker profile nodes. Configure the thriftServerHost value to point to the IP address of the MB profile node.
    Add the following thrift-related configurations in the 
     Code Block
    languagexml
    <coordination>
    
     <propertyname="dataSource">WSO2MBStoreDB</property>
    <nodeID>default</nodeID>
   
    </contextStore>
     <thriftServerHost>192.168.0.
    
</persistence>


  7. Edit the <EI_HOME>/wso2/broker/conf/axis2/axis2.xml file as follows to set up the cluster configurations.
    • Enable clustering for this node as follows: <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
    • Set the membership scheme to "wka" to enable the well-known address registration method. (This node sends cluster initiation messages to the WKA members):  
        <parameter name="membershipScheme">wka</parameter> 
    • Specify the name of the cluster this node will join as follows: <parameter name="domain">wso2.ei.domain</parameter>  
       
    • Specify the host to communicate cluster messages as follows: <parameter name="localMemberHost">xxx.xxx.xxx.xx2</parameter>
    • Specify the port to communicate cluster messages as follows: <parameter name="localMemberPort">4100</parameter>
       Info
      This port number is not affected by the port offset value specified in the  <EI_HOME>/wso2/broker/conf/carbon.  xml file. If this port number is already assigned to another server, the clustering framework automatically increments this port number. However, if there are two servers running on the same machine, ensure that a unique port is set for each server.
    • Specify the well-known members as follows: (The port value for the WKA node must be the same value as it's localMemberPort (in this case it is 4100).
       Info
      You can also use IP address ranges for the hostname (e.g.,   192.168.1.2-10). However, you can define a range only for the last portion of the IP address. Smaller the range, the faster the time it takes to discover members since each node has to scan a lesser number of potential members. The best practice is to add all the members (including itself) in all the nodes to avoid any conflicts in configurations.
       Code Block
      languagexml
      <members>    
    <member>
        <hostName>xxx.xxx.xxx.xx1</hostName>
        <port>4100</port>
    </member>
    <member>
        <hostName>xxx.xxx.xxx.xx2</hostName>
        <port>4100</port>
    </member>
</members>
  8. 103</thriftServerHost>
    <thriftServerPort>7611</thriftServerPort>
	<thriftSOTimeout>0</thriftSOTimeout>
</coordination>
    ConfigurationDescription
    coordination
    This configuration is related to brokering thrift communications.
    nodeIDIn a clustered deployment, an ID is assigned to each brokering node via the cluster node identifier. This element can be used to override the cluster node identifier for this brokering node. If the value for this element is left as default, the default node ID is generated using the IP and a universally unique identifier (UUID). The node ID of each member in a cluster must be unique.
    thriftServerHost
    This is a sub-element of the <coordination> tag. The MB profile uses Apache Thrift for communications relating to message delivery. Therefore, an Apache Thrift server is started in each brokering node in a clustered deployment. This element should point to the IP address of the Apache Thrift server. This should point to the IP address of the brokering node that hosts the thrift server. The default value for this is localhost. For example, if you are configuring a brokering node hosted in 192.168.0.102 as the thrift server, this value should be 192.168.0.102.
    thriftServerPort
    This is another sub-element of the <coordination> tag. This should point to the port of the thrift server in the MB profile.
     Info
    titleWhat is the effective thrift server port ?
    The broker.xml file indicates 7611 as the default thrift server port. However, the default thrift server port that is effective in the MB profile of WSO2 EI is 7614 (7611 + 3). This is because, by default, the server port of the MB profile of WSO2 EI is offset by 3 in the <EI_HOME>/wso2/broker/conf/
    broker
    carbon.xml
     file.
     Tip
    This file is the root configuration file of brokering. Do the changes you do to this file in all the broker profile nodes. Configure the thriftServerHost value to point to the IP address of the message broker profile node.
     Code Block
    languagexml
    <coordination>
    <nodeID>default</nodeID>
    <thriftServerHost>192.168.0.103</thriftServerHost>
    <thriftServerPort>7611</thriftServerPort>
	<thriftSOTimeout>0</thriftSOTimeout>
</coordination>
    Following are details on these configurations.
    This is used to handle half-open TCP connections between the broker nodes in a cluster. In such situations, the socket may need to have a timeout value to invalidate the connection (in milliseconds). A timeout of zero is interpreted as an infinite timeout. Be sure to set this value to 180000 milliseconds as shown below in order to handle a high number of half-open TCP connections.
    <thriftSOTimeout>180000</thriftSOTimeout>
    ConfigurationDescription
    coordination
    This configuration is related to brokering thrift communications.
    nodeID
    The node ID of each member in a cluster must be unique. In a clustered deployment, an ID is assigned to each MB node via the cluster node identifier. The node ID specified by this parameter can be used to override the cluster node identifier for this MB node. If the node ID is set to 'default', the node ID will be retrieved using the Hazelcast member ID that is specified in the axis2.xml file as shown below.
     Code Block
    <parameter name="localMemberHost">192.168.0.102</parameter>
    Note that when this parameter is set to 'default', the node ID is made up of the 'localMemberHost' and the 'localMemberPort' (defined in the axis2.xml) as shown below. The 'localMemberHost' will be the hostname of the server if the hostname can be mapped with the provided IP address. Otherwise, the 'localMemberHost' will always be the provided IP address.
     Code Block
    NODE:localMemberHost:localMemberPort
    thriftServerHost
    This is a sub-element of the <coordination> tag. WSO2 EI uses Apache Thrift for communications relating to message delivery. Therefore, an Apache Thrift server is started in each brokering node in a clustered deployment. This element should point to the IP address of the Apache Thrift server. This should point to the IP address of the brokering node that hosts the thrift server. The default value for this is localhost. For example, if you are configuring a brokering node hosted in 192.168.0.102 as the thrift server, this value should be 192.168.0.102.
    thriftServerPort
    This is another sub-element of the <coordination> tag. This should point to the port of the thrift server in the message broker profile.
     Info
    titleWhat is the effective thrift server port ?
    The broker.xml file indicates 7611 as the default thrift server port. However, the default thrift server port that is effective in the MB profile of WSO2 EI is 7614 (7611 + 3). This is because, by default, the server port of the MB profile of WSO2 EI is offset by 3 in the <EI_HOME>/wso2/broker/conf/carbon.xml file. It is important to note that the effective thrift server port depends upon the default server port offset in the carbon.xml file as well as the thriftServerPort value in the broker.xml file. Be sure to keep the correct thrift server port open in your system to avoid errors.
    It is recommended to use the same thrift server port for all broker nodes in your cluster. 
     thriftSOTimeout 
     Code Block
     file. It is important to note that the effective thrift server port depends upon the default server port offset in the carbon.xml file as well as the thriftServerPort value in the broker.xml file. Be sure to keep the correct thrift server port open in your system to avoid errors.
    It is recommended to use the same thrift server port for all broker nodes in your cluster.
     thriftSOTimeout 
    This is used to handle half-open TCP connections between the broker nodes in a cluster. In such situations, the socket may need to have a timeout value to invalidate the connection (in milliseconds). A timeout of zero is interpreted as an infinite timeout. Be sure to set this value to 180000 milliseconds as shown below.
     Code Block
    <thriftSOTimeout>180000</thriftSOTimeout>
  9.  Anchor
    enable_hazelcast
    enable_hazelcast
    Edit the <EI_HOME>/wso2/broker/conf/axis2/axis2.xml file as follows to set up the cluster configurations.
    1. Enable clustering for this node as follows: <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
    2. Set the membership scheme to "wka" to enable the well-known address registration method. (This node sends cluster initiation messages to the WKA members):  
       <parameter name="membershipScheme">wka</parameter> 
    3. Specify the name of the cluster this node will join as follows: 
      <parameter name="domain">wso2.ei.domain</parameter> 
    4. Specify the hosts to communicate cluster messages as follows: 
      <parameter name="localMemberHost">xxx.xxx.xxx.xx2</parameter>
    5. Specify the port to communicate cluster messages as follows: 
      <parameter name="localMemberPort">4100</parameter>
       Info
      This port number is not affected by the port offset value specified in the <EI_HOME>/wso2/broker/conf/carbon. xml file. If this port number is already assigned to another server, the clustering framework automatically increments this port number. However, if there are two servers running on the same machine, ensure that a unique port is set for each server.
    6. Specify the well-known members as shown below. The port value for the WKA node must be the same value as it's localMemberPort (in this case it is 4100.
       Info
      You can also use IP address ranges for the hostname (e.g., 192.168.1.2-10). However, you can define a range only for the last portion of the IP address. Smaller the range, the faster the time it takes to discover members since each node has to scan a lesser number of potential members. The best practice is to add all the members (including itself) in all the nodes to avoid any conflicts in configurations.
       Code Block
      languagexml
      <members>    
    <member>
        <hostName>xxx.xxx.xxx.xx1</hostName>
        <port>4100</port>
    </member>
    <member>
        <hostName>xxx.xxx.xxx.xx2</hostName>
        <port>4200</port>
    </member>
</members>
  10. Enable heartbeat messaging for each of the broker nodes by following the steps given below. This is required for handling TCP connections between the broker nodes and client systems (publishers and subscribers).
     Info
    titleWhat is heartbeat messaging?
     Expand
    titleClick here for more information.
    When a client is connected to the broker, both the broker and the client should be able to detect problem situations where the TCP connection is half open or where the connecting client/broker is unresponsive. This can be achieved by enabling heartbeat messaging between the broker and the client.
    The heartbeat messaging configuration allows both the broker and the client to verify whether the connection is inactive or whether the connecting system (broker or client) is inactive by periodically sending messages to each other.
    For example, you can set the heartbeat delay time to 30 seconds. This means that if the broker (or the client) does not receive the expected response from the other system within 30 seconds, it will send a heartbeat message to check if the connection is inactive or if the other system is inactive. If the connection is inactive or is half-open, there will be a clear indication of connection failure. However, if the connection is active but the other system is unresponsive, the broker or the client will continue to send heartbeat messages (every 30 seconds). By default, the broker and clients are configured to terminate (timeout) the connection after sending two heartbeat messages; however, this time out configuration can be changed.
    Note: In a scenario where the subscriber client is sending the heartbeat message to the broker node, if the connection is found to be broken or if the broker node is inactive, the connection will failover to another broker node in the cluster.
    1. Open the  qpid-config.xml file file that is stored in the in the  <EI_HOME>/wso2/broker/conf/advanced/ directory.
    2. Set the heartbeat delay as shown below. The recommended heartbeat delay is 30 seconds.
       Code Block
      <heartbeat>
    <delay>30</delay>
    <timeoutFactor>2.0</timeoutFactor>
</heartbeat>
      The elements in the above configuration are explained below.
         
      delayThe <delay> element specifies the time interval between heartbeat messages. This is the time that the broker or client will wait to receive a response from the other party. If the response is not received within this time, a heartbeat message is triggered.
      timeoutFactorThe number of heartbeat messages the broker will send to the client before terminating the connection. That is, if the timeoutFactor is 2, the broker will send heartbeat messages every 30 seconds twice, and if a response is not received from the client, the connection will be terminated.
Optional: Configuring Hazelcast properties
WSO2 products use Hazelcast as as its default clustering engine. You can configure the hazelcast properties for the product nodes by following the steps given below.
  1. Add the following property configurations to the hazelcast.properties file stored in the <EI_HOME>/wso2/broker/conf/ directory.
     Code Block
    #Disabling the hazelcast shutdown hook
hazelcast.shutdownhook.enabled=false
#Setting the hazelcast logging type to log4j
hazelcast.logging.type=log4j
    The above configurations are explained below.
    • Hazelcast shutdown hook: This configuration disables the shutdown hook in hazelcast, which ensures that the hazelcast instance shuts down gracefully whenever the product node shuts down. If the hazelcast shutdown hook is enabled (which is the default behavior of a product), you will see errors such as "Hazelcast instance is not active!" at the time of shutting down the product node: This is because the hazelcast instance shuts down too early when the shutdown hook is enabled.
    • Hazelcast logging type: This configuration sets the hazelcast logging type to log4j, which allows hazelcast logs to be written to the wso2carbon.log file.
  2. If you have enabled log4j for hazelcast logging as explained above, be sure to enter the configuration shown below in the log4j.properties file (stored in the in the  <EI_HOME>/wso2/broker/conf/  directory). This can be used to configure the log level for hazelcast logging. For a clustered production environment, it is recommended to use INFO as the log level as shown below.
     Code Block
    log4j.logger.com.hazelcast=INFO
 Anchor
RDBMS_clustering
RDBMS_clustering
...
Optional: Additional configurations
Cluster coordination and event synchronization
The cluster coordination process controls how the nodes in an MB cluster are managed. By default, the MB uses an RDBMS (the MB database that is connected to nodes in the cluster) for cluster coordination as well as cluster event synchronization. Note that this is also the same RDBMS that is used for message persistence in the MB, and thereby cluster coordination, cluster event synchronization, and message persistence works on the same network in your cluster.
The following settings in the broker.xml file (stored in the <EI_HOME>/wso2/broker/conf/ directory) 
...
 enables and configures RDBMS 
...
 cluster coordination 
...
and event synchronization. Note that it is not recommended to disable RDBMS cluster coordination and event synchronization as it can sometimes cause network partitioning in your cluster. 
 Note
The clustering mechanism of the MB profile depends on its Hazelcast engine. Therefore, Hazelcast clustering should always be enabled in the MB profile for RDBMS-based cluster coordination to work.
 Code Block
<rdbmsBasedCoordination enabled="true">
    <heartbeatInterval>5000</heartbeatInterval>
    <!-- Time to wait before informing others about coordinator change in milliseconds. This value should be
            larger than a database read time including network latency and should be less than heartbeatInterval -->
    <coordinatorEntryCreationWaitTime>3000</coordinatorEntryCreationWaitTime>
    <!-- Time interval used to poll database for membership related events in milliseconds. -->
    <eventPollingInterval>4000</eventPollingInterval>
</rdbmsBasedCoordination>
<!-- Enabling this will make the cluster notifications such as Queue changes(additions and deletions),
        Subscription changes, etc. sent within the cluster be synchronized using RDBMS. If set to false, Hazelcast
        will be used for this purpose.-->
<RDBMSBasedClusterEventSynchronization enabled="true">
    <!--Specifies the interval at which, the cluster events will be read from the database. Needs to be
            declared in milliseconds. Setting this to a very low value could downgrade the performance where as
            setting this to a large value could increase the time taken for a cluster event to be synchronized in
            all the nodes in a cluster.-->
    <eventSyncInterval>1000</eventSyncInterval>
</RDBMSBasedClusterEventSynchronization><coordinatorEntryCreationWaitTime>3000</coordinatorEntryCreationWaitTime>
    <eventPollingInterval>4000</eventPollingInterval>
</rdbmsBasedCoordination>
<RDBMSBasedClusterEventSynchronization enabled="true">
    <eventSyncInterval>1000</eventSyncInterval>
</RDBMSBasedClusterEventSynchronization>
See the parameter descriptions given below.
  • Cluster coordination parameters:
    ParameterDescription
    enabled
    This property is set to "true" by default, which indicates that RDBMS-based cluster coordination is enabled:
     Code Block
    <rdbmsBasedCoordination enabled="true">
    It is not recommended to disable this setting. If you disable this setting, the hazelcast engine will handle cluster coordinatin without using the RDBMS network, which can cause network partitioning. Find out more about how to handle network partitioning in hazelcast cluster coordination.
    heartbeatIntervalThe time interval for heartbeat messaging, which is used to check the availability of the nodes in the cluster. Declared in milliseconds.
    coordinatorEntryCreationWaitTimeThe time the system should wait before informing other nodes about a new coordinator. Declared in milliseconds. This value should be larger than the database read time (including network latency) and should be less than the heartbeatInterval value.
    eventPollingIntervalTime interval used to poll the database for membership related events. Declared in milliseconds.
  • Event synchronization parameters:
    ParameterDescription
    enabled
    This property is set to "true" by default, which indicates that RDBMS-based cluster event synchronization is enabled:
     Code Block
    <RDBMSBasedClusterEventSynchronization enabled="true">
    It is not recommended to disable this setting.
    eventSyncIntervalSpecifies the interval at which the cluster events will be read from the database. Declared in milliseconds. Setting this to a very low value could decrease server performance where as setting this to a large value could increase the time taken for a cluster event to be synchronized in all the nodes in a cluster.
 Anchor
network_partitioning
network_partitioning
Handling network partitioning in Hazelcast-based clustering
Network partitioning can happen in the network used for cluster coordination can sometimes disrupt the cluster. The Message Broker profile of WSO2 EI introduces a new configuration to handle network partitioning that may occur when cluster coordination is managed by the hazelcast engine (as opposed to the RDBMS as explained above).For example, consider a cluster of three MB nodes that uses two separate networks for cluster coordination and message persistence. The network Message Broker cluster if you have disabled RDBMS-based cluster coordination. That is, when RDBMS-based clustering is disabled, the hazelcast engine in the MB uses two separate networks:  The network with the RDBMS will persist messages, subscriptions and queues etc., and the cluster network communicates information about subscriptions, queue additions, and to decide on the coordinator of the cluster. 
For example, consider a cluster of three MB nodes that uses two separate networks for cluster coordination and message persistence. If one of the nodes in the cluster disconnects from the network used for cluster coordination, that node will separate from the cluster and function as a separate node/cluster (separate partition). That is, the three-node cluster will now be working as two separate clusters. However, the disconnected node will still be up and running, and message persistence will continue uninterrupted through the message persistence network. This will cause inconsistencies in the system because message persistence and cluster coordination will not be synchronized. In  In such a situation, it is necessary to stop the disconnected node from accepting messages.
The above situation can be prevented in the Message Broker profile by configuring the minimum node count of the cluster.  For example, if the cluster size is 5 and we have configured the minimum node count to 3 nodes, during a network partition (or when nodes crash or shut down) any partition that has 3 or more nodes will keep functioning, while the other partitions will stop processing messages.
Follow the steps given below to enable this feature.
  1. Open the Open the <EI_HOME>/wso2/broker/repository/conf/broker.xml file and set the <networkPartitionsDetection> element shown below to  file and set the <networkPartitionsDetection> element shown below to enabled="true" . Note that this configuration is disabled by default as shown below.
     Code Block
    <?xml version="1.0"?>
<networkPartitionsDetection enabled="false">
    <minimumClusterSize>1</minimumClusterSize>
</networkPartitionsDetection>
  2. Update the <minimumClusterSize> property in the above configuration to specify the the minimum node count the cluster should maintain in order to operateto operate. Note that if this value should be at least two for cluster coordination to work.  If the number of nodes in the cluster becomes less that than the configured value, the cluster will not accept any incoming traffic. That is, all subscriptions will be disconnected.
Testing the cluster
 Note
iconfalse
This documentation on testing the cluster is currently under review!
...
 the 
...
 cluster
...
Deploy artifacts to each product deployment location. 
 Info
Use a deployment synchronization mechanism to synchronize the artifacts in the <EI_HOME>/wso2/broker/repository/deployment/ directory. Always deploy artifacts first to the WSO2 EI  server profile node with the registry configured as read/write. Next, deploy the artifacts to the other nodes. 
...
Follow the steps below to test the cluster.
  1. Start the MB profile nodes. See the instructions in Running the Product.
  2. Check for ‘member joined’ log messages in all consoles.
     Info
    titleAdditional information on logs and new nodes
    When you terminate one node, all nodes identify that the node has left the cluster. The same applies when a new node joins the cluster. If you want to add another new node, copy existing node without any changes if you are running it on a new server (such as xxx.xxx.xxx.184). If you intend to use the new node on a server where another WSO2 product is running, use a copy of node and change the port offset accordingly in the <EI_HOME>/wso2/broker/conf/carbon.xml file. You also have to change localMemberPort in the <EI_HOME>/wso2/broker/conf/axis2/axis2.xml file if that product has clustering enabled. Also, map all hostnames to the relevant IP addresses when creating a new node. The log messages indicate if the new node joins the cluster.
Tuning performance of the cluster
Follow the steps below to tune performance of the cluster:
 Tip
The below example parameter values might not be the optimal values for the specific hardware configurations in your environment. Therefore, it is recommended to carry out load tests on your environment to tune the load balancer and other configurations accordingly.
...
  1. log messages indicate if the new node joins the cluster.
  2. Access the Management Console through the LB using the following URL:  https://xxx.xxx.xxx.xx2:443/carbon 
  3. Sign into the management console using admin as the username and password.
  4. Click Node List.
    Image Added
    You see that the 2 nodes are up and running successfully. The message store health needs to be Healthy and only one node can be the Coordinator Node. 
    Image Added
What's next
If you want to performance tune the WSO2 Message Broker (MB) profile, see Tuning the Message Broker Profile .