Versions Compared

Old Version 8

changes.mady.by.user Former user

Saved on

compared with

New Version 9

changes.mady.by.user Former user

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The following sections give you information and instructions on how to cluster the message broker profile of WSO2 EI with a third-party load balancer. 

...

  1. Configure the datasources to point to the REGISTRY_LOCAL1, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as follows in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file as follows:

    Tip

    Replace the username, password, and database URL of your MySQL environment accordingly.

    Code Block
    languagexml
    <datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration"> 
     <providers> 
        <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider> 
    </providers> 
    <datasources> 
        <datasource> 
            <name>REGISTRY_LOCAL1</name> 
            <description>The datasource used for registry- local</description> 
            <jndiConfig> 
                <name>jdbc/WSO2CarbonDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL1?autoReconnect=true</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
        <datasource> 
            <name>REGISTRY_DB</name> 
            <description>The datasource used for registry- config/governance</description> 
            <jndiConfig> 
                <name>jdbc/WSO2RegistryDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
         <datasource> 
            <name>WSO2_USER_DB</name> 
            <description>The datasource used for registry and user manager</description> 
            <jndiConfig> 
                <name>jdbc/WSO2UMDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
   </datasources> 
</datasources-configuration>
    Tip

    Repeat this configuration on the second WSO2 EI node to configure the datasources to point to the REGISTRY_LOCAL2, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as follows: (Change the username, password, and database URL as needed for your environment):

    Code Block
    languagexml
    <datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration">     
    <providers>
        <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider>
    </providers>
    <datasources>
        <datasource>
            <name>REGISTRY_LOCAL2</name>
            <description>The datasource used for registry- local</description>
            <jndiConfig>
                <name>jdbc/WSO2CarbonDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL2?autoReconnect=true</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
        <datasource>
            <name>REGISTRY_DB</name>
            <description>The datasource used for registry- config/governance</description>
            <jndiConfig>
                <name>jdbc/WSO2RegistryDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
         <datasource>
            <name>WSO2_USER_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2UMDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
   </datasources>
</datasources-configuration>

  2. Add the following configuration in the <EI_HOME>/wso2/broker/conf/user-mgt.xml file to configure the user stores. 

    Tip

    Enter the datasource information for the user store that you configured in the  <EI_HOME>/wso2/broker/conf/ datasources /master- datasources . xml file. You can change the admin username and password as well. However, you should do this before starting the server.

    Code Block
    languagexml
    <Configuration>
    <AddAdmin>true</AddAdmin>
    <AdminRole>admin</AdminRole>
    <AdminUser>
        <UserName>admin</UserName>
        <Password>admin</Password>
    </AdminUser>
    <EveryOneRoleName>everyone</EveryOneRoleName>
    <Property name="dataSource">jdbc/WSO2UMDB</Property>
</Configuration>

  3. Remove or uncomment the default H2-based WSO2_MB_STORE_DB following configuration in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file. 

    Code Block
    languagexml
    <datasource>
    <name>WSO2_MB_STORE_DB</name>
    <description>The datasource used for message broker database</description>
    <jndiConfig>
        <name>WSO2MBStoreDB</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <url>jdbc:h2:repository/database/WSO2MB_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url>
            <username>wso2carbon</username>
            <password>wso2carbon</password>
            <driverClassName>org.h2.Driver</driverClassName>
            <maxActive>50</maxActive>
            <maxWait>60000</maxWait>
            <testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
            <defaultAutoCommit>false</defaultAutoCommit>
        </configuration>
    </definition>
</datasource>

  4. Uncomment or add the following WSO2_MB_STORE_DB configuration in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file based on your DBMS type.

    Tip

    Update the JDBC URL to correctly point to your database and enter the username and password for your database user with the proper permissions.

    Localtabgroup
    Localtab
    titleConfiguring MSSQL
    Code Block
    languagexml
    <datasource>
    <name>WSO2_MB_STORE_DB</name>
    <jndiConfig>
        <name>jdbc/WSO2MBStoreDB</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <defaultAutoCommit>false</defaultAutoCommit>
            <dataSourceClassName>com.microsoft.sqlserver.jdbc.SQLServerXADataSource</dataSourceClassName>
            <dataSourceProps>
                <property name="URL">jdbc:sqlserver://127.0.0.1\SQLExpress</property>
                <property name="databaseName">wso2mb</property>
                <property name="user">sa</property>
                <property name="password">mssql</property>
            </dataSourceProps>
        </configuration>
    </definition>
</datasource>
    Localtab
    titleConfiguring MySQL
    Code Block
    languagexml
    <datasource>
    <name>WSO2_MB_STORE_DB</name>
    <jndiConfig>
        <name>WSO2MBStoreDB</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <driverClassName>com.mysql.jdbc.Driver</driverClassName>
            <url>jdbc:mysql://localhost/wso2_mb</url>
            <username>root</username>
            <password>root</password>
            <maxActive>50</maxActive>
            <maxWait>60000</maxWait>
            <minIdle>5</minIdle>
            <testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
            <defaultAutoCommit>false</defaultAutoCommit>
        </configuration>
    </definition>
</datasource>
    Localtab
    titleConfiguring Oracle
    Code Block
    languagexml
     <datasource>
    <name>WSO2_MB_STORE_DB</name>
    <jndiConfig>
        <name>WSO2MBStoreDB</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <defaultAutoCommit>false</defaultAutoCommit>
            <driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
            <url>jdbc:oracle:thin:@localhost:1521/orcl</url>
            <maxActive>100</maxActive>
            <maxWait>60000</maxWait>
            <minIdle>5</minIdle>
            <testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
            <username>scott</username>
            <password>tiger</password>
        </configuration>
    </definition>
</datasource>

  5. Uncomment or add the following configuration in the <EI_HOME>/wso2/broker/conf/broker.xml file. (Enter the message store based on your database type and Andes context store).

    Localtabgroup
    Localtab
    titleConfiguring MSSQL

    ...

    <persistence>
      <messageStore class="org.wso2.andes.store.rdbms.RDBMSMessageStoreImpl">
            <property name="dataSource">jdbc/MSSQLWSO2MBStoreDB</property>
      </messageStore>
     
      <andesContextStore    class="org.wso2.andes.store.rdbms.RDBMSAndesContextStoreImpl">
            <property name="dataSource">jdbc/MSSQLWSO2MBStoreDB</property>
      </andesContextStore>
    ...
    </persistence>
    Localtab
    titleConfiguring MySQL

    ...

    <persistence>
      <messageStore class="org.wso2.andes.store.rdbms.RDBMSMessageStoreImpl">
            <property name="dataSource">jdbc/MySQLMessageStore</property>
      </messageStore>
     
      <andesContextStore    class="org.wso2.andes.store.rdbms.RDBMSAndesContextStoreImpl">
            <property name="dataSource">jdbc/MySQLMessageStore</property>
      </andesContextStore>
    ...
    </persistence>
    Localtab
    titleConfiguring Oracle

    ...

    <persistence>
     <messageStore class="org.wso2.andes.store.rdbms.RDBMSMessageStoreImpl">
               <property name="dataSource">WSO2MBStoreDB</property>
    </messageStore>
     
    <contextStore class="org.wso2.andes.store.rdbms.RDBMSAndesContextStoreImpl">
               <property name="dataSource">WSO2MBStoreDB</property>
    </contextStore>
    ...
    </persistence>

  6. Edit the <EI_HOME>/wso2/broker/conf/axis2/axis2.xml file as follows to set up the cluster configurations.

    • Enable clustering for this node as follows: <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
    • Set the membership scheme to "wka" to enable the well-known address registration method. (This node sends cluster initiation messages to the WKA members): 
      <parameter name="membershipScheme">wka</parameter>
    • Specify the name of the cluster this node will join as folows: <parameter name="domain">wso2.ei.domain</parameter>
    • Specify the host to communicate cluster messages as follows: <parameter name="localMemberHost">xxx.xxx.xxx.xx2</parameter>

    • Specify the port to communicate cluster messages as follows: <parameter name="localMemberPort">4100</parameter>

      Info

      This port number is not affected by the port offset value specified in the  <EI_HOME>/wso2/broker/conf/carbon. xml file. If this port number is already assigned to another server, the clustering framework automatically increments this port number. However, if there are two servers running on the same machine, ensure that a unique port is set for each server.

    • Specify the well-known members as follows: (The port value for the WKA node must be the same value as it's localMemberPort (in this case it is 4100).

      Info

      You can also use IP address ranges for the hostname (e.g.,   192.168.1.2-10). However, you can define a range only for the last portion of the IP address. Smaller the range, the faster the time it takes to discover members since each node has to scan a lesser number of potential members.

  7. Add the following thrift-related configurations in the <EI_HOME>/wso2/broker/conf/broker.xml file.

    Tip

    This file is the root configuration file of brokering. Do the changes you do to this file in all the broker profile nodes. Configure the thriftServerHost value to point to the IP address of the message broker profile node.

    Code Block
    languagexml
    <coordination>
    <nodeID>default</nodeID>
    <thriftServerHost>192.168.0.103</thriftServerHost>
    <thriftServerPort>7611</thriftServerPort>
	<thriftSOTimeout>0</thriftSOTimeout>
</coordination>

    Follwingaredetails on these configurations.

    ConfigurationDescription
    coordination

    This configuration is related to brokering thrift communications.

    nodeIDIn a clustered deployment, an ID is assigned to each brokering node via the cluster node identifier. This element can be used to override the cluster node identifier for this brokering node. If the value for this element is left as default, the default node ID is generated using the IP and a universally unique identifier (UUID). The node ID of each member in a cluster must be unique.
    thriftServerHost

    This is a sub-element of the <coordination> tag. WSO2 EI uses Apache Thrift for communications relating to message delivery. Therefore, an Apache Thrift server is started in each brokering node in a clustered deployment. This element should point to the IP address of the Apache Thrift server. This should point to the IP address of the brokering node that hosts the thrift server. The default value for this is localhost. For example, if you are configuring a brokering node hosted in 192.168.0.102 as the thrift server, this value should be 192.168.0.102.

    thriftServerPort

    This is another sub-element of the <coordination> tag. This should point to the port of the thrift server in the message broker profile. The default port is 7611. It is recommended to use this port for all broker nodes in your cluster. 

    thriftSOTimeoutThis is used to handle half-open TCP connections between the broker nodes in a cluster. In such situations, the socket may need to have a timeout value to invalidate the connection (in milliseconds). A timeout of zero is interpreted as an infinite timeout.

Anchor
RDBMS_clustering
RDBMS_clustering
Configuring cluster coordination

The Messgae Broker profile in WSO2 EI introduces cluster coordination through an RDBMS. This means that the coordination between the nodes in a cluster can be managed through an RDBMS, just as message persistence. Therefore, the RDBMS that is connected to the MB nodes in the cluster is, by default, used for message persistence as well as cluster coordination. Shown below is the configuration in the broker.xml file (stored in the <EI_HOME>/wso2/broker/conf/ directory), which enables RDBMS-based cluster coordination. If required, you can disable the following configuration, which will allow the hazelcast engine to manage cluster coordination. However, note that you need to configure the cluster to handle network partitioning when hazelcast-based cluster coordination is used. 

Code Block
<rdbmsBasedCoordination enabled="true">
    <!-- Heartbeat interval used in the RDBMS base coordination algorithm in milliseconds -->
    <heartbeatInterval>5000</heartbeatInterval>
    <!-- Time to wait before informing others about coordinator change in milliseconds. This value should be
            larger than a database read time including network latency and should be less than heartbeatInterval -->
    <coordinatorEntryCreationWaitTime>3000</coordinatorEntryCreationWaitTime>
    <!-- Time interval used to poll database for membership related events in milliseconds. -->
    <eventPollingInterval>4000</eventPollingInterval>
</rdbmsBasedCoordination>
<!-- Enabling this will make the cluster notifications such as Queue changes(additions and deletions),
        Subscription changes, etc. sent within the cluster be synchronized using RDBMS. If set to false, Hazelcast
        will be used for this purpose.-->
<RDBMSBasedClusterEventSynchronization enabled="true">
    <!--Specifies the interval at which, the cluster events will be read from the database. Needs to be
            declared in milliseconds. Setting this to a very low value could downgrade the performance where as
            setting this to a large value could increase the time taken for a cluster event to be synchronized in
            all the nodes in a cluster.-->
    <eventSyncInterval>1000</eventSyncInterval>
</RDBMSBasedClusterEventSynchronization>

Anchor
network_partitioning
network_partitioning
Handling network partitioning in Hazelcast-based clustering

Network partitioning in the network used for cluster coordination can sometimes disrupt the cluster. The Message Broker profile of WSO2 EI introduces a new configuration to handle network partitioning that may occur when cluster coordination is managed by the hazelcast engine (as opposed to the RDBMS as explained above).

For example, consider a cluster of three MB nodes that uses two separate networks for cluster coordination and message persistence. The network with the RDBMS will persist messages, subscriptions and queues etc., and the cluster network communicates information about subscriptions, queue additions, and to decide on the coordinator of the cluster. If one of the nodes in the cluster disconnects from the network used for cluster coordination, that node will separate from the cluster and function as a separate node/cluster (separate partition) as illustrated in the diagram below. That is, the three-node cluster will now be working as two separate clusters. However, the disconnected node will still be up and running, and message persistence will continue uninterrupted through the message persistence network. This will cause inconsistencies in the system because message persistence and cluster coordination will not be synchronized. In such a situation, it is necessary to stop the disconnected node from accepting messages.

network partitioningImage Added

The above situation can be prevented in the Message Broker profile by configuring the minimum node count of the cluster. For example, if the cluster size is 5 and we have configured the minimum node count to 3 nodes, during a network partition (or when nodes crash or shut down) any partition that has 3 or more nodes will keep functioning, while the other partitions will stop processing messages.

Follow the steps given below to enable this feature.

  1. Open the <EI_HOME>/wso2/broker/repository/conf/broker.xml file and set the <networkPartitionsDetection> element shown below to enabled="true". Note that this configuration is disabled by default as shown below.

    Code Block
    <?xml version="1.0"?>
<networkPartitionsDetection enabled="false">
    <minimumClusterSize>1</minimumClusterSize>
</networkPartitionsDetection>
  2. Update the <minimumClusterSize> property in the above configuration to specify the minimum node count the cluster should maintain in order to operate. Note that if this value should be at least two for cluster coordination to work. If the number of nodes in the cluster becomes less that configured value, the cluster will not accept any incoming traffic. That is, all subscriptions will be disconnected.

Testing the cluster

Follow the steps below to test the cluster.

...