/
Clustering the Message Broker Profile
com.atlassian.confluence.content.render.xhtml.migration.exceptions.UnknownMacroMigrationException: The macro 'next_previous_links2' is unknown.

Clustering the Message Broker Profile

Feb 17, 2020

The following sections give you information and instructions on how to cluster the message broker profile of WSO2 EI.  

The clustering deployment pattern

This pattern has two WSO2 EI nodes that include the Message Broker profile to serve service requests with high availability and scalability. The following image depicts the sample pattern of this clustering deployment scenario.

This pattern uses two nodes as well-known members. It is always recommended to have all nodes of the cluster as well-known members.

When configuring your WSO2 products for clustering using Hazelcast, you need to use a specific IP address in your configurations and not localhost.

  • In this guide, the IP of the first Message Broker (MB) node is referred to as https://xxx.xxx.xxx.xx1 and the IP of the second MB node is referred to as https://xxx.xxx.xxx.xx2.

  • If you want to test out clustering in a development environment using the same server, you need to port offset one of your MB nodes.
    Specify the port offset value in the <EI_HOME>/wso2/broker/conf/carbon.xml file.

    Example:

    <Ports> ... <Offset>5</Offset> ... </Ports>

Creating the databases

All profiles of WSO2 EI uses a database to store information such as user management details and registry data. All nodes in the cluster must use one central database for config and governance registry mounts. You can create the following databases and associated datasources.

Database Name

Description

Database Name

Description

WSO2_USER_DB

JDBC user store and authorization manager.

REGISTRY_DB

Shared database for config and governance registry mounts in the product's nodes.

REGISTRY_LOCAL1

Local registry space in Node 1.

REGISTRY_LOCAL2

Local registry space in Node 2.

In addition to the databases listed above, the MB profile of WSO2 EI requires the following broker-specific data:

Database Name

Description

Database Name

Description

MB_DB

Stores instance data that are specific to the message broker profile.

It is recommended to use an industry-standard RDBMS such as Oracle, PostgreSQL, MySQL, MS SQL, etc. for most enterprise testing and production environments. However, you can also use the embedded H2 database only for the REGISTRY_LOCAL1 and REGISTRY_LOCAL2.

Follow the steps given below to create the databases required for the MB profile of WSO2 EI. 

  1. Download and install the MySQL Server.

  2. Download the MySQL JDBC driver.

  3. Download and unzip the WSO2 EI binary distribution. Be sure to select the correct product version. 

  4. Unzip the downloaded MySQL driver, and copy the MySQL JDBC driver JAR (mysql-connector-java-x.x.xx-bin.jar) to the <EI_HOME>/lib/ directory of both the WSO2 EI nodes.

  5. Add the following line to the /etc/hosts file to define the hostname for configuring permissions for the new database: <MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com

  6. Execute the following command in a terminal/command window, where username is the username you want to use to access the databases: mysql -u username -p

  7. When prompted, specify the password to access the databases.

  8. Create the databases using the following commands:

    mysql> create database WSO2_USER_DB; mysql> use WSO2_USER_DB; mysql> source <EI_HOME>/dbscripts/mysql.sql; mysql> grant all on WSO2_USER_DB.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin"; mysql> create database REGISTRY_DB; mysql> use REGISTRY_DB; mysql> source <EI_HOME>/dbscripts/mysql.sql; mysql> grant all on REGISTRY_DB.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin"; mysql> create database REGISTRY_LOCAL1; mysql> use REGISTRY_LOCAL1; mysql> source <EI_HOME>/dbscripts/mysql.sql; mysql> grant all on REGISTRY_LOCAL1.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin"; mysql> create database REGISTRY_LOCAL2; mysql> use REGISTRY_LOCAL2; mysql> source <EI_HOME>/dbscripts/mysql.sql; mysql> grant all on REGISTRY_LOCAL2.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";

  9. Create and configure the MB_DB database, which is specific to the MB profile:

    mysql> create database wso2_mb; mysql> use wso2_mb; mysql> source <EI_HOME>/wso2/broker/dbscripts/mb-store/mysql-mb.sql; mysql> grant all on wso2_mb.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";

Mounting the registry

Add the following configurations to the <EI_HOME>/wso2/broker/conf/registry.xml file of each WSO2 EI node to configure the shared registry database and mounting details. This ensures that the shared registry for governance and configurations (i.e., the REGISTRY_DB database) mounts on both WSO2 EI nodes. 

Note the following when adding these configurations:

  • The existing dbConfig called wso2registry must not be removed.

  • The datasource you specify in the <dbConfig name="sharedregistry"> tag must match the JNDI Config name you specify in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file.

  • The registry mount path denotes the type of registry. For example, ”/_system/config” refers to configuration Registry, and "/_system/governance" refers to the governance registry.

  • The <dbconfig> entry enables you to identify the datasource you configured in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file. The unique name "shared registry" refers to that datasource entry.

  • The <remoteInstance> section refers to an external registry mount. Specify the read-only/read-write nature of this instance, caching configurations and the registry root location in this section.

  • Also, specify the cache ID in the <remoteInstance> section. This enables caching to function properly in the clustered environment.

  • Define a unique name in the <id> tag for each remote instance. This is then referred to from mount configurations. In the above example, the unique ID for the remote instance is "instanceId".

  • Specify the actual mount path and target mount path in each of the mounting configurations. The target path can be any meaningful name. In this instance, it is "/_system/mbconfig".

<dbConfig name="sharedregistry"> <dataSource>jdbc/WSO2RegistryDB</dataSource> </dbConfig> <remoteInstance url="https://localhost:9443/registry"> <id>instanceid</id> <dbConfig>sharedregistry</dbConfig> <readOnly>false</readOnly> <enableCache>true</enableCache> <registryRoot>/</registryRoot> <cacheId>regadmin@jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</cacheId> </remoteInstance> <mount path="/_system/config" overwrite="true"> <instanceId>instanceid</instanceId> <targetPath>/_system/mbconfig</targetPath> </mount> <mount path="/_system/governance" overwrite="true"> <instanceId>instanceid</instanceId> <targetPath>/_system/governance</targetPath> </mount>

Configuring a message broker profile node

Time synchronising the broker nodes

Do the following configurations for all nodes of your cluster.

  1. Configure the datasources to point to the REGISTRY_LOCAL1, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as follows in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file as follows:

  2. Remove or comment out the default H2-based WSO2_MB_STORE_DB and WSO2_CARBON_DB datasource configurations from the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file. 

  3. Uncomment or add the following WSO2_MB_STORE_DB configuration in the <EI_HOME>/wso2/broker/conf/datasources/master-datasources.xml file based on your DBMS type.

  4. Add the following configuration in the <EI_HOME>/wso2/broker/conf/user-mgt.xml file of both nodes to configure the user stores. 

    <Configuration> <AddAdmin>true</AddAdmin> <AdminRole>admin</AdminRole> <AdminUser> <UserName>admin</UserName> <Password>admin</Password> </AdminUser> <EveryOneRoleName>everyone</EveryOneRoleName> <Property name="dataSource">jdbc/WSO2UMDB</Property> </Configuration>

  5. Uncomment or add the following configuration in the <EI_HOME>/wso2/broker/conf/broker.xml file.  Enter the message store based on your database type and Andes context store.

  6. Add the following thrift-related configurations in the <EI_HOME>/wso2/broker/conf/broker.xml file.

    <coordination> <nodeID>default</nodeID> <thriftServerHost>192.168.0.103</thriftServerHost> <thriftServerPort>7611</thriftServerPort> <thriftSOTimeout>0</thriftSOTimeout> </coordination>

  7. Edit the <EI_HOME>/wso2/broker/conf/axis2/axis2.xml file as follows to set up the cluster configurations.

    1. Enable clustering for this node as follows: <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

    2. Set the membership scheme to "wka" to enable the well-known address registration method. (This node sends cluster initiation messages to the WKA members): 
      <parameter name="membershipScheme">wka</parameter>

    3. Specify the name of the cluster this node will join as follows: 
      <parameter name="domain">wso2.ei.domain</parameter>

    4. Specify the hosts to communicate cluster messages as follows: 
      <parameter name="localMemberHost">xxx.xxx.xxx.xx2</parameter>

    5. Specify the port to communicate cluster messages as follows: 
      <parameter name="localMemberPort">4100</parameter>

    6. Specify the well-known members as shown below. The port value for the WKA node must be the same value as it's localMemberPort (in this case it is 4100.

      <members> <member> <hostName>xxx.xxx.xxx.xx1</hostName> <port>4100</port> </member> <member> <hostName>xxx.xxx.xxx.xx2</hostName> <port>4200</port> </member> </members>

  8. Enable heartbeat messaging for each of the broker nodes by following the steps given below. This is required for handling TCP connections between the broker nodes and client systems (publishers and subscribers).

    1. Open the  qpid-config.xml file that is stored in the  <EI_HOME>/wso2/broker/conf/advanced directory.

    2. Set the heartbeat delay as shown below. The recommended heartbeat delay is 30 seconds.

      <heartbeat> <delay>30</delay> <timeoutFactor>2.0</timeoutFactor> </heartbeat>

      The elements in the above configuration are explained below.

com.atlassian.confluence.content.render.xhtml.migration.exceptions.UnknownMacroMigrationException: The macro 'next_previous_links' is unknown.