Page Comparison

This page describes how to create a cluster by walking through the example in the Overview. Although this example uses WSO2 ESB, these steps apply to other WSO2 products as well. For details on further configuration required for the WSO2 product you are clustering, see the links in the table of contents. Note that this page describes using WSO2 Elastic Load Balancer (ELB), but you can use a third-party load balancer in its place (for configuration details, see your load balancer's documentation).

This page describes how to create a cluster by walking through the example in the Overview. Although this example uses WSO2 ESB, these steps apply to other WSO2 products as well. For details on further configuration required for the WSO2 product you are clustering, see the links in the table of contents.

Installing the products

Before you begin, download and extract WSO2 ESB 4.7.0 and WSO2 ELB 2.0.3 to a local directory on the sever. For this example, we have extracted one copy of the ELB and two copies of the ESB on the server with IP xxx.xxx.xxx.206 (the x's represent your actual IP prefix), and we extracted two copies of the ESB on the server with the IP xxx.xxx.xxx.132:

Server xxx.xxx.xxx.206:

• 1 ELB instance (Well Known Member)
• 1 ESB instance (worker node)
• 1 ESB instance (Dep-sync management / manager node )

Server xxx.xxx.xxx.132:

• 2 ESB instances (worker nodes)

Configuring the load balancer

You configure the ELB with the overall definition of the cluster and how it should distribute the load. You can achieve this by adding a few lines to a configuration file called loadbalancer.conf. You specify the detailed clustering configurations in the axis2.xml file. This section describes how to perform these steps.

Setting up load-balancing configurations

...

Locate the ESB configuration and edit it as follows:

...

...

Note that this page describes using WSO2 Elastic Load Balancer (ELB), but you can use a third-party load balancer in its place (for configuration details, see your load balancer's documentation).

For details on further configuration required for the WSO2 product you are clustering, see the links in the table of contents.

Installing the products

Before you begin, download and extract WSO2 ESB 4.7.0 and WSO2 ELB 2.0.3 to a local directory on the sever. For this example, we have extracted one copy of the ELB and two copies of the ESB on the server with IP xxx.xxx.xxx.206 (the x's represent your actual IP prefix), and we extracted two copies of the ESB on the server with the IP xxx.xxx.xxx.132:

Server xxx.xxx.xxx.206:

• 1 ELB instance (Well Known Member)
• 1 ESB instance (worker node)
• 1 ESB instance (Dep-sync management / manager node )

Server xxx.xxx.xxx.132:

• 2 ESB instances (worker nodes)

Configuring the load balancer

You configure the ELB with the overall definition of the cluster and how it should distribute the load. You can achieve this by adding a few lines to a configuration file called loadbalancer.conf. You specify the detailed clustering configurations in the axis2.xml file. This section describes how to perform these steps.

Setting up load-balancing configurations

1. Open the <ELB_HOME>/repository/conf/loadbalancer.conf file.

2. Locate the ESB configuration and edit it as follows:

   Code Block
   language html/xml
   esb { domains { wso2.esb.domain { hosts mgt.esb.wso2.com; sub_domain workermgt; tenant_range *; } } }

...

1. wso2.esb.domain

...

The ELB will divide the load among the sub-domains. With this sub-domain concept, we can virtually separate the cluster according to the task that each collection of nodes intends to perform. We defined a sub-domain called worker.

...

1. { hosts worker.esb.wso2.com; sub_domain worker; tenant_range *; } } }

In this file, we specified the domain name (wso2.esb.domain), which is used to identify the cluster. On startup, a node with this domain name will look for a cluster with this same domain name.

The ELB will divide the load among the sub-domains. With this sub-domain concept, we can virtually separate the cluster according to the task that each collection of nodes intends to perform. We defined a sub-domain called worker.

All the service requests need to be routed to the worker nodes through the ELB, which is the front end to the entire cluster. We used the hosts attribute to configure the publicly accessible host name (worker.esb.cloud-test.wso2.com), which clients can use to send their requests to the cluster. We will map the host name to the ELB server IP address later.

...

In summary, we have configured the load balancer to handle requests sent to worker.esb.cloud-test.wso2.com and to distribute the load among the worker nodes in the worker sub-domain of the wso2.esb.domain cluster. We are now ready to set up the cluster configurations.

...

1. Open the <ELB_HOME>/repository/conf/axis2/axis2.xml file.
2. Locate the Clustering section and verify or configure the properties as follows (some of these properties are already set correctly by default):
   1. Enable clustering for this node: <clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
   2. Set the membership scheme to wka to enable the Well Known Address registration method (this node will send cluster initiation messages to WKA members that we will define later): <parameter name="membershipScheme">wka</parameter>
   3. Specify a domain name for the ELB node (note that this domain it for potentially creating a cluster of ELB nodes and is not the cluster of ESB nodes that the ELB will load balance): <parameter name="domain">wso2.carbon.lb.domain</parameter>
   4. Specify the port used to communicate with this ELB node: <parameter name="localMemberPort">4000</parameter>
Note: This port number will not be affected by the port offset in carbon.xml. If this port number is already assigned to another server, the clustering framework will automatically increment this port number. However, if two servers are running on the same machine, you must ensure that a unique port is set for each server.

We have now completed the clustering-related configuration for the ELB. In the next section, we will make one last change to the ELB that will increase usability.

Configuring the ELB to listen on default ports

...

1. Open the <ELB_HOME>/repository/conf/axis2/axis2.xml file.
2. Locate the Transport Receiver section and configure the properties as follows:
   1. In

...

1. 1. the <transportReceiver name="http" class="org.apache.synapse.transport.passthru.PassThroughHttpListener">

...

1. 1.  transport, enable service requests to be sent to the ELB's default HTTP port instead of having to specify port 8280: <parameter name="port">80</parameter>
   2. In

...

1. 1. the <transportReceiver name="https" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLListener">

...

1. 1.  transport, enable service requests to be sent to the ELB's default HTTPS port instead of having to specify port 8243: <parameter name="port">443</parameter>

These port values should be left as 8280 and 8243 if you are a Linux user without root privileges, because binding to ports below 1024 requires root privileges.

In the next section, we will map the host names we specified to real IPs.

...

In the ELB, we configured a host name names in loadbalancer.conf to front the manager and worker service requests. We must now map this host name (esb.cloud-test.wso2.com) to these host names to the actual IP address. Open the server's /etc/hosts file and add the following linelines, where <ELP-IP> is the actual IP address:

<ELB-IP> worker.esb.cloud-testwso2.com 
<ELB-IP> mgt.esb.wso2.com 

In this example, it would look like this:

xxx.xxx.xxx.206 worker.esb.cloud-test.wso2.com 

...

xxx.xxx.xxx.206 mgt.esb.wso2.com

We have now finished configuring the ELB and are ready to start the ELB server.

...

• On the worker node, open <PRODUCT_HOME>/repository/conf/datasources/master-datasource.xml and configure the data sources to point to the REGISTRY_LOCAL2, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as follows (change the username, password, and database URL as needed for your environment):

  Code Block
  language html/xml
  <datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration"> <providers> <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider> </providers> <datasources> <datasource> <name>REGISTRY_LOCAL2</name> <description>The datasource used for registry- local</description> <jndiConfig> <name>jdbc/WSO2CarbonDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL2?autoReconnect=true</url> <username>regadmin</username> <password>regadmin</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource> <datasource> <name>REGISTRY_DB</name> <description>The datasource used for registry- config/governance</description> <jndiConfig> <name>jdbc/WSO2RegistryDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url> <username>regadmin</username> <password>regadmin</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource> <datasource> <name>WSO2_USER_DB</name> <description>The datasource used for registry and user manager</description> <jndiConfig> <name>jdbc/WSO2UMDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url> <username>regadmin</username> <password>regadmin</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource> </datasources> </datasources-configuration>

  Info
  Make sure to replace username and password with your MySQL database username and password.

Mounting the registry

Configure the shared registry database and mounting details in <PRODUCT_HOME>/repository/conf/registry.xml of the manager node as shown below:

<dbConfig name="sharedregistry">
	<dataSource>jdbc/WSO2RegistryDB</dataSource>
</dbConfig>

<remoteInstance url="https://localhost:9443/registry">
	<id>instanceid</id>
	<dbConfig>sharedregistry</dbConfig>
	<readOnly>false</readOnly>
	<enableCache>true</enableCache>
	<registryRoot>/</registryRoot>
	<cacheId>REGISTRY_DB@jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL1?autoReconnect=true</cacheId>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
	<instanceId>instanceid</instanceId>
	<targetPath>/_system/asNodes</targetPath>
</mount>

<mount path="/_system/governance" overwrite="true">
	<instanceId>instanceid</instanceId>
	<targetPath>/_system/governance</targetPath>
</mount>

...

• Info
  Make sure to replace username and password with your MySQL database username and password.

Mounting the registry

Configure the shared registry database and mounting details in <PRODUCT_HOME>/repository/conf/registry.xml of the manager node as shown below:

<dbConfig name="sharedregistry">
	<dataSource>jdbc/WSO2RegistryDB</dataSource>
</dbConfig>

<remoteInstance url="https://localhost:9443/registry">
	<id>instanceid</id>
	<dbConfig>sharedregistry</dbConfig>
	<readOnly>false</readOnly>
	<enableCache>true</enableCache>
	<registryRoot>/</registryRoot>
	<cacheId>REGISTRY_DB@jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL1?autoReconnect=true</cacheId>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
	<instanceId>instanceid</instanceId>
	<targetPath>/_system/asNodes</targetPath>
</mount>

<mount path="/_system/governance" overwrite="true">
	<instanceId>instanceid</instanceId>
	<targetPath>/_system/governance</targetPath>
</mount>

Configure the shared registry database and mounting details in <PRODUCT_HOME>/repository/conf/registry.xml of the worker node as shown below:

<dbConfig name="sharedregistry">
	<dataSource>jdbc/WSO2RegistryDB</dataSource>
</dbConfig>

<remoteInstance url="https://localhost:9443/registry">
	<id>instanceid</id>
	<dbConfig>sharedregistry</dbConfig>
	<readOnly>true</readOnly>
	<enableCache>true</enableCache>
	<registryRoot>/</registryRoot>
	<cacheId>REGISTRY_DB@jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL2?autoReconnect=true</cacheId>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
	<instanceId>instanceid</instanceId>
	<targetPath>/_system/asNodes</targetPath>
</mount>

<mount path="/_system/governance" overwrite="true">
	<instanceId>instanceid</instanceId>
	<targetPath>/_system/governance</targetPath>
</mount>

The following are some key points to note when adding these configurations:

• The registry mount path is used to identify the type of registry. For example, ”/_system/config” refers to configuration registry, and "/_system/governance" refers to the governance registry.
• The dbconfig entry enables you to identify the datasource you configured in the master-datasources.xml file. We use the unique name sharedregistry to refer to that datasource entry. 
• The remoteInstance section refers to an external registry mount. We can specify the read-only/read-write nature of this instance as well as caching configurations and the registry root location. In case of a worker node, the readOnly property should be true, and in case of a manager node, this property should be set to false. 
• You must define a unique name “id” for each remote instance, which is then referred to from mount configurations. In the above example, the unique ID for the remote instance is instanceId. 
• In each of the mounting configurations, we specify the actual mount path and target mount path. The targetPath can be any meaningful name. In this instance, it is /_system/asNodes.

Now your database is set up. The next step is to configure the manager and worker node.

Configuring the manager node 

In this section, we will configure data sources to allow the manager node to point to the central database, enable the manager node for clustering, change the port offset, and map the host names to IPs.

Configuring the data source

You configure datasources to allow the manager node to point to the central database. Make sure that you copy the database driver JAR to the manager node and follow the steps described in Setting up the central database.

Setting up cluster configurations for the manager node

Configuring clustering for the manager node is similar to the way you configured it for the ELB node, but the localMemberPort is 4001 instead of 4000, and you define the ELB node instead of the ESB manager node as the well-known member.

The following are some key points to note when adding these configurations:

• The registry mount path is used to identify the type of registry. For example, ”/_system/config” refers to configuration registry, and "/_system/governance" refers to the governance registry.
• The dbconfig entry enables you to identify the datasource you configured in the master-datasources.xml file. We use the unique name sharedregistry to refer to that datasource entry. 
• The remoteInstance section refers to an external registry mount. We can specify the read-only/read-write nature of this instance as well as caching configurations and the registry root location. In case of a worker node, the readOnly property should be true, and in case of a manager node, this property should be set to false. 
• You must define a unique name “id” for each remote instance, which is then referred to from mount configurations. In the above example, the unique ID for the remote instance is instanceId. 
• In each of the mounting configurations, we specify the actual mount path and target mount path. The targetPath can be any meaningful name. In this instance, it is /_system/asNodes.

Now your database is set up. The next step is to configure the manager and worker node.

Configuring the manager node 

In this section, we will configure data sources to allow the manager node to point to the central database, enable the manager node for clustering, change the port offset, and map the host names to IPs.

Configuring the data source

You configure datasources to allow the manager node to point to the central database. Make sure that you copy the database driver JAR to the manager node and follow the steps described in Setting up the central database.

Setting up cluster configurations for the manager node

Configuring clustering for the manager node is similar to the way you configured it for the ELB node, but the localMemberPort is 4001 instead of 4000, and you define the ELB node instead of the ESB manager node as the well-known member.

1. Open the <ESB_HOME>/repository/conf/axis2/axis2.xml file.
2. Locate the Clustering section and verify or configure the properties as follows (some of these properties are already set correctly by default):

• • Enable clustering for this node: <clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
  • Set the membership scheme to wka to enable the Well Known Address registration method (this node will send cluster initiation messages to WKA members that we will define later): <parameter name="membershipScheme">wka</parameter>
  • Specify the name of the cluster this node will join: <parameter name="domain">wso2.esb.domain</parameter>
  • Specify the port used to communicate cluster messages: <parameter name="localMemberPort">4001</parameter>
    Note: This port number will not be affected by the port offset in carbon.xml. If this port number is already assigned to another server, the clustering framework will automatically increment this port number. However, if two servers are running on the same machine, you must ensure that a unique port is set for each server.

  • The receiver's http/https port values are without the portOffset addition; they get auto-incremented by portOffset. The 'WSDLEPRPrefix' parameter should point to the worker node's host name (esb.cloud-test.wso2.com) and ELB's http (8280)/https (8243) transport ports.

  • Clear the members from the <members> element so that it is now empty:

    Code Block
    language html/xml
    <members> <member> <hostName>elb.wso2.com</hostName> <port>4000</port> </member> </members>

Configuring the port offset and host name

Because we are running two Carbon-based products on the same server, we must change the port offset to avoid port conflicts. Additionally, we will add the cluster host name so that any requests sent to the manager host are redirected to the cluster, where the ELB will pick them up and manage them.

1. Open <ESB_MANAGER_HOME>/repository/conf/carbon.xml.
2. Locate the <Ports> tag and change the value of its sub-tag to: <Offset>1</Offset>
3. Located the <HOSTNAME> tag and add the cluster host name: <HostName>esb.cloud-test.wso2.com</HostName>

In the next section, we will map the host names we specified to real IPs.

Mapping host names to IPs

In the manager node we have specified two host names: carbondb.mysql-wso2.com for the MySQL server and esb.cloud-test.wso2.com for the cluster. We will now map them to the actual IPs. Note that if you created the database on the same server as the manager node, you will have already added the first line, and if you created it on the same server as the ELB, you will have already added the second line.

Open the server's /etc/hosts file and add the following lines, where <MYSQL-DB-SERVER-IP> and <ELB-IP> are the actual IP addresses (in this example, xxx.xxx.xxx.206):

<MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com
<ELB-IP> esb.cloud-test.wso2.com
<ELB-IP> elb.wso2.com

In this example, it would look like this:

xxx.xxx.xxx.206 carbondb.mysql-wso2.com
xxx.xxx.xxx.206 esb.cloud-test.wso2.com
xxx.xxx.xxx.206 elb.wso2.com

We have now finished configuring the manager node and are ready to start the ESB server.

Starting the ESB server

Start the ESB server by typing the following command in the terminal:

sh <ESB_MANAGER_HOME>/bin/wso2server.sh -Dsetup 

The additional -Dsetup argument will clean the configurations, recreate the central database, and create the required tables in the database.

The ESB should print logs to the server console similar to the following:

INFO - TribesClusteringAgent Initializing cluster...
INFO - TribesClusteringAgent Cluster domain: wso2.esb.domain
INFO - TribesClusteringAgent Using wka based membership management scheme
INFO - WkaBasedMembershipScheme Receiver Server Socket bound to:/xxx.xxx.xxx.206:4001
INFO - WkaBasedMembershipScheme Receiver Server Socket bound to:/xxx.xxx.xxx.206:4001
INFO - TribesClusteringAgent Local Member 10.200.0.102:4001(wso2.esb.domain)
INFO - TribesUtil No members in current cluster
INFO - TribesClusteringAgent Cluster initialization completed

We have now finished configuring the manager node. Next, we will configure the ESB worker nodes.

Configuring the worker nodes

You configure worker nodes in very much the same way as you configured the manager node. Be sure to follow these steps for each worker node in the cluster. 

Configuring the data source

You configure the data source to connect to the central database. If there are multiple data sources, configure them to reference the central database as well. Since the data source changes are the same for all worker nodes, you can configure this file on one node and then copy it to the other worker nodes. Make sure you copy the database driver JAR to each worker node and follow the steps described in Setting up the central database.

After you have finished configuring the data source, be sure to copy this configuration to the other worker nodes in the cluster.

Setting up cluster configurations for the worker nodes

Configuring clustering for the worker nodes is similar to the way you configured it for the manager node, but the localMemberPort will vary for each worker node, you add the subDomain property, and you add the ELB and ESB manager node to the well-known members, as described in the following steps.

1. Open the <ESB_HOME>/repository/conf/axis2/axis2.xml file.
2. Locate the Clustering section and verify or configure the properties as follows (some of these properties are already set correctly by default):

• • Enable clustering for this node: <clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
  • Set the membership scheme to wka to enable the Well Known Address registration method (this node will send cluster initiation messages to WKA members that we will define later): <parameter name="membershipScheme">wka</parameter>
  • Specify the name of the cluster this node will join: <parameter name="domain">wso2.esb.domain</parameter>
  • Specify the port used to communicate cluster messages (if this node is on the same server as the ELB, manager node, or another worker node, be sure to set this to a unique value, such as 4000 and 4001 for worker nodes 1 and 3 and 4002 for worker node 2, which is on the same server as the ELB and manager node): <parameter name="localMemberPort">4000</parameter>
    Note: This port number will not be affected by the port offset in carbon.xml. If this port number is already assigned to another server, the clustering framework will automatically increment this port number. However, if two servers are running on the same machine, you must ensure that a unique port is set for each server.
  • Define the sub-domain as worker by adding the following property under the <parameter name="properties"> element: <property name="subDomain" value="worker"/>

  • Define the ELB and manager nodes as well-known members of the cluster by providing their host name and localMemberPort values. The manager node is defined here because it is required for the Deployment Synchronizer to function.

    Code Block
    language html/xml
    <members> <member> <hostName>elb.wso2.com</hostName> <port>4000</port> </member> <member> <hostName>mgt.esb.wso2.com</hostName> <port>4001</port> </member> </members>

Adjusting the port offset

Because we are running two Carbon-based products on the same server, we must change the port offset to avoid port conflicts.

1. Open <ESB_WORKER_HOME>/repository/conf/carbon.xml on each worker node.
2. Locate the <Ports> tag and change the value of its sub-tag as follows on each worker node:

• • Worker1: <Offset>0</Offset> - No changes needed, because this will be the first node on this (xxx.xxx.xxx.132) server.

  • Worker2: <Offset>2</Offset> - Set the offset to 2, because there are already two more Carbon products (ELB and ESB manager node) running on this (xxx.xxx.xxx.206) server.

  • Worker3: <Offset>3</Offset> - Set the offset of 3, because Worker1 occupies the default ports on this (xxx.xxx.xxx.132) server.

In the next section, we will map the host names we specified to real IPs.

Mapping host names to IPs

In the worker nodes, we have used three hosts names: carbondb.mysql-wso2.com for the MySQL server, elb.wso2.com for the ELB, and mgt.esb.wso2.com for the ESB manager node. We will now map them to the actual IPs.

Open the server's /etc/hosts file and add the following lines, where <MYSQL-DB-SERVER-IP>, <ELB-IP>, and <ESB-Manager-IP> are the actual IP addresses (in this example, xxx.xxx.xxx.206):

<MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com
<ELB-IP> elb.wso2.com
<ESB-Manager-IP> mgt.esb.wso2.com 

In this example, it would look like this:

xxx.xxx.xxx.206 carbondb.mysql-wso2.com
xxx.xxx.xxx.206 elb.wso2.com
xxx.xxx.xxx.206 mgt.esb.wso2.com 

We have now finished configuring the worker nodes and are ready to start them.

Starting the ESB server

Start the ESB server by typing the following command in the terminal:

sh <ESB_WORKER_HOME>/bin/wso2server.sh -DworkerNode=true 

...

1. Open the <ESB_HOME>/repository/conf/axis2/axis2.xml file.
2. Locate the Clustering section and verify or configure the properties as follows (some of these properties are already set correctly by default):
   
   1. Enable clustering for this node: <clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
   2. Set the membership scheme to wka to enable the Well Known Address registration method (this node will send cluster initiation messages to WKA members that we will define later): <parameter name="membershipScheme">wka</parameter>
   3. Specify the name of the cluster this node will join: <parameter name="domain">wso2.esb.domain</parameter>
   4. Specify the name of the cluster this node will join (this is the domain defined in the loadbalancer.conf file on the ELB):
      <parameter name="domain">wso2.esb.domain</parameter>

   5. Specify the port used to communicate cluster messages: <parameter name="localMemberPort">4001</parameter>
      Note: This port number will not be affected by the port offset in carbon.xml. If this port number is already assigned to another server, the clustering framework will automatically increment this port number. However, if two servers are running on the same machine, you must ensure that a unique port is set for each server.

   6. The receiver's http/https port values are without the portOffset addition; they get auto-incremented by portOffset. The 'WSDLEPRPrefix' parameter should point to the worker node's host name (worker.esb.wso2.com) and ELB's http (8280)/https (8243) transport ports.

   7. Ensure that you set the value of the subDomain as mgt to specify that this is the manager node, which will ensure that traffic for the manager node is routed to this member.
      <property name="subDomain" value="mgt"/>
      

   8. Edit the <members> element so that it looks as follows:

      Code Block
      language html/xml

...

<dbConfig name="sharedregistry">
	<dataSource>jdbc/WSO2RegistryDB</dataSource>
</dbConfig>

<remoteInstance url="https://localhost:9443/registry">
	<id>instanceid</id>
	<dbConfig>sharedregistry</dbConfig>
	<readOnly>true</readOnly>
	<enableCache>true</enableCache>
	<registryRoot>/</registryRoot>
	<cacheId>REGISTRY_DB@jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL2?autoReconnect=true</cacheId>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
	<instanceId>instanceid</instanceId>
	<targetPath>/_system/asNodes</targetPath>
</mount>

<mount path="/_system/governance" overwrite="true">
	<instanceId>instanceid</instanceId>
	<targetPath>/_system/governance</targetPath>
</mount>

1. 1. <members> <member> <hostName>elb.wso2.com</hostName> <port>4000</port> </member> </members>

2. Locate the port mapping section and configure the properties as follows:
   <property name="port.mapping.80" value="9764"/>
<property name="port.mapping.443" value="9444"/>
   

   Note
   This configuration will change as follows if you did not configure the ELB to listen on default ports: Code Block language html/xml <property name="port.mapping.8280" value="9764"/> <property name="port.mapping.8243" value="9444"/>

   Info

   This value should increment based on the port offset value. In this example it is incremented by 1 since the port offset for the manager node is one. In a dynamically clustered set up where you front a WSO2 Carbon instance using a WSO2 ELB, it is the responsibility of a Carbon server to send its information to ELB. You can visualize this as a "member object somehow getting passed to ELB from the Carbon server instance". In the Carbon server's clustering section, under properties, you can define any member property. This way, you can let ELB know about the information other than the basic ones. Typically, this basic information includes host names, HTTP port, HTTPS port, etc. WSO2 ESB, WSO2 API Manager etc. are somewhat special with regard to ports as they usually have two HTTP ports (compared to one HTTP port for products like WSO2 AS). Hence, here we have to somehow send this additional information to ELB. The easiest way to do this is by setting a member property. Here, we use port.mapping property. Also, in order to front these special servers, we need two HTTP ports in ELB too, which are exposed to the outside. There's a deployment decision to be made here, i.e., which HTTP port of ELB should map to which HTTP port of the server (i.e., servlet HTTP port or NHTTP HTTP port). With that in mind, let's consider only the HTTP scenario. Say, in your ESB instance, you have used 8280 as the NHTTP transport port (axis2.xml) and 9763 as the Servlet transport port (catalina-server.xml). Also, ELB has 2 HTTP ports, one is 8280 and the other is 8290. Imagine there's a member object, and in this case, the member's HTTP port would be 8280 (usually the port defined in axis2.xml gets here). But since ELB has 2 ports, there's no way to correctly map ports, by only specifying member's HTTP port. There arises the importance of port.mapping property. You have to think of this property from the perspective of ELB. Let's assume we define the above property, now this means, if a request comes to ELB, in its 8290 port (see... we're thinking from ELB's perspective), forward that request to the 9764 port of the Member. Having only this property is enough, we do not need following property: Code Block language html/xml <property name="port.mapping.8280" value="8280"></property> This occurs because the logic was written in a way that port.mapping properties get higher precedence over the default ports. This means, that when a request comes to ELB, ELB will first check whether the port it received the request from is specified as a port.mapping property. If it is, it will grab the target port from that property. If not, it will send the request to the default http port. Hence, if a request is received by the 8280 port of ELB, it will be automatically get redirected to 8280 port of the Member (since it's the HTTP port of Member). Similarly, we should define a mapping for https servlet port (8243).

Configuring the port offset and host name

Because we are running two Carbon-based products on the same server, we must change the port offset to avoid port conflicts. Additionally, we will add the cluster host name so that any requests sent to the manager host are redirected to the cluster, where the ELB will pick them up and manage them.

1. Open <ESB_MANAGER_HOME>/repository/conf/carbon.xml.
2. Locate the <Ports> tag and change the value of its sub-tag to: <Offset>1</Offset>
3. Located the <HOSTNAME> tag and add the cluster host name: <HostName>worker.esb.wso2.com</HostName>

In the next section, we will map the host names we specified to real IPs.

Mapping host names to IPs

In the manager node we have specified two host names: carbondb.mysql-wso2.com for the MySQL server and worker.esb.wso2.com for the cluster. We will now map them to the actual IPs. Note that if you created the database on the same server as the manager node, you will have already added the first line, and if you created it on the same server as the ELB, you will have already added the second line.

Open the server's /etc/hosts file and add the following lines, where <MYSQL-DB-SERVER-IP> and <ELB-IP> are the actual IP addresses (in this example, xxx.xxx.xxx.206):

<MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com
<ELB-IP> worker.esb.wso2.com
<ELB-IP> elb.wso2.com

In this example, it would look like this:

xxx.xxx.xxx.206 carbondb.mysql-wso2.com
xxx.xxx.xxx.206 worker.esb.wso2.com
xxx.xxx.xxx.206 elb.wso2.com

We have now finished configuring the manager node and are ready to start the ESB server.

Starting the ESB server

Start the ESB server by typing the following command in the terminal:

sh <ESB_MANAGER_HOME>/bin/wso2server.sh -Dsetup 

The additional -Dsetup argument will clean the configurations, recreate the central database, and create the required tables in the database.

The ESB should print logs to the server console similar to the following:

INFO - TribesClusteringAgent Initializing cluster...
INFO - TribesClusteringAgent Cluster domain: wso2.esb.domain
INFO - TribesClusteringAgent Using wka based membership management scheme
INFO - WkaBasedMembershipScheme Receiver Server Socket bound to:/xxx.xxx.xxx.206:4001
INFO - WkaBasedMembershipScheme Receiver Server Socket bound to:/xxx.xxx.xxx.206:4001
INFO - TribesClusteringAgent Local Member 10.200.0.102:4001(wso2.esb.domain)
INFO - TribesUtil No members in current cluster
INFO - TribesClusteringAgent Cluster initialization completed

We have now finished configuring the manager node. Next, we will configure the ESB worker nodes.

Configuring the worker nodes

You configure worker nodes in very much the same way as you configured the manager node. Be sure to follow these steps for each worker node in the cluster. 

Configuring the data source

You configure the data source to connect to the central database. If there are multiple data sources, configure them to reference the central database as well. Since the data source changes are the same for all worker nodes, you can configure this file on one node and then copy it to the other worker nodes. Make sure you copy the database driver JAR to each worker node and follow the steps described in Setting up the central database.

After you have finished configuring the data source, be sure to copy this configuration to the other worker nodes in the cluster.

Setting up cluster configurations for the worker nodes

Configuring clustering for the worker nodes is similar to the way you configured it for the manager node, but the localMemberPort will vary for each worker node, you add the subDomain property, and you add the ELB and ESB manager node to the well-known members, as described in the following steps.

1. Open the <ESB_HOME>/repository/conf/axis2/axis2.xml file.
2. Locate the Clustering section and verify or configure the properties as follows (some of these properties are already set correctly by default):
   1. Enable clustering for this node:
      <clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
   2. Set the membership scheme to wka to enable the Well Known Address registration method (this node will send cluster initiation messages to WKA members that we will define later):
      <parameter name="membershipScheme">wka</parameter>
   3. Specify the name of the cluster this node will join:
      <parameter name="domain">wso2.esb.domain</parameter>

   4. Specify the port used to communicate cluster messages (if this node is on the same server as the ELB, manager node, or another worker node, be sure to set this to a unique value, such as 4000 and 4001 for worker nodes 1 and 3 and 4002 for worker node 2, which is on the same server as the ELB and manager node):
      <parameter name="localMemberPort">4000</parameter>

      Note
      Note: This port number will not be affected by the port offset in carbon.xml. If this port number is already assigned to another server, the clustering framework will automatically increment this port number. However, if two servers are running on the same machine, you must ensure that a unique port is set for each server.

   5. Define the sub-domain as worker by adding the following property under the <parameter name="properties"> element: <property name="subDomain" value="worker"/>

   6. Define the ELB and manager nodes as well-known members of the cluster by providing their host name and localMemberPort values. The manager node is defined here because it is required for the Deployment Synchronizer to function.

      Code Block
      language html/xml
      <members> <member> <hostName>elb.wso2.com</hostName> <port>4000</port> </member> <member> <hostName>mgt.esb.wso2.com</hostName> <port>4001</port> </member> </members>

Adjusting the port offset

Because we are running two Carbon-based products on the same server, we must change the port offset to avoid port conflicts. When setting the port offset, note that this results in the port values all being incremented by the offset value which is how port conflicts are avoided.

1. Open <ESB_WORKER_HOME>/repository/conf/carbon.xml on each worker node.
2. Locate the <Ports> tag and change the value of its sub-tag as follows on each worker node:

• • Worker1: <Offset>0</Offset> - No changes needed, because this will be the first node on this (xxx.xxx.xxx.132) server.

  • Worker2: <Offset>2</Offset> - Set the offset to 2, because there are already two more Carbon products (ELB and ESB manager node) running on this (xxx.xxx.xxx.206) server.

  • Worker3: <Offset>3</Offset> - Set the offset of 3, because Worker1 occupies the default ports on this (xxx.xxx.xxx.132) server.

In the next section, we will map the host names we specified to real IPs.

Mapping host names to IPs

In the worker nodes, we have used three hosts names: carbondb.mysql-wso2.com for the MySQL server, elb.wso2.com for the ELB, and mgt.esb.wso2.com for the ESB manager node. We will now map them to the actual IPs.

Open the server's /etc/hosts file and add the following lines, where <MYSQL-DB-SERVER-IP>, <ELB-IP>, and <ESB-Manager-IP> are the actual IP addresses (in this example, xxx.xxx.xxx.206):

<MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com
<ELB-IP> elb.wso2.com
<ESB-Manager-IP> mgt.esb.wso2.com 

In this example, it would look like this:

xxx.xxx.xxx.206 carbondb.mysql-wso2.com
xxx.xxx.xxx.206 elb.wso2.com
xxx.xxx.xxx.206 mgt.esb.wso2.com 

We have now finished configuring the worker nodes and are ready to start them.

Starting the ESB server

Start the ESB server by typing the following command in the terminal:

sh <ESB_WORKER_HOME>/bin/wso2server.sh -DworkerNode=true 

The additional -DworkerNode=true argument indicates that this is a worker node. This parameter basically makes a server read-only. A node with this parameter will not be able to do any changes such as writing or making modifications to the deployment repository etc.

....
   -Dfile.encoding=UTF8 \
    -DworkerNode=true \
    org.wso2.carbon.bootstrap.Bootstrap $*
    status=$?
......

What you configure the axis2.xml (under the clustering section), the cluster sub domain must indicate that this node belongs to the "worker" sub domain in the cluster.

When starting the Worker1, it should display logs similar to the following in the console:

...