...
The ELB will divide the load among the sub-domains. With this sub-domain concept, we can virtually separate the cluster , according to the task that each collection of nodes intends to perform. We defined a sub-domain called worker
.
In the previous diagram, you can see that all All the service requests need to be routed to the worker nodes through the ELB, which is the front end to the entire cluster. We used the hosts
attribute to configure the the publicly accessible host name (esb.cloud-test.wso2.com
), which clients can use to send their requests to the cluster. We will map the host name to the ELB server IP address later.
...
- Open the
<ELB_HOME>/repository/conf/axis2/axis2.xml
file. - Locate the Clustering section and verify or configure the properties as follows (some of these properties are already set correctly by default):
- Enable clustering for this node:
<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
- Set the membership scheme to
wka
to enable the Well Known Address registration method (this node will send cluster initiation messages to WKA members that we will define later):<parameter name="membershipScheme">wka</parameter>
- Specify the name of the cluster this node will joina domain name for the ELB node (note that this domain it for potentially creating a cluster of ELB nodes and is not the cluster of ESB nodes that the ELB will load balance):
<parameter name="domain">wso2.esbcarbon.lb.domain</parameter>
Specify the port used to communicate
cluster messageswith this ELB node:
<parameter name="localMemberPort">4000</parameter>
Note: This port number will not be affected by the port offset incarbon.xml
. If this port number is already assigned to another server, the clustering framework will automatically increment this port number. However, if two servers are running on the same machine, you must ensure that a unique port is set for each server.Define the ESB manager node as a well-known member of the cluster by providing its host name and its
localMemberPort
(you will configure these on the manager node later):Code Block language html/xml <members> <member> <hostName>mgr.esb.wso2.com</hostName> <port>4001</port> </member> </members>
- Enable clustering for this node:
We have now completed the clustering-related configuration for the ELB. In the next section, we will make one last change to the ELB that will increase usability.
Configuring the ELB to listen on default ports
We will now change the ELB configuration to listen to the default HTTP and HTTPS ports.
...
We have now completed the clustering-related configuration for the ELB. In the next section, we will make one last change to the ELB that will increase usability.
Configuring the ELB to listen on default ports
We will now change the ELB configuration to listen to the default HTTP and HTTPS ports.
- Open the
<ELB_HOME>/repository/conf/axis2/axis2.xml
file. - Locate the Transport Receiver section and configure the properties as follows:
- In the
<transportReceiver name="http" class="org.apache.synapse.transport.passthru.PassThroughHttpListener">
transport, enable service requests to be sent to the ELB's default HTTP port instead of having to specify port 8280:<parameter name="port">80</parameter>
- In the
<transportReceiver name="https" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLListener">
transport, enable service requests to be sent to the ELB's default HTTPS port instead of having to specify port 8243:<parameter name="port">443</parameter>
- In the
In the next section, we will map the host names we specified to real IPs.
Mapping the host
...
name to
...
the IP
In the ELB we have specified two host names: , we configured a host name in loadbalancer.conf
to front the worker service requests. We must now map this host name (esb.cloud-test.wso2.com
for worker hosts and mgr.esb.wso2.com
for the manager node. We will now map them ) to the actual IPsIP address. Open the server's /etc/hosts
file and add the following linesline, where <ELP-IP>
and <ESB-Manager-IP>
are is the actual IP addresses (in this example, xxx.xxx.xxx.206)address:
Code Block | ||
---|---|---|
| ||
<ELB-IP> esb.cloud-test.wso2.com
<ESB-Manager-IP> mgr.esb.wso2.com |
In this example, it would look like this:
Code Block | ||
---|---|---|
| ||
xxx.xxx.xxx.206 esb.cloud-test.wso2.com xxx.xxx.xxx.206 mgr.esb.wso2.com |
We have now finished configuring the ELB and ready to start the ELB server.
...
Code Block | ||
---|---|---|
| ||
INFO - TribesClusteringAgent Initializing cluster... INFO - TribesClusteringAgent Cluster domain: wso2.carbon.esblb.domain INFO - TribesClusteringAgent Using wka based membership management scheme INFO - WkaBasedMembershipScheme Receiver Server Socket bound to:/xxx.xxx.xxx.206:4000 INFO - WkaBasedMembershipScheme Receiver Server Socket bound to:/xxx.xxx.xxx.206:4000 INFO - TribesClusteringAgent Local Member xxx.xxx.xxx.206:4000(wso2.carbon.esblb.domain) INFO - TribesUtil No members in current cluster INFO - TribesClusteringAgent Cluster initialization completed. |
Now that the ELB is configured and running, you create a central database for all the nodes to use.
...
Info |
---|
In most WSO2 products, only one data source is used. If there is more than one data source, make sure they reference the central databases accordingly. For example, the API Manager deployment setup requires more specific data source configurations, so it is described in a different section below. |
Setting up cluster configurations for the manager node
Configuring clustering for the manager node is very similar to the way you configured it for the ELB node, but the localMemberPort
is 4001 instead of 4000, and you define the ELB node instead of the ESB manager node as the well-known member.
- Open the
<ESB_HOME>/repository/conf/axis2/axis2.xml
file. - Locate the Clustering section and verify or configure the properties as follows (some of these properties are already set correctly by default):
- Enable clustering for this node:
<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
- Set the membership scheme to
wka
to enable the Well Known Address registration method (this node will send cluster initiation messages to WKA members that we will define later):<parameter name="membershipScheme">wka</parameter>
- Specify the name of the cluster this node will join:
<parameter name="domain">wso2.esb.domain</parameter>
- Specify the port used to communicate cluster messages:
<parameter name="localMemberPort">4001</parameter>
Note: This port number will not be affected by the port offset incarbon.xml
. If this port number is already assigned to another server, the clustering framework will automatically increment this port number. However, if two servers are running on the same machine, you must ensure that a unique port is set for each server. Define the ELB node as a well-known member of the cluster by providing its host name and its
localMemberPort
Clear the members from the<members>
element so that it is now empty:Code Block language html/xml <members> <member> <hostName>elb.wso2.com</hostName> <port>4000</port> </member> </members>
- Enable clustering for this node:
...
Configuring the port offset and host name
Because we are running two Carbon-based products on the same server, we must change the port offset to avoid port conflicts.
...
Additionally, we will add the cluster host name so that any requests sent to the manager host are redirected to the cluster, where the ELB will pick them up and manage them.
- Open
<ESB_MANAGER_HOME>/repository/conf/carbon.xml
. - Locate the
<Ports>
tag and change the value of its sub-tag to:<Offset>1</Offset>
- Located the
<HOSTNAME>
tag and add the cluster host name:<HostName>esb.cloud-test.wso2.com</HostName>
In the next section, we will map the host names we specified to real IPs.
Mapping host names to IPs
In the manager node we have specified two host names: carbondb.mysql-wso2.com
for the MySQL server and elbesb.cloud-test.wso2.com
for the ELBcluster. We will now map them to the actual IPs. Note that if you created the database on the same server as the manager node, you will have already added the first line, and if you created it on the same server as the ELB, you will have already added the second line.
Open the server's /etc/hosts
file and add the following lines, where <MYSQL-DB-SERVER-IP>
and <ELB-IP>
are the actual IP addresses (in this example, xxx.xxx.xxx.206):
Code Block | ||
---|---|---|
| ||
<MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com
<ELB-IP> elb.wso2.com |
In this example, it would look like this:
Code Block | ||
---|---|---|
| ||
xxx.xxx.xxx.206 carbondb.mysql-wso2.com
xxx.xxx.xxx.206 elb.wso2.com |
Note that if you created the database on the same server as the manager node, you may have already added the first line.
We have now finished configuring the manager node and are ready to start the ESB server.
Starting the ESB server
Start the ESB server by typing the following command in the terminal:
sh <ESB_MANAGER_HOME>/bin/wso2server.sh -Dsetup
The additional -Dsetup
argument will clean the configurations, recreate the central database, and create the required tables in the database.
The ESB should print logs to the server console similar to the following:
...
language | none |
---|
...
this example, xxx.xxx.xxx.206
...
):
Code Block | ||
---|---|---|
| ||
<MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com
<ELB-IP> esb.cloud-test.wso2.com |
In this example, it would look like this:
Code Block | ||
---|---|---|
| ||
xxx.xxx.xxx.206 carbondb.mysql-wso2.com xxx.xxx.xxx.206:4000(wso2.esb.domain) INFO - TribesClusteringAgent Trying to send initialization request to xxx.xxx.xxx.206:4000(wso2.esb.domain) |
...
esb.cloud-test.wso2.com |
We have now finished configuring the manager node and are ready to start the ESB server.
Starting the ESB server
Start the ESB server by typing the following command in the terminal:
sh <ESB_MANAGER_HOME>/bin/wso2server.sh -Dsetup
The additional -Dsetup
argument will clean the configurations, recreate the central database, and create the required tables in the database.
The ESB should print logs to the server console similar to the following:
Code Block | ||
---|---|---|
| ||
INFO - RpcMembershipRequestHandlerTribesClusteringAgent Received JOIN message from xxx.xxx.xxx.206:4001(wso2.esb.domain)Initializing cluster... INFO - MembershipManager Application member xxx.xxx.xxx.206:4001(wso2.esb.domain) joined group wso2.esb.domain INFO - TribesMembershipListener New member xxx.xxx.xxx.206:4001(TribesClusteringAgent Cluster domain: wso2.esb.domain) joined cluster. INFO - RpcInitializationRequestHandler Received GetConfigurationCommand initialization request message from INFO - TribesClusteringAgent Using wka based membership management scheme INFO - WkaBasedMembershipScheme Receiver Server Socket bound to:/xxx.xxx.xxx.206:4001(wso2.esb.domain) INFO - WkaBasedMembershipScheme Receiver DefaultGroupManagementAgentServer ApplicationSocket memberbound Hostto:/xxx.xxx.xxx.206,:4001 INFO Port:- 4001, HTTP:8280, HTTPS:8243, Domain: TribesClusteringAgent Local Member 10.200.0.102:4001(wso2.esb.domain,) INFO Sub-domain:null, Active:true joined application cluster- TribesUtil No members in current cluster INFO - TribesClusteringAgent Cluster initialization completed |
We have now finished configuring the manager node. Next, we will configure the ESB worker nodes.
...
- Open the
<ESB_HOME>/repository/conf/axis2/axis2.xml
file. - Locate the Clustering section and verify or configure the properties as follows (some of these properties are already set correctly by default):
- Enable clustering for this node:
<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
- Set the membership scheme to
wka
to enable the Well Known Address registration method (this node will send cluster initiation messages to WKA members that we will define later):<parameter name="membershipScheme">wka</parameter>
- Specify the name of the cluster this node will join:
<parameter name="domain">wso2.esb.domain</parameter>
- Specify the port used to communicate cluster messages (if this node is on the same server as the ELB, manager node, or another worker node, be sure to set this to a unique value, such as 4000 and 4001 for worker nodes 1 and 3 and 4002 for worker node 2, which is on the same server as the ELB and manager node):
<parameter name="localMemberPort">4000</parameter>
Note: This port number will not be affected by the port offset incarbon.xml
. If this port number is already assigned to another server, the clustering framework will automatically increment this port number. However, if two servers are running on the same machine, you must ensure that a unique port is set for each server. - Define the sub-domain as worker by adding the following property under the
<parameter name="properties">
element:<property name="subDomain" value="worker"/>
Define the ELB and manager nodes as well-known member members of the cluster by providing their host name and
localMemberPort
values:cluster by providing their host name andlocalMemberPort
values. The manager node is defined here because it is required by the Deployment Synchronizer.Code Block language html/xml <members> <member> <hostName>elb.wso2.com</hostName> <port>4000</port> </member> <member> <hostName>mgr<hostName>mgt.esb.wso2.com</hostName> <port>4001</port> </member> </members>
- Enable clustering for this node:
Adjusting the port offset
Because we are running two Carbon-based products on the same server, we must change the port offset to avoid port conflicts.
...
In the next section, we will map the host names we specified to real IPs.
Mapping host names to IPs
In the worker nodes, we have used three hosts names: carbondb.mysql-wso2.com
for the MySQL server, elb.wso2.com
for the ELB, and mgr.esb.wso2.com
for the ESB manager node. We will now map them to the actual IPs.
...
Code Block | ||
---|---|---|
| ||
<MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com <ELB-IP> elb.wso2.com <ESB-Manager-IP> mgrmgt.esb.wso2.com |
In this example, it would look like this:
Code Block | ||
---|---|---|
| ||
xxx.xxx.xxx.206 carbondb.mysql-wso2.com xxx.xxx.xxx.206 elb.wso2.com xxx.xxx.xxx.206 mgrmgt.esb.wso2.com |
We have now finished configuring the worker nodes and are ready to start them.
...
Code Block | ||
---|---|---|
| ||
INFO - TribesClusteringAgent Initializing cluster... INFO - TribesClusteringAgent Cluster domain: wso2.esb.domain INFO - TribesClusteringAgent Using wka based membership management scheme INFO - WkaBasedMembershipScheme Receiver Server Socket bound to:/xxx.xxx.xxx.132:4000 INFO - WkaBasedMembershipScheme Receiver Server Socket bound to:/xxx.xxx.xxx.132:4000 INFO - WkaBasedMembershipScheme Added static member xxx.xxx.xxx.206:4000(wso2.esb.domain) INFO - WkaBasedMembershipScheme Added static member xxx.xxx.xxx.206:4001(wso2.esb.domain) INFO - TribesClusteringAgent Local Member xxx.xxx.xxx.132:4000(wso2.esb.domain) INFO - TribesUtil Members of current cluster INFO - TribesUtil Member1 xxx.xxx.xxx.206:4000(wso2.esb.domain) INFO - TribesUtil Member2 xxx.xxx.xxx.206:4001(wso2.esb.domain) INFO - WkaBasedMembershipScheme Sending JOIN message to WKA members... INFO - RpcMembershipRequestHandler Received MEMBER_LIST message from xxx.xxx.xxx.206:4000(wso2.esb.domain) INFO - TribesClusteringAgent Trying to send initialization request to xxx.xxx.xxx.206:4000(wso2.esb.domain) INFO - GetConfigurationResponseCommand Received configuration initialization message INFO - TribesClusteringAgent Cluster initialization completed. |
...
Code Block | ||
---|---|---|
| ||
INFO - RpcMembershipRequestHandler Received JOIN message from from xxx.xxx.xxx.132:4000(wso2.esb.domain) INFO - MembershipManager Application member xxx.xxx.xxx.132:4000(wso2.esb.domain) joined group wso2.esb.domain INFO - MembershipManagerDefaultGroupManagementAgent Application member Host:xxx.xxx.xxx.132, Port:4000( 4000, HTTP:8280, HTTPS:8243, Domain: wso2.esb.domain)domain, Sub-domain:worker, Active:true joined group wso2.esb.domain INFO - TribesMembershipListener New member application clustert |
The manager node console should have these new messages:
Code Block | ||
---|---|---|
| ||
INFO - RpcMembershipRequestHandler Received JOIN message from xxx.xxx.xxx.132:4000(wso2.esb.domain) INFO - joinedRpcInitializationRequestHandler cluster.Received INFOGetConfigurationCommand -initialization DefaultGroupManagementAgentrequest Applicationmessage memberfrom Host:xxx.xxx.xxx.132, Port: 4000, HTTP:8280, HTTPS:8243, Domain: 4000(wso2.esb.domain, Sub-domain:worker, Active:true joined application cluster |
...
) |
If you have similar messages in your consoles, you have finished configuring the worker nodes and the cluster is running. When you terminate one node, all nodes identify that the node has left the cluster. The same applies when a new node joins the cluster. If you want to add another new worker node, you can simply copy worker1 without any changes if you are running it on a new server (such as xxx.xxx.xxx.184). If you intend to use the new node on a server where another WSO2 product is running, you can use a copy of worker1 and change the port offset accordingly in the carbon.xml
file. You may also have to change localMemberPort
in axis2.xml
if that product has clustering enabled. Be sure to map all host names to the relevant IP addresses when creating a new node.
...