Page Comparison

This topic provides instructions on how to cluster WSO2 Enterprise Service Bus (ESB) using WSO2 Elastic Load Balancer (ELB), but you can use a third-party load balancer in its place (for configuration details, see your load balancer's documentation).

...

In this scenario, we are using WSO2 ELB for the load balancer (you can also use a third-party load balancer; for details on configuration, see your load balancer's documentation). You configure the ELB with the overall definition of the cluster and how it should distribute the load. You can achieve this by adding a few lines to a configuration file called loadbalancer.conf. You specify the detailed clustering configurations in the axis2.xml file. This section describes how to perform these steps.

Setting up the database

Each Carbon-based product uses a database to store information such as user management details and registry data. Set up the databases to store information and work hand-in-hand with your cluster.

...

1. Open the <ESB_HOME>/repository/conf/axis2/axis2.xml file.
2. Locate the Clustering section and verify or configure the properties as follows (some of these properties are already set correctly by default):
   
   1. Enable clustering for this node: 
      <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
   2. Set the membership scheme to wka to enable the Well Known Address registration method (this node will send cluster initiation messages to WKA members that we will define later): 
      <parameter name="membershipScheme">wka</parameter>
   3. Specify the name of the cluster this node will join (this is the domain defined in the loadbalancer.conf file on the ELB):
      <parameter name="domain">wso2.esb.domain</parameter>
   4. Specify the host used to communicate cluster messages:
      <parameter name="localMemberHost">xxx.xxx.xxx.206</parameter>

   5. Specify the port used to communicate cluster messages: 
      <parameter name="localMemberPort">4001</parameter>

      Note
      Note: This port number will not be affected by the port offset in carbon.xml. If this port number is already assigned to another server, the clustering framework will automatically increment this port number. However, if two servers are running on the same machine, you must ensure that a unique port is set for each server.

      
      

   6. The receiver's http/https port values are without the portOffset addition; they get auto-incremented by portOffset. The 'WSDLEPRPrefix' parameter should point to the worker node's host name (esb.cloud-test.wso2.com) and ELB's http (8280)/https (8243) transport ports.

   7. Ensure that you set the value of the subDomain as mgt to specify that this is the manager node, which will ensure that traffic for the manager node is routed to this member.
      <propertyname="subDomain"value="mgt"/>

   8. Edit the <members> element so that it looks as follows:
      

      Code Block
      language html/xml
      <members> <member> <hostName>xxx.xxx.xxx.206</hostName> <port>4500</port> </member> </members>

      Info
      The IP address mentioned in the hostName represents the IP of the ELB.

      
      

3. Locate the port mapping section and configure the properties as follows:

   <property name="port.mapping.80" value="9764"/>
<property name="port.mapping.443" value="9444"/>
   

   Note
   This configuration will change as follows if you did not configure the ELB to listen on default ports: Code Block language html/xml <property name="port.mapping.8280" value="9764"/> <property name="port.mapping.8243" value="9444"/>

   Info

   This value should increment based on the port offset value. In this example it is incremented by 1 since the port offset for the manager node is one. In a dynamically clustered set up where you front a WSO2 Carbon instance using a WSO2 ELB, it is the responsibility of a Carbon server to send its information to ELB. You can visualize this as a "member object somehow getting passed to ELB from the Carbon server instance". In the Carbon server's clustering section, under properties, you can define any member property. This way, you can let ELB know about the information other than the basic ones. Typically, this basic information includes host names, HTTP port, HTTPS port, etc. WSO2 ESB, WSO2 API Manager etc. are somewhat special with regard to ports as they usually have two HTTP ports (compared to one HTTP port for products like WSO2 AS). Hence, here we have to somehow send this additional information to ELB. The easiest way to do this is by setting a member property. Here, we use port.mapping property. Also, in order to front these special servers, we need two HTTP ports in ELB too, which are exposed to the outside. There's a deployment decision to be made here, i.e., which HTTP port of ELB should map to which HTTP port of the server (i.e., servlet HTTP port or NHTTP HTTP port). With that in mind, let's consider only the HTTP scenario. Say, in your ESB instance, you have used 8280 as the NHTTP transport port (axis2.xml) and 9763 as the Servlet transport port (catalina-server.xml). Also, ELB has 2 HTTP ports, one is 8280 and the other is 8290. Imagine there's a member object, and in this case, the member's HTTP port would be 8280 (usually the port defined in axis2.xml gets here). But since ELB has 2 ports, there's no way to correctly map ports, by only specifying member's HTTP port. There arises the importance of port.mapping property. You have to think of this property from the perspective of ELB. Let's assume we define the above property, now this means, if a request comes to ELB, in its 8290 port (see... we're thinking from ELB's perspective), forward that request to the 9764 port of the Member. Having only this property is enough, we do not need following property: Code Block language html/xml <property name="port.mapping.8280" value="8280"></property> This occurs because the logic was written in a way that port.mapping properties get higher precedence over the default ports. This means, that when a request comes to ELB, ELB will first check whether the port it received the request from is specified as a port.mapping property. If it is, it will grab the target port from that property. If not, it will send the request to the default http port. Hence, if a request is received by the 8280 port of ELB, it will be automatically get redirected to 8280 port of the Member (since it's the HTTP port of Member). Similarly, we should define a mapping for https servlet port (8243).

4. Allow access the management console only through the load balancer. Configure the HTTP/HTTPS proxy ports to communicate through the load balancer by editing the <PRODUCT_HOME>/repository/conf/tomcat/catalina-server.xml file as follows.

   Code Block
   language xml
   <Connector protocol="org.apache.coyote.http11.Http11NioProtocol" port="9763" proxyPort="80" ... /> <Connector protocol="org.apache.coyote.http11.Http11NioProtocol" port="9443" proxyPort="443" ... />

   Expand
   title Click here for more information on this configuration.
   The Connector protocol tag sets the protocol to handle incoming traffic. The default value is HTTP/1.1, which uses an auto-switching mechanism to select either a blocking Java-based connector or an APR/native connector. If the PATH (Windows) or LD_LIBRARY_PATH (on most UNIX systems) environment variables contain the Tomcat native library, the APR/native connector will be used. If the native library cannot be found, the blocking Java-based connector will be used. Note that the APR/native connector has different settings from the Java connectors for HTTPS. The non-blocking Java connector used is an explicit protocol that does not rely on the auto-switching mechanism described above. The following is the value used: org.apache.coyote.http11.Http11NioProtocol The TCP port number is the value that this Connector will use to create a server socket and await incoming connections. Your operating system will allow only one server application to listen to a particular port number on a particular IP address. If the special value of 0 (zero) is used, Tomcat will select a free port at random to use for this connector. This is typically only useful in embedded and testing applications.

...

1. Open <ESB_MANAGER_HOME>/repository/conf/carbon.xml.
2. Locate the <Ports> tag and change the value of its sub-tag to: 
   <Offset>1</Offset>
3. Locate the <HOSTNAME> tag and add the cluster host name: 
<HostName>esb.wso2.com</HostName>
4. Locate the <MgtHostName> tag and uncomment it. Make sure that the management host name is defined as follows:
<MgtHostName>mgt.esb.wso2.com</MgtHostName> 

In the next section, we will map the host names we specified to real IPs.

...

• • Enable clustering for this node: 
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
  • Set the membership scheme to wka to enable the Well Known Address registration method (this node will send cluster initiation messages to WKA members that we will define later): 
    <parameter name="membershipScheme">wka</parameter>
  • Specify the name of the cluster this node will join: 
<parameter name="domain">wso2.esb.domain</parameter>
  • Specify the host used to communicate cluster messages:
    <parameter name="localMemberHost">xxx.xxx.xxx.206</parameter>
  • Specify the port used to communicate cluster messages (if this node is on the same server as the ELB, manager node, or another worker node, be sure to set this to a unique value, such as 4000 and 4001 for worker nodes 1 and 3 and 4002 for worker node 2, which is on the same server as the ELB and manager node): 
    <parameter name="localMemberPort">4002</parameter>
    Note: This port number will not be affected by the port offset in carbon.xml. If this port number is already assigned to another server, the clustering framework will automatically increment this port number. However, if two servers are running on the same machine, you must ensure that a unique port is set for each server.

  • The receiver's http/https port values are without the portOffset addition; they get auto-incremented by portOffset. The 'WSDLEPRPrefix' parameter should point to the worker node's host name (esb.cloud-test.wso2.com) and ELB's http (8280)/https (8243) transport ports.

  • Add a new property "subDomain" and set it to "worker" to denote that this node belongs to worker subdomain of the cluster as defined in loadbalancer.conf.

    Code Block
    language html/xml
    <parameter name="properties"> <property name="subDomain" value="worker"/> </parameter>

  • Define the ELB and manager nodes as well-known members of the cluster by providing their host name and localMemberPort values. The manager node is defined here because it is required for the Deployment Synchronizer to function.

    Code Block
    language html/xml
    <members> <member> <hostName>xxx.xxx.xxx.206</hostName> <port>4500</port> </member> <member> <hostName>xxx.xxx.xxx.206</hostName> <port>4001</port> </member> </members>

    Info
    The member in port 4500 is the ELB, and port 4001 is the manager node. 4500 is the value of the group_mgt_port we specify in the loadbalancer.conf file of the ELB and 4001 is the localMemberPort specified in the manager node configurations.

...

Configuring the port offset and host name

Because we are running two Carbon-based products on the same server, we must change the port offset to avoid port conflicts.

1. Open <ESB_WORKER_HOME>/repository/conf/carbon.xml on each worker node.
2. Locate the <HOSTNAME> tag and add the cluster host name: 
<HostName>esb.wso2.com</HostName>
3. Locate the <Ports> tag and change the value of its sub-tag as follows on each worker node:

...

We have now finished configuring the worker nodes and are ready to start them.

Starting the ESB server

...