Page Comparison

The following sections provide information and instructions on how to cluster the ESB profile of WSO2 Enterprise Integrator (WSO2 EI) with a third-party load balancer.

...

Install NGINX Plus or Nginx community version configured in a server within your cluster network.

Create a VHost file (ei.http.conf) in the /etc/nginx/conf.d directory and add the following configurations. This configures NGINX Plus to direct the HTTP requests to the two ESB nodes (xxx.xxx.xxx.xx1 and xxx.xxx.xxx.xx2) via the HTTP 80 port using the http://ei.wso2.com/ URL.

Code Block

title	Nginx Community Version and NGINX Plus

upstream wso2.ei.com {
        server xxx.xxx.xxx.xx1:8280;
        server xxx.xxx.xxx.xx2:8280;
}

server {
        listen 80;
        server_name ei.wso2.com;
        location / {
               proxy_set_header X-Forwarded-Host $host;
               proxy_set_header X-Forwarded-Server $host;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header Host $http_host;
               proxy_read_timeout 5m;
               proxy_send_timeout 5m;
               proxy_pass http://wso2.ei.com;

			   proxy_http_version 1.1;
        	   proxy_set_header Upgrade $http_upgrade;
        	   proxy_set_header Connection "upgrade";
        }
}

Create a VHost file (ei.https.conf) in the /etc/nginx/conf.d directory and add the following configurations. This configures NGINX Plus to direct the HTTPS requests to the two ESB nodes (xxx.xxx.xxx.xx1 and xxx.xxx.xxx.xx2) via the HTTPS 443 port using the https://ei.wso2.com/ URL.

Localtabgroup

Localtab

active	true
title	Nginx Community Version

Code Block

upstream ssl.wso2.ei.com {
    server xxx.xxx.xxx.xx1:8243;
    server xxx.xxx.xxx.xx2:8243;
    ip_hash;
}
 
server {
listen 443;
    server_name ei.wso2.com;
    ssl on;
    ssl_certificate /etc/nginx/ssl/server.crt;
    ssl_certificate_key /etc/nginx/ssl/server.key;
    location / {
               proxy_set_header X-Forwarded-Host $host;
               proxy_set_header X-Forwarded-Server $host;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header Host $http_host;
               proxy_read_timeout 5m;
               proxy_send_timeout 5m;
               proxy_pass https://ssl.wso2.ei.com;
 
               proxy_http_version 1.1;
               proxy_set_header Upgrade $http_upgrade;
               proxy_set_header Connection "upgrade";
        }
}

Localtab

title	NGINX Plus

Code Block

upstream ssl.wso2.ei.com {
	server xxx.xxx.xxx.xx1:8243;
	server xxx.xxx.xxx.xx2:8243;
 
			sticky learn create=$upstream_cookie_jsessionid
			lookup=$cookie_jsessionid
			zone=client_sessions:1m;
}

server {
listen 443;
	server_name ei.wso2.com;
	ssl on;
	ssl_certificate /etc/nginx/ssl/server.crt;
	ssl_certificate_key /etc/nginx/ssl/server.key;
	location / {
               proxy_set_header X-Forwarded-Host $host;
               proxy_set_header X-Forwarded-Server $host;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header Host $http_host;
               proxy_read_timeout 5m;
               proxy_send_timeout 5m;
			   proxy_pass https://ssl.wso2.ei.com;
 
			   proxy_http_version 1.1;
			   proxy_set_header Upgrade $http_upgrade;
			   proxy_set_header Connection "upgrade";
    	}
}

Configure Nginx to access the management console as https://ui.ei.wso2.com/carbon via HTTPS 443 port. To do this, create a VHost file (ui.ei.https.conf ) in the /etc/nginx/conf.d/ directory and add the following configurations into it.

Code Block

title	Nginx Community Version and NGINX Plus

server {
	listen 443;
	server_name ui.ei.wso2.com;
	ssl on;
	ssl_certificate /etc/nginx/ssl/server.crt;
	ssl_certificate_key /etc/nginx/ssl/server.key;

	location / {
               proxy_set_header X-Forwarded-Host $host;
               proxy_set_header X-Forwarded-Server $host;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header Host $http_host;
               proxy_read_timeout 5m;
               proxy_send_timeout 5m;
			   proxy_pass https://xxx.xxx.xxx.xx1:9443/;
 
			   proxy_http_version 1.1;
			   proxy_set_header Upgrade $http_upgrade;
			   proxy_set_header Connection "upgrade";
    	}
	error_log  /var/log/nginx/ui-error.log ;
           access_log  /var/log/nginx/ui-access.log;
}

Follow the instructions below to create SSL certificates for both ESB nodes.
Tip
Enter the host name (ei.wso2.com) as the common name when creating keys.
1. Execute the following command to create the Server Key:
  Code Block
  $sudo openssl genrsa -des3 -out server.key 1024
2. Execute the following command to request to sign the certificate:
  Code Block
  $sudo openssl req -new -key server.key -out server.csr
3. Execute the following commands to remove the passwords:
  Code Block
  $sudo cp server.key server.key.org $sudo openssl rsa -in server.key.org -out server.key
4. Execute the following command to sign your SSL Certificate:
  Code Block
  $sudo openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
5. Execute the following command to add the certificate to the <EI_HOME>/repository/resources/security/client-truststore.jks file:
  Code Block
  keytool -import -trustcacerts -alias server -file server.crt -keystore client-truststore.jks
  Tip
  Give the default password wso2carbon when prompted.
Execute the following command to restart the NGINX Plus server:
Code Block
$sudo service nginx restart
Tip
Execute the following command if you do not need to restart the server when you are simply making a modification to the VHost file:
Code Block
$sudo service nginx reload

...

Open the <EI_HOME>/conf/datasources/master-datasources.xml file, and configure the datasources to point to the relevant databases for each ESB node.

Tip
Replace the username, password, and database URL of your MySQL environment accordingly. If you have not enabled SSL, append the `useSSL=false` property to the value of the `<url>` property.

Localtabgroup

Localtab

title	ESB Node 1

For ESB node 1, configure the datasources to point to the REGISTRY_LOCAL1, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as follows:

Code Block

language	xml

<datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration"> 
     <providers> 
        <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider> 
    </providers> 
    <datasources> 
        <datasource> 
            <name>REGISTRY_LOCAL1</name> 
            <description>The datasource used for registry- local</description> 
            <jndiConfig> 
                <name>jdbc/WSO2CarbonDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL1?autoReconnect=true</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
        <datasource> 
            <name>REGISTRY_DB</name> 
            <description>The datasource used for registry- config/governance</description> 
            <jndiConfig> 
                <name>jdbc/WSO2RegistryDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
         <datasource> 
            <name>WSO2_USER_DB</name> 
            <description>The datasource used for registry and user manager</description> 
            <jndiConfig> 
                <name>jdbc/WSO2UMDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
   </datasources> 
</datasources-configuration>

Localtab

title	ESB Node 2

For node 2, configure the datasources to point to the REGISTRY_LOCAL2, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as shown below. Change the username, password, and database URL as needed for your environment.

Code Block

language	xml

<datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration">     <providers>
        <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider>
    </providers>
    <datasources>
        <datasource>
            <name>REGISTRY_LOCAL2</name>
            <description>The datasource used for registry- local</description>
            <jndiConfig>
                <name>jdbc/WSO2CarbonDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL2?autoReconnect=true</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
        <datasource>
            <name>REGISTRY_DB</name>
            <description>The datasource used for registry- config/governance</description>
            <jndiConfig>
                <name>jdbc/WSO2RegistryDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
         <datasource>
            <name>WSO2_USER_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2UMDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
   </datasources>
</datasources-configuration>

Add the following configuration in the <EI_HOME>/conf/user-mgt.xml file to configure the user stores.

Tip
Enter the datasource information for the user store that you configured in the `<EI_HOME>/conf/`datasources`/master-`datasources`.`xml file. You can change the admin username and password as well. However, you should do this before starting the server.

Code Block

language	xml

<Configuration>
  <AddAdmin>true</AddAdmin>
  <AdminRole>admin</AdminRole>
  <AdminUser>
    <UserName>admin</UserName>
    <Password>admin</Password>
  </AdminUser>
  <EveryOneRoleName>everyone</EveryOneRoleName>
  <Property name="dataSource">jdbc/WSO2UMDB</Property>
</Configuration>

Update the dataSource property in all nodes in the <EI_HOME>/conf/user-mgt.xml file as shown below to configure the datasource:
Code Block
<Property name="dataSource">jdbc/WSO2UMDB</Property>

Open the <EI_HOME>/conf/axis2/axis2.xml file for each of the two ESB nodes, and appply the following cluster configurations:

Enable clustering for each node as follows:

Code Block
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

Set the membership scheme to "wka" to enable the well-known address registration method as shown below. Each node sends cluster initiation messages to the WKA members.
Code Block
<parameter name="membershipScheme">wka</parameter>
Specify the name of the cluster to which the node joins:
Code Block
<parameter name="domain">wso2.ei.domain</parameter>

Specify the host to communicate cluster messages. For example, if the IP addresses of the two ESB nodes are xxx.xxx.xxx.xx1 and xxx.xxx.xxx.xx2, they should be specified in the configuration as shown below.

Localtabgroup

Localtab

title	For ESB Node 1

Code Block

language	xml

<parameter name="localMemberHost">xxx.xxx.xxx.xx1</parameter>

Localtab

title	For ESB Node 2

Code Block

language	xml

<parameter name="localMemberHost">xxx.xxx.xxx.xx2</parameter>

Specify the port to communicate cluster messages as follows:

Code Block
<parameter name="localMemberPort">4100</parameter>

Info
This port number is not affected by the port offset value specified in the `<EI_HOME>/conf/carbon.`xml file. If this port number is already assigned to another server, the clustering framework automatically increments this port number. However, if there are two servers running on the same machine, ensure that a unique port is set for each server.

Specify the well-known members in the cluster as shown below. For example, when you configure one ESB node, you need to specify the other nodes in the cluster as well-known members as shown below. The port value for the WKA node must be the same value as it's localMemberPort (in this case it is 4100).
You can also use IP address ranges for the hostname (e.g., 192.168.1.2-10). However, you can define a range only for the last portion of the IP address. Smaller the range, the faster the time it takes to discover members since each node has to scan a lesser number of potential members.
Info
Localtab
title For ESB Node 1
Localtabgroup
. The best practice is to add all the members (including itself) in all the nodes to avoid any conflicts in configurations.
<members>
```
    <member>
		<hostName>xxx.xxx.xxx.
```
xx1<
```
xx2</hostName>
		<port>4100</port> 
	</member>
</members>
```
Code Block
language xml
<members> <member> <hostName>xxx.xxx.xxx.
xx2<
xx1</hostName> <port>4100</port> </member>
</members>
Localtab
title For ESB Node 2
Code Block
language xml
Uncomment and edit the WSDLEPRPrefix element under org.apache.synapse.transport.passthru.PassThroughHttpListener in the transportReceiver section.
Code Block
language xml
<parameter name="WSDLEPRPrefix" locked="false">http://ei.wso2.com:80</parameter>
Uncomment and edit the WSDLEPRPrefix element under org.apache.synapse.transport.passthru.PassThroughHttpSSLListener in the transportReceiver section.
Code Block
language xml
<parameter name="WSDLEPRPrefix" locked="false">https://ei.wso2.com:443</parameter>

Specify the port offset value in the <EI_HOME>/conf/carbon.xml file.

Warning
This step is optional and only required if all server instances are running on the same machine. This is not recommended for production environments. Change all ports used in your configurations based on the offset value if you are setting a port offset.

Expand

title	Click here for more information on configuring the port offset.

When you run multiple products/clusters or multiple instances of the same product on the same server or virtual machines (VMs), change their default ports with an offset value to avoid port conflicts. An offset defines the number by which all ports in the runtime (e.g., HTTP(S) ports) are increased. For example, if the default HTTP port is 9763 and the offset is 1, the effective HTTP port will change to 9764. For each additional product instance, set the port offset to a unique value. The offset of the default ports is zero.

The port value will automatically increase as shown in the Port Value column in the following table, allowing all five WSO2 product instances or servers to run on the same machine.

WSO2 product instance	Port Offset	Port Value
WSO2 server 1	0	9443
WSO2 server 2	1	9444
WSO2 server 3	2	9445
WSO2 server 4	3	9446
WSO2 server 5	4	9447

Code Block

language	xml

<Ports>
	...
	<Offset>0</Offset>
	...
</Ports>

Edit the <EI_HOME>/conf/carbon.xml file as follows to configure the hostname: <HostName>ei.wso2.com</HostName>
Add the host entries to your DNS, or “/etc/hosts” file (in Linux) in all the nodes of the cluster to map the hostnames to the IP addresses. For an example, you can map the IP address of the database server. In this example, MySQL is used as the database server, so <MYSQL-DB-SERVER-IP> is the actual IP address of the database server and the host entry is as follows: <IP-of-MYSQL-DB-SERVER> carbondb.mysql-wso2.com

Edit the <EI_HOME>/conf/tomcat/catalina-server.xml file as follows:

Add proxyPort="80" to the org.apache.coyote.http11.Http11NioProtocol class with the port defined as 9763:
Code Block
language xml
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol" port="9763" proxyPort="80" ... />
Add proxyPort="443" to the org.apache.coyote.http11.Http11NioProtocol class with the port defined as 9443:
Code Block
language xml
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol" port="9443" proxyPort="443" ... />

Expand

title	Click here for more information on this configuration.

The Connector protocol tag sets the protocol to handle incoming traffic. The default value is HTTP/1.1, which uses an auto-switching mechanism to select either a blocking Java-based connector or an APR/native connector. If the PATH (Windows) or LD_LIBRARY_PATH (on most UNIX systems) environment variables contain the Tomcat native library, the APR/native connector will be used. If the native library cannot be found, the blocking Java-based connector will be used. Note that the APR/native connector has different settings from the Java connectors for HTTPS.

The non-blocking Java connector used is an explicit protocol that does not rely on the auto-switching mechanism described above. The following is the value used:
org.apache.coyote.http11.Http11NioProtocol

The TCP port number is the value that this Connector will use to create a server socket and await incoming connections. Your operating system will allow only one server application to listen to a particular port number on a particular IP address. If the special value of 0 (zero) is used, Tomcat will select a free port at random to use for this connector. This is typically only useful in embedded and testing applications.

...

Deploying artifacts across the nodes

...

nodes

...

Use the following deployment synchronization recommendations based on the rate of change of artifacts that happen in your cluster:
For
...
a high
...
rate of changes (i.e., if changes happen very frequently):
-
...
Network File Share (NFS)
For
...
a medium
...
rate of change
-
...
Remote Synchronization (Rsync)
For
...
a low
...
rate of changes (i.e., if changes happen once a week
...
):
- use the configuration management system to handle artifacts
- other deployment options (e.g., Puppet, Chef etc.)
Note
However, using a shared file system is the first preference that WSO2 recommends to synchronize the artifacts among the nodes because you can use it to publish artifacts to any of the nodes without any issues. You need to use Rsync only if you are unable to have a shared file system. This is because Rsync does not support two-way synchronization: That is, Rsync allows write permissions only to one node, and thereby creates the vulnerability of a single point of failure.

Using Network File Share (NFS)

You can use a common shared file system such as Network File System (NFS) or any other shared file system as the content synchronization mechanism. You need to mount the <EI_HOME>/repository/deployment/server directory of the two nodes to the shared file system to share all the artifacts between both nodes.

Using Remote Synchronization (Rsync)

If you are unable to maintain a shared file system, you can synchronize content using Rsync. The Rsync tool, which is a file copying tool is another common approach for synchronizing artifacts across all cluster nodes. Therefore, you can first deploy artifacts in one node of the cluster and then use Rsync to copy those artifacts to other nodes as described below.

Create a file called nodes-list.txt that lists ,which lists all the nodes in the deployment. The following is a sample of the file for two nodes.
Tip
Different nodes are separated by new into individual lines.
Code Block
language text
title nodes-list.txt
ubuntu@192.168.1.1:~/setup/192.168.1.1/ei_node/repository/deployment/server ubuntu@192.168.1.2:~/setup/192.168.1.2/ei_node/repository/deployment/server

Create a file to synchronize the to synchronize the <PRODUCT_HOME>/repository/deployment/server/ directory between directory between the nodes.

Note
You must create your own SSH key and define it as the `pem_file`. Alternatively, you can use an existing SSH key. Specify the `ei_server_dir` depending on the location in your local machine. Change the `logs.txt` file path and the lock location based on where they are locatedin located in your machine.

Tip
Configure rsync in rsyncin the `<EI_HOME>/repository/tenant/` directory to share the tenant artifacts across the cluster.

Code Block

language	applescript
title	rsync-for-ei-depsync.sh

#!/bin/sh 
ei_server_dir=~/wso2ei-6.2.0/repository/deployment/server/
pem_file=~/.ssh/carbon-440-test.pem
 
 
#delete the lock on exit
trap 'rm -rf /var/lock/depsync-lock' EXIT
 
mkdir /tmp/carbon-rsync-logs/ 
 

#keep a lock to stop parallel runs
if mkdir /var/lock/depsync-lock; then
  echo "Locking succeeded" >&2
else
  echo "Lock failed - exit" >&2
  exit 1
fi 
 

#get the nodes-list.txt
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd`
popd > /dev/null
echo $SCRIPTPATH
 
 
for x in `cat ${SCRIPTPATH}/nodes-list.txt`
do
echo ================================================== >> /tmp/carbon-rsync-logs/logs.txt;
echo Syncing $x;
rsync --delete -arve "ssh -i  $pem_file -o StrictHostKeyChecking=no" $ei_server_dir $x >> /tmp/carbon-rsync-logs/logs.txt
echo ================================================== >> /tmp/carbon-rsync-logs/logs.txt;
done

Execute the following command in your CLI to create a Cron job that executes the above file every minute for deployment synchronization.
Code Block
language actionscript3
* * * * * /home/ubuntu/setup/rsync-for-depsync/rsync-for-ei-depsync.shs

Testing the cluster

Follow the steps below to test the cluster.

...

Versions Compared

Old Version 17

New Version Current

Key

Deploying artifacts across the nodes

nodes

Using Network File Share (NFS)

Using Remote Synchronization (Rsync)