This site contains the documentation that is relevant to older WSO2 product versions and offerings.
For the latest WSO2 documentation, visit https://wso2.com/documentation/.

Clustering the Business Process Profile

Owned by Former user (Deleted)
Last updated: May 03, 2018 by Former user (Deleted)
26 min read

The following sections give you information and instructions on how to cluster the business process profile of WSO2 EI with a third-party load balancer. 

The clustering deployment pattern

This pattern has two WSO2 EI nodes to serve service requests with high availability and scalability. It allows access to the Management Console through an external load balancer and directs service requests to nodes through this load balancer. The following image depicts the sample pattern this clustering deployment scenario will follow.

This pattern uses two nodes as well-known members. It is always recommended to have all nodes of the cluster as well-known members.

When configuring your WSO2 products for clustering to host them in your production environment, it is necessary to use a specific IP address and not localhost or host names in your configurations.

Configuring the load balancer

The load balancer automatically distributes incoming traffic across multiple WSO2 product instances. It enables you to achieve greater levels of fault tolerance in your cluster and provides the required balancing of load needed to distribute traffic.

Note the following facts when configuring the load balancer:

  • These configurations are not required if your clustering setup does not have a load balancer.

  • Load balancer ports are HTTP 80 and HTTPS 443 in the deployment pattern shown in the above diagram. If your system uses any other ports, be sure to replace 80 and 443 values with the corresponding ports when you follow the configuration steps in this section.

  • The load balancer directs requests to the server on a round robin basis. For example, the load balancer will direct requests to node 1 (xxx.xxx.xxx.xx1) of the cluster as follows:

It is recommended to use NGINX Plus as your load balancer of choice.

Follow the steps below to configure NGINX Plus version 1.7.11 or NGINX community version 1.9.2 as the load balancer.

  1. Install NGINX Plus or Nginx community version configured in a server within your cluster network.

  2. Create a VHost file (ei.http.conf) in the /etc/nginx/conf.d directory and add the following configurations into it.This configures NGINX Plus to direct the HTTP requests to the two WSO2 EI nodes (xxx.xxx.xxx.xx1 and xxx.xxx.xxx.xx2) via the HTTP 80 port using the http://ei.wso2.com/ URL. 

    Nginx Community Version and NGINX Plus
    upstream wso2.ei.com {
        server xxx.xxx.xxx.xx1:8280;
        server xxx.xxx.xxx.xx2:8280;
}

server {
        listen 80;
        server_name ei.wso2.com;
        location / {
               proxy_set_header X-Forwarded-Host $host;
               proxy_set_header X-Forwarded-Server $host;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header Host $http_host;
               proxy_read_timeout 5m;
               proxy_send_timeout 5m;
               proxy_pass http://wso2.ei.com;

			   proxy_http_version 1.1;
        	   proxy_set_header Upgrade $http_upgrade;
        	   proxy_set_header Connection "upgrade";
        }
}

  3. Create a VHost file (ei.https.conf) in the /etc/nginx/conf.d directory and add the following configurations into it. This configures NGINX Plus to direct the HTTPS requests to the two WSO2 EI nodes (xxx.xxx.xxx.xx1 and xxx.xxx.xxx.xx2) via the HTTPS 443 port using the https://ei.wso2.com/ URL. 

  4. Configure Nginx to access the Management Console as https://mgt.as.wso2.com/carbon via HTTPS 443 port. To do this, create a VHost file (ui.as.https.conf) in the /etc/nginx/conf.d/ directory and add the following configurations into it.

    Nginx Community Version and NGINX Plus
    server {
	listen 443;
	server_name ui.ei.wso2.com;
	ssl on;
	ssl_certificate /etc/nginx/ssl/server.crt;
	ssl_certificate_key /etc/nginx/ssl/server.key;

	location / {
               proxy_set_header X-Forwarded-Host $host;
               proxy_set_header X-Forwarded-Server $host;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_set_header Host $http_host;
               proxy_read_timeout 5m;
               proxy_send_timeout 5m;
			   proxy_pass https://xxx.xxx.xxx.xx1:9443/;
 
			   proxy_http_version 1.1;
			   proxy_set_header Upgrade $http_upgrade;
			   proxy_set_header Connection "upgrade";
    	}
	error_log  /var/log/nginx/ui-error.log ;
           access_log  /var/log/nginx/ui-access.log;
}

  5. Follow the instructions below to create SSL certificates for both WSO2 EI nodes.

    Enter the host name (ei.wso2.com ) as the common name when creating keys.

    1. Execute the following command to create the Server Key: $sudo opensslgenrsa -des3 -out server.key 1024
    2. Execute the following command to request to sign the certificate: $sudo openssl req -new -key server.key -out server.csr
    3. Execute the following commands to remove the passwords:
      $sudo cp server.key server.key.org 
      $sudo openssl rsa -in server.key.org -out server.key

    4. Execute the following commands to sign your SSL Certificate: $sudo openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

    5. Execute the following command to add the certificate to the <EI_HOME>/repository/resources/security/client-truststore.jks file: keytool -import -trustcacerts -alias server -file server.crt -keystore client-truststore.jks

  6. Execute the following command to restart the NGINX Plus server: $sudo service nginx  restart 

    Execute the following command if you do not need to restart the server when you are simply making a modification to the VHost file: $sudo service nginx reload 

Creating the databases

All profiles of WSO2 EI uses a database to store information such as user management details and registry data. All nodes in the cluster must use one central database for config and governance registry mounts. You can create the following databases and associated datasources.

Database NameDescription
WSO2_USER_DB
JDBC user store and authorization manager
REGISTRY_DBShared database for config and governance registry mounts in the product's nodes
REGISTRY_LOCAL1Local registry space in Node 1
REGISTRY_LOCAL2Local registry space in Node 2

It is recommended to use an industry-standard RDBMS such as Oracle, PostgreSQL, MySQL, MS SQL, etc. for most enterprise testing and production environments. However, you can also use the embedded H2 database only for the REGISTRY_LOCAL1 and REGISTRY_LOCAL2.

Following the steps below to create the databases necessary.

These instructions assume you are installing MySQL as your relational database management system (RDBMS), but you can install another supported RDBMS as needed. 

  1. Download and install MySQL Server.

  2. Download the MySQL JDBC driver.

  3. Download and unzip the WSO2 EI binary distribution. 

    Throughout this guide, <EI_HOME> refers to the extracted directory of the WSO2 EI product distribution.

  4. Unzip the downloaded MySQL driver, and copy the MySQL JDBC driver JAR (mysql-connector-java-x.x.xx-bin.jar) into the <EI_HOME>/lib/ directory of both WSO2 EI nodes.

  5. Add the following line to the /etc/hosts file to define the host name for configuring permissions for the new database: <MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com

    Do this step only if your database is not on your local machine and on a separate server.

  6. Execute the following command in a terminal/command window, where username is the username you want to use to access the databases: mysql -u username -p

  7. Specify the password to access the databases with the username you specified when prompted.

  8. Create the databases using the following commands:

    mysql> create database WSO2_USER_DB;
mysql> use WSO2_USER_DB;
mysql> source <EI_HOME>/dbscripts/mysql.sql;
mysql> grant all on WSO2_USER_DB.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";

mysql> create database REGISTRY_DB;
mysql> use REGISTRY_DB;
mysql> source <EI_HOME>/dbscripts/mysql.sql;
mysql> grant all on REGISTRY_DB.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";

mysql> create database REGISTRY_LOCAL1;
mysql> use REGISTRY_LOCAL1;
mysql> source <EI_HOME>/dbscripts/mysql.sql;
mysql> grant all on REGISTRY_LOCAL1.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";
 
mysql> create database REGISTRY_LOCAL2;
mysql> use REGISTRY_LOCAL2;
mysql> source <EI_HOME>/dbscripts/mysql.sql;
mysql> grant all on REGISTRY_LOCAL2.* TO regadmin@"carbondb.mysql-wso2.com" identified 
by "regadmin";

    About using MySQL in different operating systems

    For users of Microsoft Windows, when creating the database in MySQL, it is important to specify the character set as latin1. Failure to do this may result in an error (error code: 1709) when starting your cluster. This error occurs in certain versions of MySQL (5.6.x) and is related to the UTF-8 encoding. MySQL originally used the latin1 character set by default, which stored characters in a 2-byte sequence. However, in recent versions, MySQL defaults to UTF-8 to be friendlier to international users. Hence, you must use latin1 as the character set as indicated below in the database creation commands to avoid this problem. Note that this may result in issues with non-latin characters (like Hebrew, Japanese, etc.). The following is how your database creation command should look.

    mysql> create database <DATABASE_NAME> character set latin1;

    For users of other operating systems, the standard database creation commands will suffice. For these operating systems, the following is how your database creation command should look.

    mysql> create database <DATABASE_NAME>;

 Follow the steps below to create and configure the following databases to cluster the business process profile of WSO2 EI.

Database NameDescription
BPEL_DB
Stores process/task models and instance data of the BPEL/WS-human tasks engines
BPMN_DBStores process and instance data for BPMN


Create these databases using the following commands:

<EI_HOME>, username and password are the same as those you specified in the previous steps.

mysql> create database BPEL_DB;
mysql> use BPEL_DB;
mysql> source <EI_HOME>/wso2/business-process/dbscripts/bpm/bpel/create/mysql.sql;
mysql> grant all on BPEL_DB.* TO username@localhost identified by "password";
 
mysql> create database BPMN_DB;
mysql> use BPMN_DB;
mysql> source <EI_HOME>/wso2/business-process/dbscripts/bps/bpmn/create/activiti.mysql.create.engine.sql;
mysql> source <EI_HOME>/wso2/business-process/dbscripts/bps/bpmn/create/activiti.mysql.create.history.sql;
mysql> source <EI_HOME>/wso2/business-process/dbscripts/bps/bpmn/create/activiti.mysql.create.identity.sql;
mysql> grant all on BPMN_DB.* TO username@localhost identified by "password";

Mounting the registry

The business process profile of WSO2 EI uses the registry to store and share the deployed artifact versions across the clustered environment. Hence, the instructions below enable read/write to registry (i.e., read only false) in one node of the cluster (i.e., Node 1) and make the registry read only for all other nodes as follows:

Note the following when adding these configurations:

  • The existing dbConfig called wso2registry must not be removed.
  • The datasource you specify in the <dbConfig name="sharedregistry"> tag must match the JNDI Config name you specify in the <EI_HOME>/wso2/business-process/conf/datasources/master-datasources.xml file.
  • Registry mount path is used to identify the type of registry. For example ”/_system/config” refers to configuration registry and "/_system/governance" refers to governance registry.
  • This configuration identifies the datasource you configured in the <EI_HOME>/ wso2/business-process/conf/datasources/master-datasources.xml file using the dbConfig entry. The unique name “ wso2bpsregistry” refers to that datasource entry . 

  • The <remoteInstance> section refers to an external registry mount. Specify the read-only/read-write nature of this instance as well as caching configurations and the registry root location in it.

  • Also, specify the cache ID in the <remoteInstance> section for caching to function properly in the clustered environment. Cache ID is same as the JDBC connection URL of the registry database.

    This value is the Cache ID of the remote instance. It should be in the format of $database_username@$database_url, where $database_username is the username of the remote instance database and $database_url is the remote instance database URL. This cacheID denotes the enabled cache. In this case, the database it should connect to is REGISTRY_DB, which is the database shared across all the nodes. You can find that in the mounting configurations of the same datasource that is being used.

  • Define a unique name in the <id> tag for each remote instance. This is then referred to from mount configurations. In the above example, the unique ID for the remote instance is "instanceId".

  • Specify the actual mount path and target mount path in each of the mounting configurations. The target path can be any meaningful name. In this instance, it is "/_system/bpsconfig".

  1. Add the following configuration in the <EI_HOME>/wso2/business-process/repository/conf/registry.xml file of Node 1 to configure the registry mounts. 

    Registry mounting configuration for EI Node 1 (Business Process profile )
    <dbConfig name="wso2bpsregistry">
  <dataSource>jdbc/WSO2RegistryDB</dataSource>
</dbConfig>

<remoteInstance url="https://localhost:9443/registry">
  <id>instanceid</id>
  <dbConfig>wso2bpsregistry</dbConfig>
  <readOnly>false</readOnly>
  <enableCache>true</enableCache>
  <registryRoot>/</registryRoot>
  <cacheId>root@jdbc:mysql://localhost:3306/REGISTRY_DB</cacheId>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
  <instanceId>instanceid</instanceId>
  <targetPath>/_system/bpsconfig</targetPath>
</mount>

<mount path="/_system/governance" overwrite="true">
  <instanceId>instanceid</instanceId>
  <targetPath>/_system/governance</targetPath>
</mount>

  2. Add the following configuration in the <EI_HOME>/wso2/business-process/repository/conf/registry.xml file of Node 2 to configure the registry mounts. 

    Registry mounting configuration for all other EI (Business Process profile ) nodes in the cluster
    <dbConfig name="wso2bpsregistry">
  <dataSource>jdbc/WSO2RegistryDB</dataSource>
</dbConfig>

<remoteInstance url="https://localhost:9443/registry">
  <id>instanceid</id>
  <dbConfig>wso2bpsregistry</dbConfig>
  <readOnly>true</readOnly>
  <enableCache>true</enableCache>
  <registryRoot>/</registryRoot>
  <cacheId>root@jdbc:mysql://localhost:3306/REGISTRY_DB</cacheId>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
  <instanceId>instanceid</instanceId>
  <targetPath>/_system/bpsconfig</targetPath>
</mount>

<mount path="/_system/governance" overwrite="true">
  <instanceId>instanceid</instanceId>
  <targetPath>/_system/governance</targetPath>
</mount>

Configuring the business process profile node

Do the following configurations for all nodes of your cluster.

  1. Configure the datasources to point to the REGISTRY_LOCAL1, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as follows in the <EI_HOME>/conf/datasources/master-datasources.xml file as follows:

    Replace the username, password, and database URL of your MySQL environment accordingly.

    <datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration"> 
     <providers> 
        <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider> 
    </providers> 
    <datasources> 
        <datasource> 
            <name>REGISTRY_LOCAL1</name> 
            <description>The datasource used for registry- local</description> 
            <jndiConfig> 
                <name>jdbc/WSO2CarbonDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL1?autoReconnect=true</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
        <datasource> 
            <name>REGISTRY_DB</name> 
            <description>The datasource used for registry- config/governance</description> 
            <jndiConfig> 
                <name>jdbc/WSO2RegistryDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
         <datasource> 
            <name>WSO2_USER_DB</name> 
            <description>The datasource used for registry and user manager</description> 
            <jndiConfig> 
                <name>jdbc/WSO2UMDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
   </datasources> 
</datasources-configuration>

    Repeat this configuration on the second WSO2 EI node to configure the datasources to point to the REGISTRY_LOCAL2, WSO2_REGISTRY_DB, and WSO2_USER_DB databases as follows: (Change the username, password, and database URL as needed for your environment):

    <datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration">     
    <providers>
        <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider>
    </providers>
    <datasources>
        <datasource>
            <name>REGISTRY_LOCAL2</name>
            <description>The datasource used for registry- local</description>
            <jndiConfig>
                <name>jdbc/WSO2CarbonDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL2?autoReconnect=true</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
        <datasource>
            <name>REGISTRY_DB</name>
            <description>The datasource used for registry- config/governance</description>
            <jndiConfig>
                <name>jdbc/WSO2RegistryDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
         <datasource>
            <name>WSO2_USER_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2UMDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
   </datasources>
</datasources-configuration>

  2. Add the following configuration in the <EI_HOME>/wso2/business-process/conf/user-mgt.xml file to configure the user stores. 

    Enter the datasource information for the user store that you configured in the  <EI_HOME>/conf/ datasources /master-datasources.xml file. You can change the admin username and password as well. However, you should do this before starting the server.

    <Configuration>  
<AddAdmin>true</AddAdmin>
  <AdminRole>admin</AdminRole>
  <AdminUser>
    <UserName>admin</UserName>
    <Password>admin</Password>
  </AdminUser>
  <EveryOneRoleName>everyone</EveryOneRoleName>
  <Property name="dataSource">jdbc/WSO2UMDB</Property>
</Configuration>
  3. Update the dataSource property in all nodes in the <EI_HOME>/wso2/business-process/conf/user-mgt.xml file as shown below to configure the datasource: <Property name="dataSource">jdbc/WSO2UMDB</Property>

  4. Add the following configuration in the <EI_HOME>/wso2/business-process/conf/datasources/bps-datasources.xml file to configure the connection to the business process profile related database as shown below. Change the driver class, database URL, username, and password as needed for your environment:

    Set the <defaultAutoCommit> property to true in every node with the business process profile. This is an important setting for the BPEL engine.

    <datasource>
            <name>BPM_DS</name>
            <description></description>
            <jndiConfig>
                <name>bpmds</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/BPM_DB</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                    <useDataSourceFactory>false</useDataSourceFactory>
	             <defaultAutoCommit>true</defaultAutoCommit>
 	             <maxActive>100</maxActive>
                    <maxIdle>20</maxIdle>
	             <maxWait>10000</maxWait>
                </configuration>
            </definition>
  </datasource>

  5. Add the following configurations in the <EI_HOME>/wso2/business-process/conf/datasources/activiti-datasources.xml file.

    <datasource>
            <name>ACTIVITI_DB</name>
            <description>The datasource used for activiti engine</description>
            <jndiConfig>
                <name>jdbc/ActivitiDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/BPMN_DB</url>
                    <username>root</username>
                    <password>root</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
</datasource>

  6. Edit the <EI_HOME>/wso2/business-process/conf/axis2/axis2.xml file as follows to set up the cluster configurations.

    • Enable clustering for this node as follows: <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
    • Set the membership scheme to "wka" to enable the well-known address registration method. (This node sends cluster initiation messages to the WKA members):  
      <parameter name="membershipScheme">wka</parameter>
    • Specify the name of the cluster this node will join as follows: <parameter name="domain">wso2.ei.domain</parameter> 
    • Specify the host to communicate cluster messages as follows: <parameter name="localMemberHost">xxx.xxx.xxx.xx1</parameter>

    • Specify the port to communicate cluster messages as follows: <parameter name="localMemberPort">4100</parameter>

      This port number is not affected by the port offset value specified in the  <EI_HOME>/conf/carbon.xml file. If this port number is already assigned to another server, the clustering framework automatically increments this port number. However, if there are two servers running on the same machine, ensure that a unique port is set for each server.

    • Specify the well-known members as follows: (The port value for the WKA node must be the same value as it's localMemberPort (in this case it is 4100).

      You can also use IP address ranges for the hostname (e.g.,   192.168.1.2-10). However, you can define a range only for the last portion of the IP address. Smaller the range, the faster the time it takes to discover members since each node has to scan a lesser number of potential members.

  7. Change the <taskServerMode> configuration in the <EI_HOME>wso2/business-process/conf/etc/tasks-config.xml file as follows: <taskServerMode>STANDALONE</taskServerMode>

    WSO2 EI ships with the task server component. By default, this component waits for two task server nodes when you enable clustering. Hence, change this entry to STANDALONE to start the business process profile. 

    The task server configuration does not have an impact on the server runtime functionality. Hence, using AUTO or STANDALONE for this will not affect how the BPEL processes are executed during runtime.

    However, when the AUTO setting is enabled in this default setting when clustering is enabled, the server will wait till it picks up another node so that there are two task server instances up and running. Hence, you will need to start both nodes simultaneously. Therefore, if you want to use AUTO, change the taskServerCount to 1 so that you can start the management node first.

  8. Add the following configurations in the <EI_HOME>/wso2/business-process/conf/bps.xml file:

    This is required only if you are using BPEL/ WS-Human Task functionality.

    • Enable distributed lock as follows: <tns:UseDistributedLock>true</tns:UseDistributedLock>

      This entry enables the Hazelcast-based synchronizations mechanism to prevent concurrent modification of the instance state by cluster members. 

    • Configure the scheduler thread pool size as follows: <tns:ODESchedulerThreadPoolSize>0</tns:ODESchedulerThreadPoolSize>

      Thread pool size must always be smaller than the maximum active database connections configured in the <EI_HOME>/wso2/business-processconf/datasources/master-datasources.xml file. When configuring the thread pool size, allocate 10-15 threads per core depending on your setup. Leave some additional number of database connections since WSO2 EI uses database connections for API management as well.

      Example settings for a two node cluster are:

      • Oracle Server configured database connection size - 250

      • maxActive entry in the <EI_HOME>/wso2/business-process/conf/datasources/master-datasources.xml file for each node - 100

      • SchedulerTreadPool size for each node - 50

    • Optionally, uncomment the elements accordingly and give an unique ID to assign a unique ID to a node as follows: <tns:NodeId>node1</tns:NodeId>

Configuring Hazelcast properties

WSO2 products use Hazelcast as its default clustering engine. You can configure the hazelcast properties for the product nodes by following the steps given below.

  1. Create the hazelcast.properties file with the following property configurations, and copy the file to the <EI_HOME>/wso2/business-process/conf/ directory. 

    #Disabling the hazelcast shutdown hook
hazelcast.shutdownhook.enabled=false
#Setting the hazelcast logging type to log4j
hazelcast.logging.type=log4j

    The above configurations are explained below.

    • Hazelcast shutdown hook: This configuration disables the shutdown hook in hazelcast, which ensures that the hazelcast instance shuts down gracefully whenever the product node shuts down. If the hazelcast shutdown hook is enabled (which is the default behavior of a product), you will see errors such as "Hazelcast instance is not active!" at the time of shutting down the product node: This is because the hazelcast instance shuts down too early when the shutdown hook is enabled.
    • Hazelcast logging type: This configuration sets the hazelcast logging type to log4j, which allows hazelcast logs to be written to the wso2carbon.log file.

  2. If you have enabled log4j for hazelcast logging as shown above, be sure to enter the configuration shown below in the log4j.properties file (stored in the <EI_HOME>/wso2/business-process/conf/ directory). This can be used to configure the log level for hazelcast logging. For a clustered production environment, it is recommended to use INFO as the log level as shown below.

    log4j.logger.com.hazelcast=INFO

Deploying artifacts across the nodes

One common approach for synchronizing artifacts across all cluster nodes is to use rsync tool, which is a file copying tool. Therefore, you can first deploy artifacts in one node of the cluster and then use rsync to copy those artifacts to other nodes as described below.

Use the following other deployment synchronization recommendations based on the rate of changes that happen in your cluster:

  • For a high rate of changes (i.e., if changes happen very frequently):
    - network file share
  • For a low rate of changes (i.e., if changes happen once a week):
    - use the configuration management system to handle artifact
    - other deployment options (e.g., Puppet, Chef etc.)

  1. Create a file called nodes-list.txt that lists all the nodes in the deployment. The following is a sample of the file for two nodes.

    Different nodes are separated by new lines

    nodes-list.txt
    ubuntu@192.168.1.1:~/setup/192.168.1.1/ei_node/wso2/business-process/repository/deployment/server
ubuntu@192.168.1.2:~/setup/192.168.1.2/ei_node/wso2/business-process/repository/deployment/server

  2. Create a file to synchronize the <PRODUCT_HOME>/wso2/business-process/repository/deployment/server/directory between the nodes.

    You must create your own SSH key and define it as the pem_file. Alternatively, you can use an existing SSH key. Specify the ei_server_dir depending on the location in your local machine. Change the logs.txt file path and the lock location based on where they arelocatedin your machine.

    Configure rsync in the <EI_HOME>/wso2/business-process/repository/tenant/ directory to share the tenant artifacts across the cluster.

    rsync-for-ei-depsync.sh
    #!/bin/sh 
ei_server_dir=~/wso2ei-6.2.0/wso2/buisness-process/repository/deployment/server/
pem_file=~/.ssh/carbon-440-test.pem
 
 
#delete the lock on exit
trap 'rm -rf /var/lock/depsync-lock' EXIT
 
mkdir /tmp/carbon-rsync-logs/
 
 
#keep a lock to stop parallel runs
if mkdir /var/lock/depsync-lock; then
  echo "Locking succeeded" >&2
else
  echo "Lock failed - exit" >&2
  exit 1
fi
 
 
#get the nodes-list.txt
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd`
popd > /dev/null
echo $SCRIPTPATH
 
for x in `cat ${SCRIPTPATH}/nodes-list.txt`
do
echo ================================================== >> /tmp/carbon-rsync-logs/logs.txt;
echo Syncing $x;
rsync --delete -arve "ssh -i  $pem_file -o StrictHostKeyChecking=no" $ei_server_dir $x >> /tmp/carbon-rsync-logs/logs.txt
echo ================================================== >> /tmp/carbon-rsync-logs/logs.txt;
done

  3. Execute the following command in your CLI to create a Cron job that executes the above file every minute for deployment synchronization.  

    *   *  *   *   *     /home/ubuntu/setup/rsync-for-depsync/rsync-for-ei-depsync.sh

Testing the cluster

Follow the steps below to test the cluster.

  1. Restart the configured load balancer.

  2. Deploy artifacts to each product deployment location. 

    Use a deployment synchronization mechanism to synchronize the artifacts in the <EI_HOME>/wso2/business-process/repository/deployment/ directory. Always deploy artifacts first to the WSO2 EI server profile node with the registry configured as read/write. Next, deploy the artifacts to the other nodes.

  3. Execute the following command and start both WSO2 EI nodes: sh <EI_HOME>/bin/business-process.sh

  4. Check for ‘member joined’ log messages in all consoles.

    Additional information on logs and new nodes

    When you terminate one node, all nodes identify that the node has left the cluster. The same applies when a new node joins the cluster. If you want to add another new node, copy existing node without any changes if you are running it on a new server (such as xxx.xxx.xxx.184). If you intend to use the new node on a server where another WSO2 product is running, use a copy of node and change the port offset accordingly in the <EI_HOME>/wso2/business-process/conf/carbon.xml file. You also have to change localMemberPort in the <EI_HOME>/wso2/business-process/conf/axis2/axis2.xml file if that product has clustering enabled. Also, map all hostnames to the relevant IP addresses when creating a new node. The log messages indicate if the new node joins the cluster.

  5. Access the Management Console through the LB using the following URL:  https://xxx.xxx.xxx.xx1:443/carbon  
  6. Test load distribution via the following URLs: http://xxx.xxx.xxx.xx1:80/ or https://xxx.xxx.xxx.xx1:443/

  7. Add a sample proxy service with the log mediator in the inSequence so that it will display logs in the terminals, and then observe the cluster messages sent.

  8. Send a request to the endpoint through the load balancer to verify that the proxy service is activated only on the active node(s) while the nodes remain passive. This is to test that the load balancer manages the active and passive states of the nodes, activating nodes as needed and leaving the rest in passive mode. For example, you would send the request to the following URL: http://{Load_Balancer_Mapped_URL_for_worker}/services/{Sample_Proxy_Name}

Tuning performance of the cluster

Follow the steps below to tune performance of the cluster:

The below example parameter values might not be the optimal values for the specific hardware configurations in your environment. Therefore, it is recommended to carry out load tests on your environment to tune the load balancer and other configurations accordingly.

  1. Change the following default memory allocation settings for the server node and the JVM tuning parameters in the server startup scripts (i.e., the <EI_HOME>/bin/business-process.sh or <EI_HOME>/bin/business-process.bat file) according to the expected server load: -Xms256m -Xmx1024m -XX:MaxPermSize=256m

  2. Modify important system files, which affect all programs running on the server. It is recommended to familiarize yourself with these files using Unix/Linux documentation before editing them.