Deploying the WSO2 ESB on Your Infrastructure

Deploying the WSO2 ESB correctly on your infrastructure is a vital task for the success of the usage of the ESB. You should first analyze the requirement and then select the best matching deployment mode. There are different set of possible deployment modes with different configurations.

Clustered Deployment Guide

This document describes the steps you need to follow to setup a working WSO2 ESB cluster in your environment.

Clustering the ESB has two different aspects:

• Configuration sharing across the cluster nodes.
• Runtime state replication among the cluster nodes.

Configuration Sharing Across the Cluster Nodes

The configuration sharing is done using the WSO2 Governance Registry. All the ESB nodes in the cluster are pointed to one instance of WSO2 Governance Registry and through this registry configuration is shared.

WSO2 Governance Registry consists of three registry spaces. See also Registry.

• Local Registry - Per running instance of every product.
• Configuration Registry - Shared among all the instance of a cluster of one product.
• Governance Registry - Centrally managed and shared across multiple products and multiple instances of those.

Here a local registry per each instance is used, mounting the configuration registry of each ESB instance to the same configuration registry and sharing the governance space among Governance Registry and 2-ESB instances.

Setting Up WSO2 ESB Cluster with Governance Registry

After extraction of the WSO2 ESB distribution, configure the database.

Both Governance Registry and ESB are shipped with embedded H2 registry. However, for clustering purposes MySQL is used.

In the following setup, the GReg is running on MySQL database that means its local/config/governance registries are stored in a MySQL database and its user-manager tables are also inside the same database.

In ESB instances, both master and slave ESBs are using the unique local registries, both configuration registries mounted to the same directory of GReg and Governance Registry is mounted to the Governance Registry of GReg.

All the ESB instances in the cluster are sharing a common database for user-manager.

ESB Cluster Database Configuration

If you are having your database remotely, please configure mysql instance to allow remote connections to accept.

Use the documentation of WSO2 Governance Registry to learn more about database configurations with different databases.

Running Multiple Instance of the WSO2 Carbon Servers in the Same Machine

If you are running multiple instance same or different WSO2 Product Instances, you need to configure ports for each instance. You can configure $CARBON_HOME/repository/conf/carbon.xml by using portOffset.

<Offset>0</Offset>

For example, Offset=2 and HTTPS port=9443 will set the effective HTTPS port to 9445.

Configuring GReg

1. Change the configuration in registry.xml and user-mgt.xml.

1.1. Open the GREG_HOME/repository/conf/registry.xml file, remove the original dbConfig element and add the database configuration below.

<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
   <url>jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</url>
   <userName>root</userName>
   <password>root123</password>
   <driverName>com.mysql.jdbc.Driver</driverName>
   <maxActive>50</maxActive>
   <maxWait>60000</maxWait>
   <minIdle>5</minIdle>
</dbConfig>

1.2. Open the GREG_HOME/repository/conf/user-mgt.xml file and change the database configuration as shown below.

<Property name="url">jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true </Property>
<Property name="userName">root</Property>
<Property name="password">root123</Property>
<Property name="driverName">com.mysql.jdbc.Driver</Property>
<Property name="maxActive">50</Property>
<Property name="maxWait">60000</Property>
<Property name="minIdle">5</Property>

1.3. Copy MySQL jdbc driver library to GREG_HOME/repository/components/lib directory since by default we do not ship the MySQL drivers in GReg.

2. In the above configuration, the database is pointed in to same database amagregdb, and now start the GREG instance with -Dsetup.

This will create tables for user manager database and registry database

3. After starting GReg instance successfully, create the following directory from GReg UI. This directory will be used by the ESB instances as their mounted registry.

/_system/esbConfig.

4. To configure ESB master node, change the default database configuration, to point its local registry to amamaster, point user manager database to amausermgtdb and mount its configuration and governance registries from GReg database.

4.1. Add local registry configuration and user-manager configuration.

4.1.1. Open the ESB_HOME/repository/conf/registry.xml file.                     
4.1.2. Remove the original dbConfig element.             
4.1.3. Add the database configuration shown below.

<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
   <url>jdbc:mysql://x.x.x.x:3306/amamaster?autoReconnect=true</url>
   <userName>root</userName>
   <password>root123</password>
   <driverName>com.mysql.jdbc.Driver</driverName>
   <maxActive>50</maxActive>
   <maxWait>60000</maxWait>
   <minIdle>5</minIdle>
</dbConfig>

4.1.4. Open the ESB_HOME/repository/conf/user-mgt.xml file and change the database configuration as shown below.

<Property name="url">jdbc:mysql://x.x.x.x:3306/amausermgtdb?autoReconnect=true </Property>
<Property name="userName">root</Property>
<Property name="password">root123</Property>
<Property name="driverName">com.mysql.jdbc.Driver</Property>
<Property name="maxActive">50</Property>
<Property name="maxWait">60000</Property>
<Property name="minIdle">5</Property>

4.2. To configure the mounting configuration, more parameters should be added to registry.xml file.

4.2.1. Open the ESB_HOME/repository/conf/registry.xml file.                    
4.2.2. Remove the original dbConfig element.                     
4.2.3. Add the following database configuration as shown below.

<dbConfig name="esbMountRegistry">
    <url>jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</url>
    <userName>root</userName>
    <password>root123</password>
    <driverName>com.mysql.jdbc.Driver</driverName>
    <maxActive>50</maxActive>
    <maxWait>60000</maxWait>
    <minIdle>5</minIdle>
</dbConfig>

<remoteInstance url="https://x.x.x.x:9443/registry">
    <id>Mount1</id>
    <dbConfig>esbMountRegistry</dbConfig>
    <readOnly>false</readOnly>
    <registryRoot>/</registryRoot>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
    <instanceId>Mount1</instanceId>
    <targetPath>/_system/esbConfig</targetPath>
</mount>

<mount path="/_system/governance" overwrite="true">
    <instanceId>Mount1</instanceId>
    <targetPath>/_system/governance</targetPath>
</mount>

4.2.4. Change the remoteInstance URL according to the configuration of the GReg running machine.

4.2.5. Copy MySQL jdbc driver library to ESB_HOME/repository/components/lib directory since by default the MySQL drivers are not shipped in ESB.

5. To configure ESB slave node, change the default database configuration, to point its local registry to amaslave1, point user manager database to amausermgtdb and mount its configuration and governance registries from GReg database.

5.1. Add local registry configuration and user-manager configuration.

5.1.1. Open the ESB_SLAVE_HOME/repository/conf/registry.xml file.                        
5.1.2. Remove the original dbConfig element.                               
5.1.3. Add the database configuration shown below.

<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
    <url>jdbc:mysql://x.x.x.x:3306/amaslave1?autoReconnect=true</url>
    <userName>root</userName>
    <password>root123</password>
    <driverName>com.mysql.jdbc.Driver</driverName>
    <maxActive>50</maxActive>
    <maxWait>60000</maxWait>
    <minIdle>5</minIdle>
</dbConfig>

<dbConfig name="esbMountRegistry">
    <url>jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</url>
    <userName>root</userName>
    <password>root123</password>
    <driverName>com.mysql.jdbc.Driver</driverName>
    <maxActive>50</maxActive>
    <maxWait>60000</maxWait>
    <minIdle>5</minIdle>
</dbConfig>

<remoteInstance url="https://x.x.x.x:9443/registry">
    <id>Mount1</id>
    <dbConfig>esbMountRegistry</dbConfig>
    <readOnly>true</readOnly>
    <registryRoot>/</registryRoot>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
    <instanceId>Mount1</instanceId>
    <targetPath>/_system/esbConfig</targetPath>
</mount>

<mount path="/_system/governance" overwrite="true">
    <instanceId>Mount1</instanceId>
    <targetPath>/_system/governance</targetPath>
</mount>

5.1.4. Open the ESB_SLAVE_HOME/repository/conf/user-mgt.xml file and change the database configuration as shown below.

<Configuration>
    <AdminRole>admin</AdminRole>
    <AdminUser>
    <UserName>admin</UserName>
    <Password>admin</Password>
    </AdminUser>
    <EveryOneRoleName>everyone</EveryOneRoleName> <\!-\- By default users in this role sees the registry root \-->
    <ReadOnly>true</ReadOnly>
    <Property name="url">jdbc:mysql://localhost:3306/amausermgtdb?autoReconnect=true</Property>
    <Property name="userName">root</Property>
    <Property name="password">root</Property>
    <Property name="driverName">com.mysql.jdbc.Driver</Property>
    <Property name="maxActive">50</Property>
    <Property name="maxWait">60000</Property>
    <Property name="minIdle">5</Property>
</Configuration>

5.1.5. Change the remoteInstance URL according to the configuration of the GRed running machine.

5.2. Copy MySQL jdbc driver library to ESB_SLAVE_HOME/repository/components/lib directory, since by default the MySQL drivers are not shipped in ESB.

6. Load registry database schema to amaslave1 database before starting the slave ESB node. You can find the required SQL script at ESB_SLAVE_HOME/dbscripts directory.

7. Run the wso2server.sh or wso2server.bat of master node based on your platform with -Dsetup option for the first time start up to create the database tables, and after creating database tables, do not use -Dsetup option in the start.sh CARBON_HOME/bin/wso2server.sh -Dsetup.

8. After setting up the ESB's pointing to a single Registry there are two options for sharing the configuration.

• Manual configuration sharing - In this mode ESB's can be configured to load the configuration from the Registry instead of the file system. The configuration loading happens only when the ESB starts up. So if a change is made to the master ESB node, other ESB's has to be restarted to get the new configuration. A lot of users prefer this mode because it guarantees a consistent configuration at the start up and configuration is not changed while the ESB is running.

If the configuration has to be loaded from the Registry following configuration has to be uncommented and changed in the carbon.xml.

<MediationConfig>
    <LoadFromRegistry>false</LoadFromRegistry>
    <SaveToFile>false</SaveToFile>
    <Persistence>enabled</Persistence>
    <RegistryPersistence>enabled</RegistryPersistence>
</MediationConfig>

• Deployment Synchronizer - The next approach is to use the deployment synchronizer shipped with the ESB. With the deployment synchronizer, configurations can be updated in the slave nodes at the runtime:
  • Configuration has to be changed in the master node.
  • The check in that configuration to the registry.
  • Check out the configuration from the slave nodes.

Deployment synchronizer can be used to synchronize the main repository of a Carbon server with a collection in the registry. This feature is sometimes also referred to as the registry based repository synchronizer. Deployment synchronizer has a number of use cases:

• Maintaining an up-to-date backup of the Carbon repository.
• Sharing a single Carbon repository among multiple servers (through a shared registry).
• Enforcing artifact updates in the registry to be deployed on a server at runtime.
• Enforcing artifact updates in the registry to be deployed on a server at runtime.

In this scenario, an ESB Cluster of two nodes are sharing the same configuration registry. With the deployment synchronizer, it is possible to maintain all the ESB nodes in the cluster in sync through the shared registry. The master ESB node upload its local repository to the registry using the deployment synchronizer. Slave ESB nodes (slave nodes) can then download the same repository from the registry and deploy locally.

To support this use case, synchronizer has to be run in auto commit mode in the master node. When in auto commit mode, it will periodically upload the changed artifacts in the local repository to the registry.

Similarly, slave nodes should run the synchronizer in the auto checkout mode. If needed, registry eventing can be employed to run the checkout operations so that a checkout will be made only when some artifact has changed in the shared registry.

Runtime State Replication

WSO2 ESB flows are stateless for the most part. But in very rare deployments users have requested to share some of the runtime information among the cluster nodes. For example, Caching can be used with runtime state sharing.

Runtime state replication is based on the Tribes group management system. It provides group membership handling and group communication for clustered Carbon Server instances. Although ESB is shipped with this built in Tribes based implementation, other clustering implementations based on different group management systems can be plugged in easily. The only thing you have to do is to implement a set of clustering interfaces provided by Carbon Server.

ESB clustering currently does not support distributed locking for session data replication. Therefore, we have to deploy primary backup clusters for stateful services. This restriction does not apply to stateless services and the user can direct client requests to any node in the cluster for such services.

ESB State Replication Configuration

ESB clustering is configured using the axis2.xml file. So you have to enable clustering in the axis2.xml of each node.

For more details about Carbon Server clustering, please, see the article from the WSO2 library named "WSO Carbon Clustering Configuration Language."

1. Now enable clustering by changing the configuration in the axis2.xml.

2. Open the axis2.xml and set the value true for the enable attribute of the clustering element as follows.

<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">

3. You may also change clustering properties to suit your deployment as explained in the above article. However, the default configuration is sufficient for the demonstration.