This documentation is for WSO2 ESB version 4.0.3. View documentation for the latest release.

Deploying the WSO2 ESB on Your Infrastructure

Deploying the WSO2 ESB correctly on your infrastructure is a vital task for the success of the usage of the ESB. You should first analyze the requirement and then select the best matching deployment mode. There are different set of possible deployment modes with different configurations.

Clustered Deployment Guide

This document describes the steps you need to follow to setup a working WSO2 ESB cluster in your environment.

Clustering the ESB has two different aspects:

Configuration Sharing Across the Cluster Nodes

The configuration sharing is done using the WSO2 Governance Registry. All the ESB nodes in the cluster are pointed to one instance of WSO2 Governance Registry and through this registry configuration is shared.

WSO2 Governance Registry consists of three registry spaces. See also Registry.

  • Local Registry - Per running instance of every product.
  • Configuration Registry - Shared among all the instance of a cluster of one product.
  • Governance Registry - Centrally managed and shared across multiple products and multiple instances of those.

Here a local registry per each instance is used, mounting the configuration registry of each ESB instance to the same configuration registry and sharing the governance space among Governance Registry and 2-ESB instances.

WSO2 ESB clustered deployment load balance

Setting Up WSO2 ESB Cluster with Governance Registry

After extraction of the WSO2 ESB distribution, configure the database.

Both Governance Registry and ESB are shipped with embedded H2 registry. However, for clustering purposes MySQL is used.

In the following setup, the GReg is running on MySQL database that means its local/config/governance registries are stored in a MySQL database and its user-manager tables are also inside the same database.

In ESB instances, both master and slave ESBs are using the unique local registries, both configuration registries mounted to the same directory of GReg and Governance Registry is mounted to the Governance Registry of GReg.

All the ESB instances in the cluster are sharing a common database for user-manager.

Tip

If you are using an LDAP you do not have to use a database for user-manager.

ESB Cluster Database Configuration

Instance Name

Registry Space

Instance Name
Registry Space Mysql database name/mounted path (x.x.x.x)

ESB Master (x.x.x.x)

local

amamaster

 

config

amagregdb /_system/esbConfig

 

governance

amagregdb /_system/governance

 

user-manager

amausermgtdb

ESB Slave1 (x.x.x.x)

local

amaslave1

 

config

amagregdb /_system/esbConfig

 

governance

amagregdb /_system/governance

 

user-manager

amausermgtdb

GREG (x.x.x.x)

All

amagregdb

Tip

In the above table there are 5 separate databases with following names:

  • amagregdb
  • amausermgtdb
  • amaslave1
  • amamaster

If you are having your database remotely, please configure mysql instance to allow remote connections to accept.

Use the documentation of WSO2 Governance Registry to learn more about database configurations with different databases.

Running Multiple Instance of the WSO2 Carbon Servers in the Same Machine

If you are running multiple instance same or different WSO2 Product Instances, you need to configure ports for each instance. You can configure $CARBON_HOME/repository/conf/carbon.xml by using portOffset.

<Offset>0</Offset>

For example, Offset=2 and HTTPS port=9443 will set the effective HTTPS port to 9445.

Configuring GReg

1. Change the configuration in registry.xml and user-mgt.xml.

1.1. Open the GREG_HOME/repository/conf/registry.xml file, remove the original dbConfig element and add the database configuration below.

Note

IP addresses and database URLs have to change according to your setup.

<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
   <url>jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</url>
   <userName>root</userName>
   <password>root123</password>
   <driverName>com.mysql.jdbc.Driver</driverName>
   <maxActive>50</maxActive>
   <maxWait>60000</maxWait>
   <minIdle>5</minIdle>
</dbConfig>

1.2. Open the GREG_HOME/repository/conf/user-mgt.xml file and change the database configuration as shown below.

Note

IP addresses and URLs have to change according to your setup.

<Property name="url">jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true </Property>
<Property name="userName">root</Property>
<Property name="password">root123</Property>
<Property name="driverName">com.mysql.jdbc.Driver</Property>
<Property name="maxActive">50</Property>
<Property name="maxWait">60000</Property>
<Property name="minIdle">5</Property>

1.3. Copy MySQL jdbc driver library to GREG_HOME/repository/components/lib directory since by default we do not ship the MySQL drivers in GReg.

2. In the above configuration, the database is pointed in to same database amagregdb, and now start the GREG instance with -Dsetup.

This will create tables for user manager database and registry database

Tip

Pick the database configuration from currentDBConfig element in registry.xml since there might be number of database configurations in your registry.xml, which we put to perform mounting from other databases, but only one database configuration is configured as currentDBConfig).

3. After starting GReg instance successfully, create the following directory from GReg UI. This directory will be used by the ESB instances as their mounted registry.

/_system/esbConfig.

4. To configure ESB master node, change the default database configuration, to point its local registry to amamaster, point user manager database to amausermgtdb and mount its configuration and governance registries from GReg database.

4.1. Add local registry configuration and user-manager configuration.

4.1.1. Open the ESB_HOME/repository/conf/registry.xml file.                     
4.1.2. Remove the original dbConfig element.             
4.1.3. Add the database configuration shown below.

Note

IP addresses and database URLs have to change according to your setup.

<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
   <url>jdbc:mysql://x.x.x.x:3306/amamaster?autoReconnect=true</url>
   <userName>root</userName>
   <password>root123</password>
   <driverName>com.mysql.jdbc.Driver</driverName>
   <maxActive>50</maxActive>
   <maxWait>60000</maxWait>
   <minIdle>5</minIdle>
</dbConfig>

4.1.4. Open the ESB_HOME/repository/conf/user-mgt.xml file and change the database configuration as shown below.

Note

IP addresses and URLs have to change according to your setup.

<Property name="url">jdbc:mysql://x.x.x.x:3306/amausermgtdb?autoReconnect=true </Property>
<Property name="userName">root</Property>
<Property name="password">root123</Property>
<Property name="driverName">com.mysql.jdbc.Driver</Property>
<Property name="maxActive">50</Property>
<Property name="maxWait">60000</Property>
<Property name="minIdle">5</Property>

4.2. To configure the mounting configuration, more parameters should be added to registry.xml file.

4.2.1. Open the ESB_HOME/repository/conf/registry.xml file.                    
4.2.2. Remove the original dbConfig element.                     
4.2.3. Add the following database configuration as shown below.

Note

IP addresses and database URLs have to change according to your setup.

<dbConfig name="esbMountRegistry">
    <url>jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</url>
    <userName>root</userName>
    <password>root123</password>
    <driverName>com.mysql.jdbc.Driver</driverName>
    <maxActive>50</maxActive>
    <maxWait>60000</maxWait>
    <minIdle>5</minIdle>
</dbConfig>

<remoteInstance url="https://x.x.x.x:9443/registry">
    <id>Mount1</id>
    <dbConfig>esbMountRegistry</dbConfig>
    <readOnly>false</readOnly>
    <registryRoot>/</registryRoot>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
    <instanceId>Mount1</instanceId>
    <targetPath>/_system/esbConfig</targetPath>
</mount>

<mount path="/_system/governance" overwrite="true">
    <instanceId>Mount1</instanceId>
    <targetPath>/_system/governance</targetPath>
</mount>

4.2.4. Change the remoteInstance URL according to the configuration of the GReg running machine.

Note

InstanceIds, id and dbConfig elements should be mapped properly if you are using different names for them.

4.2.5. Copy MySQL jdbc driver library to ESB_HOME/repository/components/lib directory since by default the MySQL drivers are not shipped in ESB.

5. To configure ESB slave node, change the default database configuration, to point its local registry to amaslave1, point user manager database to amausermgtdb and mount its configuration and governance registries from GReg database.

5.1. Add local registry configuration and user-manager configuration.

5.1.1. Open the ESB_SLAVE_HOME/repository/conf/registry.xml file.                        
5.1.2. Remove the original dbConfig element.                               
5.1.3. Add the database configuration shown below.

Note

IP addresses and database URLs have to change according to your setup.

<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
    <url>jdbc:mysql://x.x.x.x:3306/amaslave1?autoReconnect=true</url>
    <userName>root</userName>
    <password>root123</password>
    <driverName>com.mysql.jdbc.Driver</driverName>
    <maxActive>50</maxActive>
    <maxWait>60000</maxWait>
    <minIdle>5</minIdle>
</dbConfig>

<dbConfig name="esbMountRegistry">
    <url>jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</url>
    <userName>root</userName>
    <password>root123</password>
    <driverName>com.mysql.jdbc.Driver</driverName>
    <maxActive>50</maxActive>
    <maxWait>60000</maxWait>
    <minIdle>5</minIdle>
</dbConfig>

<remoteInstance url="https://x.x.x.x:9443/registry">
    <id>Mount1</id>
    <dbConfig>esbMountRegistry</dbConfig>
    <readOnly>true</readOnly>
    <registryRoot>/</registryRoot>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
    <instanceId>Mount1</instanceId>
    <targetPath>/_system/esbConfig</targetPath>
</mount>

<mount path="/_system/governance" overwrite="true">
    <instanceId>Mount1</instanceId>
    <targetPath>/_system/governance</targetPath>
</mount>

5.1.4. Open the ESB_SLAVE_HOME/repository/conf/user-mgt.xml file and change the database configuration as shown below.

Note

IP addresses and URLs have to change according to your setup. Here we change to mode readOnly in database configuration.


<Configuration>
    <AdminRole>admin</AdminRole>
    <AdminUser>
    <UserName>admin</UserName>
    <Password>admin</Password>
    </AdminUser>
    <EveryOneRoleName>everyone</EveryOneRoleName> <\!-\- By default users in this role sees the registry root \-->
    <ReadOnly>true</ReadOnly>
    <Property name="url">jdbc:mysql://localhost:3306/amausermgtdb?autoReconnect=true</Property>
    <Property name="userName">root</Property>
    <Property name="password">root</Property>
    <Property name="driverName">com.mysql.jdbc.Driver</Property>
    <Property name="maxActive">50</Property>
    <Property name="maxWait">60000</Property>
    <Property name="minIdle">5</Property>
</Configuration>

5.1.5. Change the remoteInstance URL according to the configuration of the GRed running machine.

Note

InstanceIds, id and dbConfig elements should be mapped properly if you are using different names for them.

Note

If you compare the configuration with master node, we are only changing the configuration of local registry and all the other configurations are same with the master node, but the registry access mode in mounting is readOnly but the local registry configuration should be read-write since it is specific for each node.

5.2. Copy MySQL jdbc driver library to ESB_SLAVE_HOME/repository/components/lib directory, since by default the MySQL drivers are not shipped in ESB.

6. Load registry database schema to amaslave1 database before starting the slave ESB node. You can find the required SQL script at ESB_SLAVE_HOME/dbscripts directory.

7. Run the wso2server.sh or wso2server.bat of master node based on your platform with -Dsetup option for the first time start up to create the database tables, and after creating database tables, do not use -Dsetup option in the start.sh CARBON_HOME/bin/wso2server.sh -Dsetup.

Tip

You do not need to specify the -Dsetup option for ESB slave node because we have created DB schema in the previous step.

Note

-Dsetup does not work with slave node because we share a same database for user manager and current -Dsetup option tries to create user manager schema also at the startup if you use -Dsetup.

8. After setting up the ESB's pointing to a single Registry there are two options for sharing the configuration.

  • Manual configuration sharing - In this mode ESB's can be configured to load the configuration from the Registry instead of the file system. The configuration loading happens only when the ESB starts up. So if a change is made to the master ESB node, other ESB's has to be restarted to get the new configuration. A lot of users prefer this mode because it guarantees a consistent configuration at the start up and configuration is not changed while the ESB is running.

Tip

By default, ESB loads the configuration from the File System.

If the configuration has to be loaded from the Registry following configuration has to be uncommented and changed in the carbon.xml.

<MediationConfig>
    <LoadFromRegistry>false</LoadFromRegistry>
    <SaveToFile>false</SaveToFile>
    <Persistence>enabled</Persistence>
    <RegistryPersistence>enabled</RegistryPersistence>
</MediationConfig>

Note

LoadFromRegistry parameter has to be changed to true in the slave nodes.

  • Deployment Synchronizer - The next approach is to use the deployment synchronizer shipped with the ESB. With the deployment synchronizer, configurations can be updated in the slave nodes at the runtime:
    • Configuration has to be changed in the master node.
    • The check in that configuration to the registry.
    • Check out the configuration from the slave nodes.

Deployment synchronizer can be used to synchronize the main repository of a Carbon server with a collection in the registry. This feature is sometimes also referred to as the registry based repository synchronizer. Deployment synchronizer has a number of use cases:

  • Maintaining an up-to-date backup of the Carbon repository.
  • Sharing a single Carbon repository among multiple servers (through a shared registry).
  • Enforcing artifact updates in the registry to be deployed on a server at runtime.
  • Enforcing artifact updates in the registry to be deployed on a server at runtime.

In this scenario, an ESB Cluster of two nodes are sharing the same configuration registry. With the deployment synchronizer, it is possible to maintain all the ESB nodes in the cluster in sync through the shared registry. The master ESB node upload its local repository to the registry using the deployment synchronizer. Slave ESB nodes (slave nodes) can then download the same repository from the registry and deploy locally.

To support this use case, synchronizer has to be run in auto commit mode in the master node. When in auto commit mode, it will periodically upload the changed artifacts in the local repository to the registry.

Similarly, slave nodes should run the synchronizer in the auto checkout mode. If needed, registry eventing can be employed to run the checkout operations so that a checkout will be made only when some artifact has changed in the shared registry.

Note

Carbon repository is located in the repository/deployment/server directory by default. Once enabled, deployment synchronizer uploads the contents of this directory to a collection named repository/deployment/server in the configuration registry (this is configurable).

Runtime State Replication

WSO2 ESB flows are stateless for the most part. But in very rare deployments users have requested to share some of the runtime information among the cluster nodes. For example, Caching can be used with runtime state sharing.

Runtime state replication is based on the Tribes group management system. It provides group membership handling and group communication for clustered Carbon Server instances. Although ESB is shipped with this built in Tribes based implementation, other clustering implementations based on different group management systems can be plugged in easily. The only thing you have to do is to implement a set of clustering interfaces provided by Carbon Server.

ESB clustering currently does not support distributed locking for session data replication. Therefore, we have to deploy primary backup clusters for stateful services. This restriction does not apply to stateless services and the user can direct client requests to any node in the cluster for such services.

ESB State Replication Configuration

ESB clustering is configured using the axis2.xml file. So you have to enable clustering in the axis2.xml of each node.

For more details about Carbon Server clustering, please, see the article from the WSO2 library named "WSO Carbon Clustering Configuration Language."

1. Now enable clustering by changing the configuration in the axis2.xml.

Tip

By default, clustering is turned off to avoid additional overhead for individual deployments.

2. Open the axis2.xml and set the value true for the enable attribute of the clustering element as follows.

<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">

3. You may also change clustering properties to suit your deployment as explained in the above article. However, the default configuration is sufficient for the demonstration.