The instructions in this topic walk you through the steps for creating a cluster of WSO2 Business Process Server (BPS) instances. WSO2 BPS Clustering is done mainly to ensure high availability and scalability. The following sections expand on clustering concepts, architecture and instructions to set up a cluster.
Table of Contents |
---|
BPS clustering concepts
High Availability
High availability means there is redundancy in the system such that the service is available to the outside world irrespective of individual component failures. For example, if we have a two node cluster, even if one node fails, the other node must continue to serve requests until the failed node is restored again.
...
Load balancing is the method of distributing workload to multiple server nodes. In order to set up a properly functioning BPS cluster you would require a load balancer. The function of the load balancer is to monitor the availability of the server nodes in the cluster and route requests to all the available nodes in a fair manner. The load balancer would be the external facing interface of the cluster and it would receive all the requests coming to the cluster. It then distributes this load to all available nodes. If a node has failed, then the load balancer will not route requests to that node until that node is back online.
BPS clustering deployment architecture
In order to build a WSO2 Business Process Server cluster you would require the following.
...
A Manager Node (Node1 in above diagram) is where the workflow artifacts (business processes/Human Tasks) are first deployed. The Worker Nodes (Node2 in above diagram) will look at the configuration generated by the manager node for a given deployment artifact and then deploy those artifacts in its runtime. BPS requires this method of deployment because it does automatic versioning of the deployed BPEL/human task artifacts. Hence, in order to have the same version number for a given deployment artifact across all the nodes, we need to do the versioning at one node (manager node). A BPS instance decides whether it is a manager node or a worker node by looking at its configuration registry mounting.
BPS and the registry
In the simplest terms, the registry is an abstraction over a database schema. It provides an API using which you can store data and retrieve data to a database. WSO2 BPS embeds the registry component and hence has a built-in registry. The registry is divided into three spaces.
...
BPS handles long running processes and human tasks. This means, the runtime state of the process instances/ human task instances have to be persisted to a database. BPS persistence database is the databases where we store these process/task configuration data and process/task instance state.
Installing BPS
- Download the latest version of BPS.
- Unzip the BPS zipped archive, and make a copy for each additional BPS node you want to create.
Setting up the databases
These instructions assume you are installing MySQL as your relational database management system (RDBMS), but you can install another supported RDBMS as needed. You must create the following databases and configure the associated datasources.
...
See instructions on setting up databases for installing and creating
WSO2_USER_DB
,REGISTRY_DB
,REGISTRY_LOCAL1
andREGISTRY_LOCAL2
. In addition to that, you need to create theBPS_DB
and theBPMN_DB
.Create the
BPS_DB and BPMN_DB
databases using the following commands, where<BPS_HOME>
is the path to any of the BPS instances you installed, andusername
andpassword
are the same as those you specified in the previous steps:Code Block language none mysql> create database BPS_DB; mysql> use BPS_DB; mysql> source <BPS_HOME>/dbscripts/bps/bpel/create/mysql.sql; mysql> grant all on BPS_DB.* TO username@localhost identified by "password"; mysql> create database BPMN_DB; mysql> use BPMN_DB; mysql> source <BPS_HOME>/dbscripts/bps/bpmn/create/activiti.mysql.create.engine.sql; mysql> source <BPS_HOME>/dbscripts/bps/bpmn/create/activiti.mysql.create.history.sql; mysql> source <BPS_HOME>/dbscripts/bps/bpmn/create/activiti.mysql.create.identity.sql; mysql> grant all on BPMN_DB.* TO username@localhost identified by "password";
On the first BPS node, open
<BPS_HOME>/repository/conf/datasources/master-datasources.xml
and configure the data sources to point to theWSO2_USER_DB
,WSO2_REGISTRY_DB
, andREGISTRY_LOCAL1
databases (change the username, password, and database URL as needed for your environment). Repeat this configuration on the second BPS node, this time configuring the local registry to point toREGISTRY_LOCAL2
.
For details on how to do this, see the Setting up the Database topic.On each BPS node, open
<BPS_HOME>/repository/conf/datasources/bps-datasources.xml
, and configure the connection to the BPS database as follows (change the driver class, database URL, username, and password as needed for your environment):Code Block language xml <datasource> <name>BPS_DS</name> <description></description> <jndiConfig> <name>bpsds</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://localhost:3306/BPS_DB</url> <username>root</username> <password>root</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> <useDataSourceFactory>false</useDataSourceFactory> <defaultAutoCommit>true</defaultAutoCommit> <maxActive>100</maxActive> <maxIdle>20</maxIdle> <maxWait>10000</maxWait> </configuration> </definition> </datasource>
Note Note: The entry
<defaultAutoCommit>true</defaultAutoCommit>
is set totrue
. This is an important setting for the BPEL engine. You must do this for each node in the cluster.On each BPS node, open the
<BPS_HOME>/repository/conf/datasources/activiti-datasources.xml
file and add the relevant entries.Code Block language xml <datasource> <name>ACTIVITI_DB</name> <description>The datasource used for activiti engine</description> <jndiConfig> <name>jdbc/ActivitiDB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://localhost:3306/BPMN_DB</url> <username>root</username> <password>root</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource>
On each BPS node, open
<BPS_HOME>/repository/conf/registry.xml
and configure the registry mounts. Registry mount path is used to identify the type of registry. For example ”/_system/config
” refers to configuration registry and "/_system/governance
" refers to governance registry. The following is an example configuration for BPS registry mounting. Leave the configuration for local registry as it is and add the following new entries.Code Block title Registry configuration for BPS manager node <dbConfig name="wso2bpsregistry"> <dataSource>jdbc/WSO2RegistryDB</dataSource> </dbConfig> <remoteInstance url="https://localhost:9443/registry"> <id>instanceid</id> <dbConfig>wso2bpsregistry</dbConfig> <readOnly>false</readOnly> <enableCache>true</enableCache> <registryRoot>/</registryRoot> <cacheId>root@jdbc:mysql://localhost:3306/REGISTRY_DB</cacheId> </remoteInstance> <mount path="/_system/config" overwrite="true"> <instanceId>instanceid</instanceId> <targetPath>/_system/bpsConfig</targetPath> </mount> <mount path="/_system/governance" overwrite="true"> <instanceId>instanceid</instanceId> <targetPath>/_system/governance</targetPath> </mount>
Make note of the following for more details on configuring this.
- This configuration enables you to identify the data source you configured in the master-datasources.xml file using the
dbConfig
entry and we give a unique name to refer to that datasource entry, which is “wso2bpsregistry
”. - The
remoteInstance
section refers to an external registry mount. We can specify the read-only/read-write nature of this instance as well as caching configurations and the registry root location. Additionally you must specify thecacheId
for caching to function properly in the clustered environment. Note thatcacheId
is same as the JDBC connection URL to the registry database. You define a unique name “id” for each remote instance, which is then referred from mount configurations. - In the above example, the unique id for remote instance is
instanceId
. - In each of the mounting configurations, specify the actual mount path and target mount path.
Code Block language html/xml title Registry configuration for BPS slave node <dbConfig name="wso2bpsregistry"> <dataSource>jdbc/WSO2RegistryDB</dataSource> </dbConfig> <remoteInstance url="https://localhost:9443/registry"> <id>instanceid</id> <dbConfig>wso2bpsregistry</dbConfig> <readOnly>true</readOnly> <enableCache>true</enableCache> <registryRoot>/</registryRoot> <cacheId>root@jdbc:mysql://localhost:3306/ REGISTRY_DB</cacheId> </remoteInstance> <mount path="/_system/config" overwrite="true"> <instanceId>instanceid</instanceId> <targetPath>/_system/bpsConfig</targetPath> </mount> <mount path="/_system/governance" overwrite="true"> <instanceId>instanceid</instanceId> <targetPath>/_system/governance</targetPath> </mount>
Make note of the following for more details on configuring this.
- This configuration is same as above with
readOnly
property set totrue
for remote instance configuration.
- This configuration enables you to identify the data source you configured in the master-datasources.xml file using the
On each BPS node, open
<BPS_HOME>/repository/conf/user-mgt.xml
and configure the user stores. In the user-mgt.xml file, enter the datasource information for the user store that you configured previously in the master-datasoures.xml file. You can change the admin username and password as well. However, you should do this before starting the server.Code Block language html/xml <Configuration> <AddAdmin>true</AddAdmin> <AdminRole>admin</AdminRole> <AdminUser> <UserName>admin</UserName> <Password>admin</Password> </AdminUser> <EveryOneRoleName>everyone</EveryOneRoleName> <Property name="dataSource">jdbc/WSO2UMDB</Property> </Configuration>
On each BPS node, open
<BPS_HOME>/repository/conf/axis2/axis2.xml
and configure the clustering section. The axis2.xml file is used to enable clustering. Well known address (WKA) based clustering method is normally used for BPS clustering. In WKA based clustering, you must have a subset of cluster members configured in all the members of the cluster. At least one well known member has to be operational at all times.Code Block language html/xml <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true"> <parameter name="membershipScheme">wka</parameter> <parameter name="localMemberHost">127.0.0.1</parameter> <parameter name="localMemberPort">4000</parameter> <members> <member> <hostName>127.0.0.1</hostName> <port>4000</port> </member> <member> <hostName>127.0.0.1</hostName> <port>4010</port> </member> </members> </clustering>
Make note of the following for more details on configuring this.
Change the
enable
parameter totrue
.Find the parameter
membershipSchema
and set it aswka
.Configure the
loadMemberHost
andlocalMemberPort
entries. These must be different values for the master and slave if they are on the same server to prevent any conflicts.Under the
members
section, add thehostName
andport
for each WKA member. As we have only two nodes in our sample cluster configuration, we will configure both nodes as WKA nodes.
On each BPS node, open
<BPS_HOME>/repository/conf/etc/tasks-config.xml
and change thetaskServerMode
configuration. BPS is shipped with the task server component as well. By default, when we enable clustering, this component waits for two task server nodes. Hence we need to change this entry toSTANDALONE
in order to start the BPS server.
<taskServerMode>STANDALONE</taskServerMode>
Info title About using AUTO Note that the Task Server configuration does not have an impact on the BPS server runtime functionality. Hence, using
AUTO
orSTANDALONE
here will not affect how the BPEL processes are executed during runtime.However, note that the default setting
<taskServerCount>2</taskServerCount>
in the<BPS_HOME>/repository/conf/etc/tasks-config.xml
file has an impact here if you useAUTO
. When theAUTO
setting is enabled, and clustering is enabled, the server will wait till it picks up another node so that there are two Task Server instances up and running. Hence you will need to start both nodes simultaneously.So if you want to use
AUTO
, change thetaskServerCount
to1
so that you can start the management node first.- On each BPS node, open
<BPS_HOME>/repository/conf/bps.xml
and configure the following:- Enable distributed lock - This entry enables the Hazelcast-based synchronizations mechanism to prevent concurrent modification of the instance state by cluster members.
<tns:UseDistributedLock>true</tns:UseDistributedLock>
Configure the scheduler thread pool size - Thread pool size must always be smaller than the
maxActive
database connections configured in the<BPS_HOME>/repository/conf/datasources/master-datasources.xml
file. When configuring the thread pool size, allocate 10-15 threads per core depending on your setup. Leave some additional number of database connections since BPS uses database connections for management API as well.
<tns:ODESchedulerThreadPoolSize>0</tns:ODESchedulerThreadPoolSize>
Panel Example settings for a two node cluster.
- Oracle Server configured database connection size - 250.
-
maxActive
entry in master-datasources.xml files for each node - 100-
SchedulerTreadPool
size for each node - 50Node ID - Optionally, a unique id can be assigned to each node. Uncomment following elements and give an unique id as following,
<tns:NodeId>node1</tns:NodeId>
- Enable distributed lock - This entry enables the Hazelcast-based synchronizations mechanism to prevent concurrent modification of the instance state by cluster members.
Running the products in a cluster
Deploy artifacts to each product deployment location (
<BPS_HOME>/repository/deployment/....
) by either manually copying the artifacts or using Deployment Synchronizer. On each BPS node, open<BPS_HOME>/repository/conf/carbon.xml
and install SVNKit (svnClientBundle-1.0.0.jar
) from http://dist.wso2.org/tools/svnClientBundle-1.0.0.jar to the<BPS_HOME>/repository/
components/dropins
folder.If you want automatic deployment of artifacts across the cluster nodes, you can enable the deployment synchronizer feature in the carbon.xml file.
Code Block language html/xml <DeploymentSynchronizer> <Enabled>true</Enabled> <AutoCommit>true</AutoCommit> <AutoCheckout>true</AutoCheckout> <RepositoryType>svn</RepositoryType> <SvnUrl>http://10.100.3.115/svn/repos/as</SvnUrl> <SvnUser>wso2</SvnUser> <SvnPassword>wso2123</SvnPassword> <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId> </DeploymentSynchronizer>
Deployment synchronizer functions by committing the artifacts to the configured SVN location from one node (the node with
AutoCommit
option set totrue
) and sending cluster messages to all other nodes about the addition/change of the artifact. When the cluster message is received, all other nodes will do an SVN update resulting in obtaining the changes to relevant deployment directories. Now the server will automatically deploy these artifacts. For the master node, keepAutoCommit
andAutoCheckout
entries astrue
. For all other nodes, change theAutoCommit
entry tofalse
.Start the BPS nodes. Use the following if your nodes are worker nodes:
Code Block sh <BPS_HOME>/bin/wso2server.sh
Performance tuning in the BPS cluster
In the server startup script, you can configure the memory allocation for the server node as well as the JVM tuning parameters. If you open the wso2server.sh or wso2server.bat file located at the <BPS_HOME>/bin
directory and go to the bottom of the file, you will find those parameters. Change them according to the expected server load. The following is the default memory allocation for a WSO2 server.
-Xms256m -Xmx1024m -XX:MaxPermSize=256m
Note |
---|
|
Load balancing
If needed, you can install a hardware load balancer or an HTTP load balancer such as NGINX Plus as the front end to the BPS nodes. See Configuring NGINX for details on configuring Nginx as the load balancer for a WSO2 product cluster.