This site contains the documentation that is relevant to older WSO2 product versions and offerings.
For the latest WSO2 documentation, visit https://wso2.com/documentation/.

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 14 Next »

This page guides you through setting up a clustered deployment of WSO2 Open Banking API Manager. 

You can install multiple instances of WSO2 products in a cluster to ensure that if one instance becomes unavailable or is experiencing high traffic, another instance will seamlessly handle the requests. For complete information on clustering concepts, see Clustering Overview in the Common Product Administration Guide .

Creating a cluster of WSO2 Open Banking API Manager instances involves a standard two-node cluster. To ensure that the instances share governance registry artifacts, you must create a JDBC mount.



At a high level, use the given options to cluster API Manager with a minimum of two nodes. The first section includes instructions on setting up databases. The second section involves setting up a standard two node cluster, the third section involves setting up the API Manager server in a clustered environment and additional configurations if you need to set up a load balancer to front your cluster.

In a standard WSO2 Open Banking 1.4.0 deployment, users can skip the steps mentioned below.

  • Configuring the user store

  • Configuring the datasources

  • Mounting the registry

 Click here to find more information on the steps...

Configuring the user store

WSO2 products allow you to configure multiple user stores to store your users and their roles. Your user store can be one of the following:

  • A Directory Service that can communicate over LDAP protocol like OpenLDAP

  • Active Directory

  • A database that can communicate over JDBC

Note: The instructions in this page demonstrate configuring a JDBC user store. The user store configurations are available in the <WSO2_OB_KM_HOME>/repository/conf/user-mgt.xml file. Point all the cluster nodes to the same user store.

For more information on how to set up other types of user stores, see Configuring User Stores.



Configuring the datasources

  1. Create databases. See Setting up the Physical Database in the WSO2 Administration Guide for database scripts and more information. 
    This documentation demonstrates deployment with user management database (WSO2UM_DB) and an identity database (WSO2AM_DB).

    Alternatively, you can create more databases for each type of data to separate the data logically. Note that this will NOT make a difference in performance and is not actually necessary.

    However, if you wish to separate the data logically into separate databases, see Setting Up Separate Databases for Open Banking Key Manager Clustering.

  2. Configure the datasource for the databases in both nodes of your cluster in the <WSO2_OB_KM_HOME>/repository/conf/datasources/master-datasources.xml file that contains database related configurations.
    For instructions on how to configure the datasource depending on the type of database you created, see  Changing the Carbon Database in the WSO2 Product Administration Guide. Following is a sample configuration of the user management, identity, and registry databases for a MySQL database. Make sure your datasource configurations are as follows:
    <datasource>
    	<name>WSO2_CARBON_DB</name>
    	<description>The datasource used for registry and user manager</description>
    	<jndiConfig>
    		<name>jdbc/WSO2CarbonDB</name>
    	</jndiConfig>
    	<definition type="RDBMS">
    		<configuration>
    			<url>jdbc:h2:repository/database/WSO2CARBON_DB;DB_CLOSE_ON_EXIT=FALSE</url>
    			<username>wso2carbon</username>
    			<password>wso2carbon</password>
    			<driverClassName>org.h2.Driver</driverClassName>
    			<maxActive>50</maxActive>
    			<maxWait>60000</maxWait>
    			<testOnBorrow>true</testOnBorrow>
    			<validationQuery>SELECT 1</validationQuery>
    			<validationInterval>30000</validationInterval>
    		</configuration>
    	</definition>
    </datasource>
    <datasource>
    	<name>WSO2AM_DB</name>
    	<description>The datasource used for API Manager database</description>
    	<jndiConfig>
    		<name>jdbc/WSO2AM_DB</name>
    	</jndiConfig>
    	<definition type="RDBMS">
    		<configuration>
    			<url>jdbc:mysql://localhost:3306/openbank_apimgtdb?autoReconnect=true&useSSL=false</url>
    			<username>root</username>
    			<password>root</password>
    			<driverClassName>com.mysql.jdbc.Driver</driverClassName>
    			<maxActive>150</maxActive>
    			<maxWait>60000</maxWait>
    			<testOnBorrow>true</testOnBorrow>
    			<validationQuery>SELECT 1</validationQuery>
    			<validationInterval>30000</validationInterval>
    			<defaultAutoCommit>false</defaultAutoCommit>
    		</configuration>
    	</definition>
    </datasource>
    <datasource>
    	<name>WSO2UM_DB</name>
    	<description>The datasource used by user manager</description>
    	<jndiConfig>
    		<name>jdbc/WSO2UM_DB</name>
    	</jndiConfig>
    	<definition type="RDBMS">
    		<configuration>
    			<url>jdbc:mysql://localhost:3306/openbank_userdb?autoReconnect=true&useSSL=false</url>
    			<username>root</username>
    			<password>root</password>
    			<driverClassName>com.mysql.jdbc.Driver</driverClassName>
    			<maxActive>150</maxActive>
    			<maxWait>60000</maxWait>
    			<testOnBorrow>true</testOnBorrow>
    			<validationQuery>SELECT 1</validationQuery>
    			<validationInterval>30000</validationInterval>
    			<defaultAutoCommit>false</defaultAutoCommit>
    		</configuration>
    	</definition>
    </datasource>
    <datasource>
    	<name>WSO2CONFIG_DB</name>
    	<description>The datasource used by the registry</description>
    	<jndiConfig>
    		<name>jdbc/WSO2Config_DB</name>
    	</jndiConfig>
    	<definition type="RDBMS">
    		<configuration>
    			<url>jdbc:mysql://localhost:3306/openbank_iskm_configdb?autoReconnect=true&useSSL=false</url>
    			<username>root</username>
    			<password>root</password>
    			<driverClassName>com.mysql.jdbc.Driver</driverClassName>
    			<maxActive>150</maxActive>
    			<maxWait>60000</maxWait>
    			<testOnBorrow>true</testOnBorrow>
    			<validationQuery>SELECT 1</validationQuery>
    			<validationInterval>30000</validationInterval>
    			<defaultAutoCommit>false</defaultAutoCommit>
    		</configuration>
    	</definition>
    </datasource>
    <datasource>
    	<name>WSO2REG_DB</name>
    	<description>The datasource used by the registry</description>
    	<jndiConfig>
    		<name>jdbc/WSO2REG_DB</name>
    	</jndiConfig>
    	<definition type="RDBMS">
    		<configuration>
    			<url>jdbc:mysql://localhost:3306/openbank_govdb?autoReconnect=true&useSSL=false</url>
    			<username>root</username>
    			<password>root</password>
    			<driverClassName>com.mysql.jdbc.Driver</driverClassName>
    			<maxActive>150</maxActive>
    			<maxWait>60000</maxWait>
    			<testOnBorrow>true</testOnBorrow>
    			<validationQuery>SELECT 1</validationQuery>
    			<validationInterval>30000</validationInterval>
    			<defaultAutoCommit>false</defaultAutoCommit>
    		</configuration>
    	</definition>
    </datasource> 
  3. Open the <WSO2_OB_KM_HOME>/repository/conf/finance/open-banking.xml file and configure the WSO2OpenBankingDB datasource in both nodes of your cluster.
    <DataSource>
    	<!-- Include a data source name (jndiConfigName) from the set of data sources defined in master-datasources.xml 
        -->
    	<Name>jdbc/WSO2OpenBankingDB</Name>
    </DataSource>

Mounting the registry

Mount the governance and configuration registry in the <WSO2_OB_KM_HOME>/repository/conf/registry.xml file to share the registry across all nodes in the cluster. For more information on mounting the registry, see Sharing Databases in a Cluster.

Make sure the WSO2Config_DB and WSO2REG_DB configurations are updated in both nodes as follows:

<dbConfig name="configRegistry">
	<dataSource>jdbc/WSO2Config_DB</dataSource>
</dbConfig>

<remoteInstance url="https://localhost:9443/registry">
	<id>configInstance</id>
	<dbConfig>configRegistry</dbConfig>
	<readOnly>false</readOnly>
	<enableCache>true</enableCache>
	<registryRoot>/</registryRoot>
	<cacheId>jdbc:mysql://localhost:3306/openbank_iskm_configdb</cacheId>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
	<instanceId>configInstance</instanceId>
	<targetPath>/_system/config</targetPath>
</mount>

<dbConfig name="governanceRegistry">
	<dataSource>jdbc/WSO2REG_DB</dataSource>
</dbConfig>

<remoteInstance url="https://localhost:9443/registry">
	<id>governanceInstance</id>
	<dbConfig>governanceRegistry</dbConfig>
	<readOnly>false</readOnly>
	<enableCache>true</enableCache>
	<registryRoot>/</registryRoot>
	<cacheId>jdbc:mysql://localhost:3306/openbank_govdb</cacheId>
</remoteInstance>

<mount path="/_system/governance" overwrite="true">
	<instanceId>governanceInstance</instanceId>
	<targetPath>/_system/governance</targetPath>
</mount>

Note: The production recommendation is to set the <versionResourcesOnChange> property in the <WSO2_OB_KM_HOME>/repository/conf/registry.xml file to false. This is because the automatic versioning of resources can be an extremely expensive operation.

<versionResourcesOnChange>false</versionResourcesOnChange>

In the Registry browser, verify that the governance collection is shown with the symlink icon.

  1. Start the Key Manager server. See Starting up and verifying product nodes for more information.
  2. Log in to the Management Console.
  3. Navigate to Home > Registry > Browse.



Changing hostnames and ports

Configure the API Manager node1 using the following steps.

  1. Go to the <WSO2_OB_APIM_HOME>/repository/conf/tomcat/catalina-server.xml file and configure the proxy ports as follows:

    <Connector protocol="org.apache.coyote.http11.Http11NioProtocol" port="9443" proxyPort="443"
    <Connector protocol="org.apache.coyote.http11.Http11NioProtocol" port="9763" proxyPort="80"
  2. By default, WSO2 products identify the hostname of the current machine through the Java API. However, this value sometimes yields erroneous results in some environments. Therefore, users are recommended to configure hostname. The following procedure explains how to change the hostname and management hostname of WSO2 OB APIM as required for your production environment. 

    1. Open the <WSO2_OB_APIM_HOME>/repository/conf/carbon.xml file and set the HostName and MgtHostName property as shown below.

    2. This hostname is used by the WSO2 OB APIM cluster. It must be specified in the /etc/hosts file as:

      127.0.0.1   wso2.obapim.com

    3. Open the <WSO2_OB_APIM_HOME>/repository/conf/api-manager.xml file and set the HostName and MgtHostName property as shown below. 

      1. Configure the Store URL, which is used to access the Store web app from the Publisher application. 

      2. Configure the Publisher URL, which is the URL that is used to access the tenant Store and public Store.

      3. Configure the environment configurations by defining the endpoint URL for both the REST APIs and Web Socket (WS) APIs that are hosted in the API Gateway.

  3. Generate a keystore, export the public certificate from the keystore, and import that certificate to the client­-truststore.jks  file. 
    For more information, see Creating New Keystores in the WSO2 Administration Guide.

Follow all the configuration steps that were done in node1 for node2 as well. 


Enabling artifact synchronization

Deployment artifact synchronization can be done using rsync, which is a file copying tool. These changes must be done in the manager node and in the same directory.

  1. Create a file named workers-list.txt, somewhere in your machine, that lists all the worker nodes in the deployment. The following is a sample of the file where there are two worker nodes.

    Different nodes are separated by new lines.

    workers-list.txt
    ubuntu@192.168.1.1:~/setup/192.168.1.1/as/as_worker/repository/deployment/server
    ubuntu@192.168.1.2:~/setup/192.168.1.2/as/as_worker/repository/deployment/server

    If you have configured tenants in worker nodes, you need to add the repository/tenants directory of the worker node to the list to synchronize tenant space. For example, if the node ubuntu@192.168.1.1 needs to be synced with both the super tenant and the tenant space, the following two entries should be added to the workers-list.txt file.

    workers-list.txt
    ubuntu@192.168.1.1:~/setup/192.168.1.1/apim/apim_worker/repository/deployment/server
    ubuntu@192.168.1.1:~/setup/192.168.1.1/apim/apim_worker/repository/tenants
  2. Create a file to synchronize the <WSO2_OB_APIM_HOME>/repository/deployment/server folders between the manager and all worker nodes.

    You must create your own SSH key and define it as the pem_file. Alternatively, you can use an existing SSH key. For information on generating and using the SSH Keys, go to the SSH documentation. Specify the manager_server_dir depending on the location in your local machine. Change the logs.txt file path and the lock location based on where they are located in your machine.

    rsync-for-carbon-depsync.sh
    #!/bin/sh
    
    manager_server_dir=~/wso2as-5.2.1/repository/deployment/server/
    pem_file=~/.ssh/carbon-440-test.pem
    
    
    #delete the lock on exit
    trap 'rm -rf /var/lock/depsync-lock' EXIT 
    
    mkdir /tmp/carbon-rsync-logs/
    
    
    #keep a lock to stop parallel runs
    if mkdir /var/lock/depsync-lock; then
      echo "Locking succeeded" >&2
    else
      echo "Lock failed - exit" >&2
      exit 1
    fi
    
    
    #get the workers-list.txt
    pushd `dirname $0` > /dev/null
    SCRIPTPATH=`pwd`
    popd > /dev/null
    echo $SCRIPTPATH
    
    
    for x in `cat ${SCRIPTPATH}/workers-list.txt`
    do
    echo ================================================== >> /tmp/carbon-rsync-logs/logs.txt;
    echo Syncing $x;
    rsync --delete -arve "ssh -i  $pem_file -o StrictHostKeyChecking=no" $manager_server_dir $x >> /tmp/carbon-rsync-logs/logs.txt
    echo ================================================== >> /tmp/carbon-rsync-logs/logs.txt;
    done
  3. Create a Cron job that executes the above file every minute for deployment synchronization. Do this by running the following command in your command line.

    You can only run the Cron job on one given node (master) at a given time. If you switch the Cron job to another node, you must stop the Cron job on the existing node and start a new Cron job on the new node after updating it with the latest files.

    *   *  *   *   *     /home/ubuntu/setup/rsync-for-depsync/rsync-for-depsync.sh

Configuring the Proxy Server and the Load Balancer

A load balancer or reverse proxy is required to map external traffic with ports and URLs that WSO2 APIM uses internally. Follow the instructions below to configure load balancing together with reverse proxying.

  • Create a SSL certificate for the load balancer

This step is only applicable for a High Availablity (HA) setup where multiple nodes are fronted by a load balancer.

 Create a SSL certificate for the load balancer using the following instructions...
  1. Create the Server Key.

    sudo openssl genrsa -des3 -out <key_name>.key 1024
  2. Submit the certificate signing request (CSR).

    sudo openssl req -new -key <key_name>.key -out server.csr
  3. Remove the password.

    sudo cp <key_name>.key <key_name>.key.org 
    sudo openssl rsa -in <key_name>.key.org -out <key_name>.key
  4. Sign your SSL Certificate.

    sudo openssl x509 -req -days 365 -in server.csr -signkey <key_name>.key -out <certificate_name>.crt
  • Configure the load balancer/reverse proxy server
    In the following instructions, you are instructed to use NGINX to handle the load balancing requirements.

Although the following section instructs you to use NGINX as the load balancer, you can use any load balancer in your deployment based on your preference.

 What is NGINX?

NGINX is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server. For more information, see https://www.nginx.com/.

 Carry out the following steps to configure the load balancer to front multiple nodes...
  1. Install NGINX in a server configured in your cluster.

    The NGINX version that you need to install varies based on the WSO2 OB APIM components that the load balancer is fronting.

    DeploymentAPI-M NodesLBReason
    Single all-in-one deploymentN/ANGINX CommunityThis deployment does not need Sticky Sessions (Session Affinity).
    Active-active deployment using single all-in-one nodesN/ANGINX PlusThis deployment requires Sticky Sessions, but NGINX Community version does not support it. You can use ip_hash as the sticky algorithm.
    Distributed deploymentGateway with a single Gateway ManagerNGINX Community versionThe Gateway node in this deployment does not need Sticky Sessions.
    Gateway with multiple Gateway ManagersNGINX PlusThe Gateway Manager nodes require Sticky Sessions, but NGINX Community version does not support it. You can use ip_hash as the sticky algorithm. Sticky Sessions are needed for port 9443 in the Gateway, and not needed for the pass through ports in the Gateway (8243, 8280).
  2. Copy the key and certificate files that you generated in step 1 - (4.)  to the /etc/nginx/ssl/ location.

  3. Configure NGINX to direct the HTTP and HTTPs requests based on your deployment.

    1. Run the following command to identify the exact location of the <NGINX_HOME> directory. Inspect the output and identify the --prefix tag as it provides the location of the <NGINX_HOME> directory.

       nginx -V
    2. Update the ngnix.conf file with the required NGINX configuration given below. If not, you can create a file with the .conf suffix and copy it to the <NGINX_HOME>/conf.d directory.

      • All ports are default ports assuming no port offsets are used.
      • The key and the certificate for SSL is assumed to be at the <NGINX_HOME>/ssl/ location. The placeholders {cert_name} and {key_name} are the name of the certificate and key generated.
      • The directories used for access and error logs should be created if they do not exist.

      The ports and URLs that are used internally by API Manager are given below:

      Usage

      URL

      Port

      HTTPS Servlet (UI Consoles)

      localhost

      9443

      NIO transport (HTTP API Traffic)

      localhost

      8280

      NIO transport (HTTPS API Traffic)

      localhost

      8243

      Restart the NGINX server:

      sudo service nginx restart

      You do not need to restart the server if you are simply making a modification to the VHost file. The following command is sufficient in such cases.

      sudo service nginx reload



Starting up and verifying product nodes

If both nodes are running on the same server, set the port offset to avoid port conflicts.

 Click here to see how it is done

By default, the Open Banking API Manager server port offset is 0. So if you're running both API Manager nodes in the same server the port offset can be as follows:

OB_APIM_NODE1: offset 0

OB_APIM_NODE2: offset 1

Changing the offset for default ports

When you run multiple WSO2 products, multiple instances of the same product, or multiple WSO2 product clusters on the same server or virtual machines (VMs), you must change their default ports with an offset value to avoid port conflicts. The default HTTP and HTTPS ports (without offset) of a WSO2 product are 9763 and 9443 respectively. Port offset defines the number by which all ports defined in the runtime such as the HTTP/S ports will be changed. For example, if the default HTTP port is 9763 and the port offset is 1, the effective HTTP port will change to 9764. For each additional WSO2 product instance, you set the port offset to a unique value. The default port offset is 0.

There are two ways to set an offset to a port:

  • Pass the port offset to the server during startup. The following command starts the server with the default port incremented by 1:

    ./wso2server.sh -DportOffset=1
  • Set the Ports configuration in <WSO2_OB_APIM_HOME>/repository/conf/carbon.xml with the desired value as follows:

    <Ports>
    	<Offset>1</Offset>
    </Ports>

Usually, when you offset the port of the server, all ports it uses are changed automatically. However, there are few exceptions as follows in which you have to change the ports manually according to the offset. The following table indicates the changes that occur when the offset value is modified.

WSO2 Server instance

PortOffset

Sample Default Port Value

WSO2 Product 1

0

9443

WSO2 Product 2

1

9444

WSO2 Product 3

2

9445

WSO2 Product 4

3

9446

WSO2 Product 5

4

9447

  1. Start Nginx.

  2. Go to <WSO2_OB_APIM_HOME>/bin and start the nodes using the following command on both nodes:

    ./wso2server.sh
  3. Access the Management console using the following URL: https://wso2.obapim.com/carbon/





  • No labels