Page Comparison

This topic illustrates how to cluster WSO2 Governance Registry instances and how to provide database sharing for the clustered environment. Nginx has been used as the load balancer for this cluster. The following sections contain all the configurations required to configure Governance Registry instances and Nginx.

Note
Note: This is relevant for WSO2 Governance Registry versions 5.0.0, 5.1.0, 5.2.0, 5.3.0, and 5.34.0.

Table of Contents

maxLevel	3
minLevel	3

...

Define the host name for configuring permissions for the new database by opening the /etc/hosts file in your server and adding the following line. You must enter the IP address here in place of <MYSQL-DB-SERVER-IP>.
Code Block
<MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com

Open a terminal and log-in to the MySQL server. Once there, run the following MySQL commands.

Code Block

mysql> create database governancedb;
mysql> use governancedb;
mysql> source $GREG_HOME/dbscripts/mysql.sql;
mysql> grant all on governancedb.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";
 
mysql> create database WSO2_AM_DB;
mysql> use WSO2_AM_DB;
mysql> source $GREG_HOME/dbscripts/apimgt/mysql.sql;
mysql> grant all on WSO2_AM_DB.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";
 
mysql> create database WSO2_USER_DB;
mysql> use WSO2_USER_DB;
mysql> source $GREG_HOME/dbscripts/mysql.sql;
mysql> grant all on WSO2_USER_DB.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";

The above commands generate the relevant tables for each database. Since we have generated the registry database (governancedb) tables you do not need to start the server with -Dsetup.

Note
Note: `$GREG_HOME` represents the absolute file path of the WSO2 Governance Registry installation. Also, the `<GREG_HOME>/dbscripts/apimgt/mysql.sql` file included above is not available by default with WSO2 G-Reg. You need to execute it only if you have WSO2 API Managaer Manager features installed with WSO2 G-REG.

Create the JDBC mounts (Config and Governance registry spaces). The following steps must be done for all the Governance Registry nodes of the cluster "Read Write" that communicate with the registry database ("governancedb") in a read-write manner.

Add the following configurations to the <GREG_HOME>/repository/conf/datasources/master-datasources.xml file. These configure the datasources in WSO2 Governance Registry for the respective databases.

Code Block

language	xml
title	WSO2_CARBON_DB datasource configuration

<datasource>
        <name>WSO2_CARBON_DB_mount</name>
        <description>The datasource used for registry and user manager</description>
        <jndiConfig>
            <name>jdbc/WSO2CarbonDB_mount</name>
        </jndiConfig>
        <definition type="RDBMS">
            <configuration>
                <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/governancedb</url>
                <username>regadmin</username>
                <password>regadmin</password>
                <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                <maxActive>50</maxActive>
                <maxWait>60000</maxWait>
                <testOnBorrow>true</testOnBorrow>
                <validationQuery>SELECT 1</validationQuery>
                <validationInterval>30000</validationInterval>
            </configuration>
        </definition>
</datasource>

Code Block

language	xml
title	WSO2_AM_DB datasource configuration

<datasource>
      	<name>WSO2_AM_DB</name>
      	<description>The datasource used for API Manager database</description>
      	<jndiConfig>
         	<name>jdbc/WSO2AM_DB</name>
      	</jndiConfig>
      	<definition type="RDBMS">
         	<configuration>
                <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_AM_DB?autoReconnect=true&amp;relaxAutoCommit=true</url>
   	            <username>regadmin</username>
   	            <password>regadmin</password>
   	            <driverClassName>com.mysql.jdbc.Driver</driverClassName>
   	            <maxActive>50</maxActive>
   	            <maxWait>60000</maxWait>
   	            <testOnBorrow>true</testOnBorrow>
   	            <validationQuery>SELECT 1</validationQuery>
   	            <validationInterval>30000</validationInterval>
         	</configuration>
   	</definition>
 </datasource>

Code Block

language	xml
title	WSO2_USER_DB datasource configurations

<datasource>
        <name>WSO2_USER_DB</name>
        <description>The datasource used for registry and user manager</description>
        <jndiConfig>
            <name>jdbc/WSO2UMDB</name>
        </jndiConfig>
        <definition type="RDBMS">
            <configuration>
                <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url>
                <username>regadmin</username>
                <password>regadmin</password>
                <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                <maxActive>50</maxActive>
                <maxWait>60000</maxWait>
                <testOnBorrow>true</testOnBorrow>
                <validationQuery>SELECT 1</validationQuery>
                <validationInterval>30000</validationInterval>
             </configuration>
        </definition>
</datasource>

In addition to the above datasource configurations, each Governance Registry node must have a local database configured to it for runtime data and related data. Add the following configurations to the <GREG_HOME>/repository/conf/datasources/master-datasources.xml file. These configure the datasources in WSO2 Governance Registry for the local database. The <url> must be unique for each node.

Code Block

language	xml
title	WSO2_CARBON_DB local datasource configurations

<datasource>
        <name>WSO2_CARBON_DB</name>
        <description>The datasource used for registry and user manager</description>
        <jndiConfig>
            <name>jdbc/WSO2CarbonDB</name>
        </jndiConfig>
        <definition type="RDBMS">
            <configuration>
                <url>jdbc:h2:repository/database/WSO2CARBON_DB_WORKER1;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url>
                <username>regadmin</username>
                <password>regadmin</password>
                <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                <maxActive>50</maxActive>
                <maxWait>60000</maxWait>
                <testOnBorrow>true</testOnBorrow>
                <validationQuery>SELECT 1</validationQuery>
                <validationInterval>30000</validationInterval>
             </configuration>
        </definition>
</datasource>

To configure the datasource for user management, update the dataSource property found in the <GREG_HOME>/repository/conf/user-mgt.xml file as shown below. this is done so that the right database is used for user management.
Code Block
language xml
<Property name="dataSource">jdbc/WSO2UMDB</Property>

Add the following configurations to the <GREG_HOME>/repository/conf/registry.xml file. These are mounting configurations for the Config and Registry.

Code Block

language	xml

<dbConfig name="wso2registry_mount">
        <dataSource>jdbc/WSO2CarbonDB_mount</dataSource>
 </dbConfig>

<remoteInstance url="https://governance.cluster.wso2.com:9443/registry">
        <id>instanceid</id>
        <dbConfig>wso2registry_mount</dbConfig>
        <readOnly>false</readOnly>
        <enableCache>true</enableCache>
        <registryRoot>/</registryRoot>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
        <instanceId>instanceid</instanceId>
        <targetPath>/_system/config</targetPath>
    </mount>

<mount path="/_system/governance" overwrite="true">
        <instanceId>instanceid</instanceId>
        <targetPath>/_system/governance</targetPath>
</mount>

Copy the MySQL JDBC driver to the <GREG_HOME>/repository/components/lib directory.

If the mount was successfully established to the remote database, you will see INFO logs in the console similar to the following.

Code Block

[2015-02-19 11:41:20,565]  INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} -  Configured Registry in 83ms
[2015-02-19 11:41:20,605]  INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} -  Connected to mount at wso2registry_mount in 4ms
[2015-02-19 11:41:20,799]  INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} -  Connected to mount at wso2registry_mount in 1ms
[2015-02-19 11:41:20,819]  INFO {org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent} -  Registry Mode	: READ-WRITE

...

Open the <GREG_HOME>/repository/conf/axis2/axis2.xml file and do the following changes.

Enable clustering for all the Governance Registry instances by setting the following entry in the axis2.xml file to true.
Code Block
language xml
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
Set the membership scheme to wka to enable the well-known address registration method (this node will send cluster initiation messages to WKA members that we will define later):
Code Block
language xml
<parameter name="membershipScheme">wka</parameter>
Specify the name of the cluster this node will join:
Code Block
language xml
<parameter name="domain">wso2.governance.domain</parameter>
Specify the host used to communicate cluster messages. You may run into issues when using host names in products based on WSO2 Carbon 4.2.0, so it is recommended that you use the IP address directly here. You can also use IP address ranges here. For example, 192.168.1.2-10.
Code Block
language xml
<parameter name="localMemberHost">192.168.1.2</parameter>
Specify the port used to communicate cluster messages. This port number will not be affected by the port offset in carbon.xml. If this port number is already assigned to another server, the clustering framework will automatically increment this port number. However, if two servers are running on the same machine, you must ensure that a unique port is set for each server. As a best practice, specify a unique localMemberPort for each Governance Registry node.
Code Block
language xml
<parameter name="localMemberPort">4250</parameter>

Change the subDomain property value in axis2.xml file. WSO2 Carbon-based products support a deployment model where the architecture components are separated as management nodes, which deploy artifacts and change configurations, and worker nodes, which process requests. However, because there is no service artifact deployment in Governance Registry, the nodes are not separated and are all treated identically. So, in short, there is no concept of worker/manager separation in Governance Registry clusters. In a clustered setup, you must configure this explicitly in the axis2.xml file.

Code Block

language	xml

<parameter name="properties">
		<property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
		<property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
 		<property name="subDomain" value="mgt"/>
 </parameter>

In the axis2.xml file, you must add well-known member nodes for each Governance Registry instance that you have configured. If you have three instances with 4250, 4251 and 4252 configured as their respective localMemberPort and if you are configuring the 4250 instance, then the configuration should be as follows. This reflects that the other two Governance Registry instances are defined as the well-known members.
Code Block
language xml
<members> <member> <hostName>192.168.1.3</hostName> <port>4251</port> </member> <member> <hostName>192.168.1.4</hostName> <port>4252</port> </member> </members>

If you run multiple Governance Registry instances on the same server, avoid port conflicts by changing the <Offset> value in the <GREG_HOME>/repository/conf/carbon.xml file for each subsequent node. The offset value basically increments the port number of each instance. For example, set it to 1 in the second node to increment the port numbers by 1 and set it to 2 in the third node to increment the port numbers by 2.
Code Block
language xml
<Offset>1</Offset>
Add host names in the <GREG_HOME>/repository/conf/carbon.xml file in each instance respectively. Once again, you can use the IP address here instead of an actual host name.
Code Block
language xml
<HostName>192.168.1.2</HostName> <MgtHostName>192.168.1.2</MgtHostName>

Configure the HTTP/HTTPS proxy ports to communicate through the load balancer. To do this, edit the <GREG_HOME>/repository/conf/tomcat/catalina-server.xml file as shown below.

Code Block

language	xml

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
              port="9763"
              proxyPort="80"
...
<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
              port="9443"
              proxyPort="443"
...

Map the host name to IP by adding an entry similar to the following in the /etc/hosts file.
Code Block
192.168.1.2 governance.cluster.wso2.com

...

If you did not already create a database driver, create it as follows:
1. Download the driver JAR file of the respective RDBMS type from its vendor's website.
2. Copy the driver JAR file to the <G-REG_HOME>/repository/components/lib directory.
Update the WSO2_SOCIAL_DB datasource, which is defined in the <G-REG_HOME>/repository/conf/datasources/social-datasources.xml file, based on your requirements.

By default, WSO2 G-Reg ships the following query adapters, where org.wso2.carbon.social.db.adapter.GenericQueryAdapter is the default query adapter.

DB Type	Query Adapter
H2	`org.wso2.carbon.social.db.adapter.GenericQueryAdapter`
MySQL	`org.wso2.carbon.social.db.adapter.GenericQueryAdapter`
MSSQL	`org.wso2.carbon.social.db.adapter.MSSQLQueryAdapter`
Oracle11g	`org.wso2.carbon.social.db.adapter.Oracle11gQueryAdapter`
PostgreSQL	o`rg.wso2.carbon.social.db.adapter.GenericQueryAdapter`

If you are using another DB type, create a query adapter to support storing data in that DB type as follows.

Create a custom query adapter by extending the GenericQueryAdapter and implementing the AdapterInterface.
Create a JAR file for the custom query adapter.
Add the query adapter into the <G-REG_HOME>/repository/components/lib directory.

If you are using another DB type other than H2 or MySQL, replace the default org.wso2.carbon.social.db.adapter.GenericQueryAdapter query adapter in the <G-REG_HOME>/repository/conf/social.xml file, with the query adapter that corresponds to your DB, as shown in the example below.
Code Block
<SocialConfig> <QueryAdapterClass>org.wso2.carbon.social.db.adapter.GenericQueryAdapter</QueryAdapterClass> </SocialConfig>
Restart the WSO2 G-Reg nodes.

...

Install Nginx using the following command in Linux.
Code Block
$sudo apt-get install nginx
After installation, you must create the necessary SSL certificates. See Create SSL Certificates /wiki/spaces/CLUSTER44x/pages/9732195 for more information on how to do this.

Access the /etc/nginx/nginx.conf file and add the following in the http{} block. This configures Nginx to direct the HTTP requests to the Governance Registry nodes by balancing the load.

Code Block

upstream ssl.wso2.governance.com {
	ip_hash;
      		server 192.168.1.2:9443;
      		server 192.168.1.3:9443;
      		server 192.168.1.4:9443;
    	}    
     	server {
      	listen 443;
      	server_name governance.cluster.wso2.com;
      	ssl on;
      	ssl_certificate /home/user/server.crt;
      	ssl_certificate_key /home/user/server.key;
      	location /carbon {
          		proxy_pass https://ssl.wso2.governance.com;
      		}
		location /t {
          			proxy_pass https://ssl.wso2.governance.com;
      		}
   	}

The upstream attributes mentioned in the above code block must be provided according to your Governance Registry instances. Path to server.crt and server.key should be given as shown above using the created paths respectively.

...

Versions Compared

Old Version 24

New Version Current

Key