Clustering Governance Registry 5.x.0

This topic illustrates how to cluster WSO2 Governance Registry instances and how to provide database sharing for the clustered environment. Nginx has been used as the load balancer for this cluster. The following sections contain all the configurations required to configure Governance Registry instances and Nginx.

Note: This is relevant for WSO2 Governance Registry versions 5.0.0, 5.1.0, 5.2.0, and 5.3.0.

Environment

The following must be present in your environment for the instructions in this topic to work.

WSO2 Governance Registry
Nginx 1.4.6
MySQL version 5.5.41

Architecture overview

As illustrated in the following diagram, there are three WSO2 Governance Registry nodes fronted with the Nginx load balancer. MySQL is used as the database for the Config and Governance registries and JDBC mounts are created between the database and the Governance Registry nodes. Also MySQL database for WSO2AM_DB is used to share WSO2 API Manager related data across the nodes and WSO2_UM_DB is used to manage users.

Creating the DB mounts for the cluster

The steps in this section provide instructions on how to create the DB mounts for the cluster.

Define the host name for configuring permissions for the new database by opening the /etc/hosts file in your server and adding the following line. You must enter the IP address here in place of <MYSQL-DB-SERVER-IP>.
```
<MYSQL-DB-SERVER-IP>	carbondb.mysql-wso2.com
```

Open a terminal and log-in to the MySQL server. Once there, run the following MySQL commands.

mysql> create database governancedb;
mysql> use governancedb;
mysql> source $GREG_HOME/dbscripts/mysql.sql;
mysql> grant all on governancedb.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";
 
mysql> create database WSO2_AM_DB;
mysql> use WSO2_AM_DB;
mysql> source $GREG_HOME/dbscripts/apimgt/mysql.sql;
mysql> grant all on WSO2_AM_DB.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";
 
mysql> create database WSO2_USER_DB;
mysql> use WSO2_USER_DB;
mysql> source $GREG_HOME/dbscripts/mysql.sql;
mysql> grant all on WSO2_USER_DB.* TO regadmin@"carbondb.mysql-wso2.com" identified by "regadmin";

The above commands generate the relevant tables for each database. Since we have generated the registry database (governancedb) tables you do not need to start the server with -Dsetup.

Note: $GREG_HOME represents the absolute file path of the WSO2 Governance Registry installation.

Create the JDBC mounts (Config and Governance registry spaces). The following steps must be done for all the Governance Registry nodes of the cluster "Read Write" that communicate with the registry database ("governancedb") in a read-write manner.

Add the following configurations to the <GREG_HOME>/repository/conf/datasources/master-datasources.xml file. These configure the datasources in WSO2 Governance Registry for the respective databases.

WSO2_CARBON_DB datasource configuration

<datasource>
        <name>WSO2_CARBON_DB_mount</name>
        <description>The datasource used for registry and user manager</description>
        <jndiConfig>
            <name>jdbc/WSO2CarbonDB_mount</name>
        </jndiConfig>
        <definition type="RDBMS">
            <configuration>
                <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/governancedb</url>
                <username>regadmin</username>
                <password>regadmin</password>
                <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                <maxActive>50</maxActive>
                <maxWait>60000</maxWait>
                <testOnBorrow>true</testOnBorrow>
                <validationQuery>SELECT 1</validationQuery>
                <validationInterval>30000</validationInterval>
            </configuration>
        </definition>
</datasource>

WSO2_AM_DB datasource configuration

<datasource>
      	<name>WSO2_AM_DB</name>
      	<description>The datasource used for API Manager database</description>
      	<jndiConfig>
         	<name>jdbc/WSO2AM_DB</name>
      	</jndiConfig>
      	<definition type="RDBMS">
         	<configuration>
                <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_AM_DB?autoReconnect=true&amp;relaxAutoCommit=true</url>
   	            <username>regadmin</username>
   	            <password>regadmin</password>
   	            <driverClassName>com.mysql.jdbc.Driver</driverClassName>
   	            <maxActive>50</maxActive>
   	            <maxWait>60000</maxWait>
   	            <testOnBorrow>true</testOnBorrow>
   	            <validationQuery>SELECT 1</validationQuery>
   	            <validationInterval>30000</validationInterval>
         	</configuration>
   	</definition>
 </datasource>

WSO2_USER_DB datasource configurations

<datasource>
        <name>WSO2_USER_DB</name>
        <description>The datasource used for registry and user manager</description>
        <jndiConfig>
            <name>jdbc/WSO2UMDB</name>
        </jndiConfig>
        <definition type="RDBMS">
            <configuration>
                <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url>
                <username>regadmin</username>
                <password>regadmin</password>
                <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                <maxActive>50</maxActive>
                <maxWait>60000</maxWait>
                <testOnBorrow>true</testOnBorrow>
                <validationQuery>SELECT 1</validationQuery>
                <validationInterval>30000</validationInterval>
             </configuration>
        </definition>
</datasource>

In addition to the above datasource configurations, each Governance Registry node must have a local database configured to it for runtime data and related data. Add the following configurations to the <GREG_HOME>/repository/conf/datasources/master-datasources.xml file. These configure the datasources in WSO2 Governance Registry for the local database. The <url> must be unique for each node.

WSO2_CARBON_DB local datasource configurations

<datasource>
        <name>WSO2_CARBON_DB</name>
        <description>The datasource used for registry and user manager</description>
        <jndiConfig>
            <name>jdbc/WSO2CarbonDB</name>
        </jndiConfig>
        <definition type="RDBMS">
            <configuration>
                <url>jdbc:h2:repository/database/WSO2CARBON_DB_WORKER1;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url>
                <username>regadmin</username>
                <password>regadmin</password>
                <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                <maxActive>50</maxActive>
                <maxWait>60000</maxWait>
                <testOnBorrow>true</testOnBorrow>
                <validationQuery>SELECT 1</validationQuery>
                <validationInterval>30000</validationInterval>
             </configuration>
        </definition>
</datasource>

To configure the datasource for user management, update the dataSource property found in the <GREG_HOME>/repository/conf/user-mgt.xml file as shown below. this is done so that the right database is used for user management.
```
<Property name="dataSource">jdbc/WSO2UMDB</Property>
```

Add the following configurations to the <GREG_HOME>/repository/conf/registry.xml file. These are mounting configurations for the Config and Registry.

<dbConfig name="wso2registry_mount">
        <dataSource>jdbc/WSO2CarbonDB_mount</dataSource>
 </dbConfig>

<remoteInstance url="https://governance.cluster.wso2.com:9443/registry">
        <id>instanceid</id>
        <dbConfig>wso2registry_mount</dbConfig>
        <readOnly>false</readOnly>
        <enableCache>true</enableCache>
        <registryRoot>/</registryRoot>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
        <instanceId>instanceid</instanceId>
        <targetPath>/_system/config</targetPath>
    </mount>

<mount path="/_system/governance" overwrite="true">
        <instanceId>instanceid</instanceId>
        <targetPath>/_system/governance</targetPath>
</mount>

Copy the MySQL JDBC driver to the <GREG_HOME>/repository/components/lib directory.

If the mount was successfully established to the remote database, you will see INFO logs in the console similar to the following.

[2015-02-19 11:41:20,565]  INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} -  Configured Registry in 83ms
[2015-02-19 11:41:20,605]  INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} -  Connected to mount at wso2registry_mount in 4ms
[2015-02-19 11:41:20,799]  INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} -  Connected to mount at wso2registry_mount in 1ms
[2015-02-19 11:41:20,819]  INFO {org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent} -  Registry Mode	: READ-WRITE

Enabling clustering in Governance Registry

The configurations in this section must be added to all the Governance Registry nodes in the cluster.

Open the <GREG_HOME>/repository/conf/axis2/axis2.xml file and do the following changes.
1. Enable clustering for all the Governance Registry instances by setting the following entry in the axis2.xml file to true.
```
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
```
2. Set the membership scheme to wka to enable the well-known address registration method (this node will send cluster initiation messages to WKA members that we will define later):
```
<parameter name="membershipScheme">wka</parameter>
```
3. Specify the name of the cluster this node will join:
```
<parameter name="domain">wso2.governance.domain</parameter>
```
4. Specify the host used to communicate cluster messages. You may run into issues when using host names in products based on WSO2 Carbon 4.2.0, so it is recommended that you use the IP address directly here. You can also use IP address ranges here. For example, 192.168.1.2-10.
```
<parameter name="localMemberHost">192.168.1.2</parameter>
```
5. Specify the port used to communicate cluster messages. This port number will not be affected by the port offset in carbon.xml. If this port number is already assigned to another server, the clustering framework will automatically increment this port number. However, if two servers are running on the same machine, you must ensure that a unique port is set for each server. As a best practice, specify a unique localMemberPort for each Governance Registry node.
```
<parameter name="localMemberPort">4250</parameter>
```
6. Change the subDomain property value in axis2.xml file. WSO2 Carbon-based products support a deployment model where the architecture components are separated as management nodes, which deploy artifacts and change configurations, and worker nodes, which process requests. However, because there is no service artifact deployment in Governance Registry, the nodes are not separated and are all treated identically. So, in short, there is no concept of worker/manager separation in Governance Registry clusters. In a clustered setup, you must configure this explicitly in the axis2.xml file.
```
<parameter name="properties">
		<property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
		<property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
 		<property name="subDomain" value="mgt"/>
 </parameter>
```
7. In the axis2.xml file, you must add well-known member nodes for each Governance Registry instance that you have configured. If you have three instances with 4250, 4251 and 4252 configured as their respective localMemberPort and if you are configuring the 4250 instance, then the configuration should be as follows. This reflects that the other two Governance Registry instances are defined as the well-known members.
```
<members>
	<member>
		<hostName>192.168.1.3</hostName>
		<port>4251</port>
	</member>


	<member>
		<hostName>192.168.1.4</hostName>
		<port>4252</port>
	</member>
</members>
```
If you run multiple Governance Registry instances on the same server, avoid port conflicts by changing the <Offset> value in the <GREG_HOME>/repository/conf/carbon.xml file for each subsequent node. The offset value basically increments the port number of each instance. For example, set it to 1 in the second node to increment the port numbers by 1 and set it to 2 in the third node to increment the port numbers by 2.
```
<Offset>1</Offset>
```
Add host names in the <GREG_HOME>/repository/conf/carbon.xml file in each instance respectively. Once again, you can use the IP address here instead of an actual host name.
```
<HostName>192.168.1.2</HostName>
<MgtHostName>192.168.1.2</MgtHostName>
```

Configure the HTTP/HTTPS proxy ports to communicate through the load balancer. To do this, edit the <GREG_HOME>/repository/conf/tomcat/catalina-server.xml file as shown below.

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
              port="9763"
              proxyPort="80"
...
<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
              port="9443"
              proxyPort="443"
...

Map the host name to IP by adding an entry similar to the following in the /etc/hosts file.
```
192.168.1.2		governance.cluster.wso2.com
```

Sharing the social database

The WSO2_SOCIAL_DB social database is used to persist Store-side social activities (e.g., comments, rating, likes etc.). Follow the steps below to share it among the nodes of the WSO2 G-Reg cluster.

If you did not already create a database driver, create it as follows:
1. Download the driver JAR file of the respective RDBMS type form the vendor's website.
2. Copy the driver JAR file to the <G-REG_HOME>/repository/components/lib directory.
Update the WSO2_SOCIAL_DB datasource, which is defined in the <G-REG_HOME>/repository/conf/datasources/social-datasources.xml file, based on your requirements.

Optionally, create a query adapter.

WSO2 G-Reg is shipped with the following query adapters, where org.wso2.carbon.social.db.adapter.GenericQueryAdapter is the default query adapter.

DB Type	Query Adapter
H2	`org.wso2.carbon.social.db.adapter.GenericQueryAdapter`
MySQL	`org.wso2.carbon.social.db.adapter.GenericQueryAdapter`
MSSQL	`org.wso2.carbon.social.db.adapter.MSSQLQueryAdapter`
Oracle11g	`org.wso2.carbon.social.db.adapter.Oracle11gQueryAdapter`
PostgreSQL	o`rg.wso2.carbon.social.db.adapter.GenericQueryAdapter`

If you are using a different DB other than the above, create a query adapter as follows:

Create a custom query adapter by extending the GenericQueryAdapter and implementing the AdapterInterface.
Create a JAR file for the custom query adapter.
Add the query adapter into the <G-REG_HOME>/repository/components/lib directory.

The default org.wso2.carbon.social.db.adapter.GenericQueryAdapter query adapter is used for H2 and MYSQL. If you are using another DB type, replace the default query adapter in the <G-REG_HOME>/repository/conf/social.xml file, with the query adapter that corresponds to your DB, as shown in the example below.
```
<SocialConfig>
    <QueryAdapterClass>org.wso2.carbon.social.db.adapter.GenericQueryAdapter</QueryAdapterClass>
</SocialConfig>
```
Restart the WSO2 G-Reg servers.

Configuring the Nginx load balancer

The steps in this section provide instructions on how to install and configure Nginx for this cluster.

Install Nginx using the following command in Linux.
```
$sudo apt-get install nginx
```
After installation, you must create the necessary SSL certificates. See Create SSL Certificates for more information on how to do this.

Access the /etc/nginx/nginx.conf file and add the following in the http{} block. This configures Nginx to direct the HTTP requests to the Governance Registry nodes by balancing the load.

upstream ssl.wso2.governance.com {
	ip_hash;
      		server 192.168.1.2:9443;
      		server 192.168.1.3:9443;
      		server 192.168.1.4:9443;
    	}    
     	server {
      	listen 443;
      	server_name governance.cluster.wso2.com;
      	ssl on;
      	ssl_certificate /home/user/server.crt;
      	ssl_certificate_key /home/user/server.key;
      	location /carbon {
          		proxy_pass https://ssl.wso2.governance.com;
      		}
		location /t {
          			proxy_pass https://ssl.wso2.governance.com;
      		}
   	}

The upstream attributes mentioned in the above code block must be provided according to your Governance Registry instances. Path to server.crt and server.key should be given as shown above using the created paths respectively.

Now you should be able to start the Governance Registry instances and access the management console. If the clustering is successful, you should be able to see message similar to the following in your console.

NFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} -  Member joined [7deefe39-fb87-45ee-b306-cb09b71f23b4]: governance.cluster.wso2.com/192.168.1.4:4252