Creating a Governance Registry Cluster

You can deploy a cluster of Governance Registry instances to achieve load balancing (distribute the requests among the instances) and high availability (if one instance stops, requests are routed to another instance). This page describes how to create a Governance Registry cluster in the following sections:

• Architecture Overview
• Create the Database Instance
• Create JDBC Mounts
• Enable Clustering
• Configure the Load Balancer
• Configure the Governance Registry Nodes

For information on clustering other WSO2 products, see Clustering Landscape for WSO2 Stack.

Architecture Overview

Governance Registry is separated into three registries:

• Config registry - Stores the configurations and related information of a product
• Governance registry - Stores governance artifacts and other related information
• Local registry - Stores information that is local to a node in a cluster

When you cluster Governance Registry, local registries should not be shared and must be kept in different databases, whereas both the Config and Governance registries can be shared across nodes in a cluster. The local registries use the default H2 database that is embedded in Governance Registry, so you only need to configure the Config and Governance registries.

In this example, we use MySQL as the database for the Config and Governance registries and create JDBC mounts between the database and the Governance Registry nodes. WSO2 Elastic Load Balancer is used to handle the distribution of incoming traffic to the nodes. This example uses a single database instance, but a cluster of databases is recommended in a production environment as shown in the illustration below.

Note: Carbon supports a deployment model in which its architecture components are separated as 'management' nodes, which deploy artifacts and change configurations, and 'worker' nodes, which process requests. This approach is useful for service-hosting products like WSO2 ESB and Application Server that perform intensive request processing, but because there is no concept of request processing in Governance Registry, the nodes are not separated and are all treated identically.

Create the Database Instance

1. Open a terminal and create a database instance called governancedb as follows, replacing wso2greg-4.5.3 with the version of Governance Registry you are using.
   create database governancedb;
mysql -u root -p governancedb < wso2greg-4.5.3/dbscripts/mysql.sql

Create JDBC Mounts

Next, you create JDBC mounts for the Config and Governance registry spaces from all the "Read Write" Governance Registry nodes to the governancedb database. For more information on mounts, see Remote Instance and Mount Configuration Details.

1. Under your Governance Registry directory, navigate to repository/conf/datasources and open the master-datasources.xml file.

2. In <datasources>, add the following <datasource> entry to define the data source that will be used to connect to the database:
<datasource>
  <name>WSO2_CARBON_DB_mount</name>
  <description>The datasource used for registry and user manager</description>
  <jndiConfig>
    <name>jdbc/WSO2CarbonDB_mount</name>
  </jndiConfig>
  <definition type="RDBMS">
    <configuration>
      <url>jdbc:mysql://localhost:3306/governancedb</url>
      <username>root</username>
      <password>root123</password>
      <driverClassName>com.mysql.jdbc.Driver</driverClassName>
      <maxActive>50</maxActive>
      <maxWait>60000</maxWait>
      <testOnBorrow>true</testOnBorrow>
      <validationQuery>SELECT 1</validationQuery>
      <validationInterval>30000</validationInterval>
    </configuration>
  </definition>
</datasource> 

3. Navigate up a level to the /repository/conf directory, and add the following configuration to the registry.xml file to create the mounts, one for the Config space and one for the Governance space:
<dbConfig name="wso2registry_mount">
  <dataSource>jdbc/WSO2CarbonDB_mount</dataSource>
</dbConfig>

<remoteInstance url="https://localhost:9443/registry">
  <id>instanceid</id>
  <dbConfig>wso2registry_mount</dbConfig>
  <readOnly>false</readOnly>
  <enableCache>true</enableCache>
  <registryRoot>/</registryRoot>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
  <instanceId>instanceid</instanceId>
  <targetPath>/_system/config</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
  <instanceId>instanceid</instanceId>
  <targetPath>/_system/governance</targetPath>
</mount>

Enable Clustering

Next, you enable clustering by modifying the Axis2 configuration file. Governance Registry uses local instance cache and cache invalidation messages to update the cache across the cluster nodes, and Axis2 cluster messages are used to send those invalidation messages.

Configure the Load Balancer

Next, you configure WSO2 Elastic Load Balancer in front of the Governance Registry nodes to distribute requests to the nodes as needed. If you are using a load balancer other than Elastic Load Balancer, configure it to provide a single URL in front of the Governance Registry nodes. If you need high availability in the load balancing layer, you could also create a cluster of load balancers and include the Governance Registry instances in the virtual IP and server pool.

1. Download and unzip Elastic Load Balancer.

2. In the ELB directory that's created from the ZIP file, navigate to repository/conf/, and open the loadbalancer.conf file.

3. In the governance domains section, add wso2.governance.domain after the three existing domains, so that the governance section now looks like this:
governance {

     hosts         governance.cloud-test.wso2.com;

     #url_suffix   governance.wso2.com;

           domains   {

               wso2.governance1.domain {

                   tenant_range    1-100;

               }

               wso2.governance2.domain {

                   tenant_range    101-200;

               }

               wso2.governance3.domain {

                   tenant_range    *;

               }
            wso2.governance.domain {
                hosts governance.cluster.wso2.com;
                tenant_range *;
            }

           }

   }

4. Navigate to the repository/conf/axis2 directory and open the axis2.xml file.

5. Navigate to the localMemberHost parameter and uncomment it so that it will be exposed to members of the cluster.
<parameter name="localMemberHost">127.0.0.1</parameter>

6. Update your system's hosts file with the following entry:
127.0.0.1 governance.cluster.wso2.com


7. Start Elastic Load Balancer by navigating to the bin directory and executing wso2server.sh or wso2server.bat.

Configure the Governance Registry Nodes

The final step is to configure and start the Governance Registry nodes.

1. On the first Governance Registry node, navigate to <$GREG_HOME>/repository/conf/axis2 and open the axis2.xml file.

2. Change the following parameters to the values shown below:
<parameter name="membershipScheme">wka</parameter>
<parameter name="domain">wso2.governance.domain</parameter>
<parameter name="localMemberHost">governance.cluster.wso2.com</parameter>
<parameter name="localMemberPort">4250</parameter>

3. In the <members> section, verify that there is a member with the details of the load balancer and the Governance Registry node. The following shows the default for Elastic Load Balancer (port 4000) and the Governance Registry node (port 4001). If you are using a different load balancer, update the first member with the correct details.


   <members>

       <member>

           <hostName>127.0.0.1</hostName> 

           <port>4000</port>

       </member>

       <member>

           <hostName>127.0.0.1</hostName>

           <port>4001</port>

       </member>

   </members> 

4. Navigate to repository/conf/tomcat and open the catalina-server.xml file.

5. For the HTTP and HTTPS connectors (the first two connectors in the file), change the proxyPort attribute as follows and move it down into the list of uncommented attributes for that connector:
   HTTP connector: change proxyPort from 80 to 8280
   HTTPS connector : change proxyPort from 443 to 8243

6. Navigate to repository/conf and open carbon.xml.

7. Update the  "HostName" and  "MgtHostName" elements as follows:
<HostName>governance.cluster.wso2.com</HostName>
<MgtHostName>governance.cluster.wso2.com</MgtHostName>

8. Open the user-mgt.xml file and update the "datasource" property as follows to enable user manager tables to be created in "governancedb":
<Property name="dataSource">jdbc/WSO2CarbonDB_mount</Property>

9. Repeat these steps for the other two Governance Registry nodes, changing the value of "localMemberPort" to 4251 and 4252 respectively.

10. Start each of the Governance Registry instances as described in Running the Product.
    If you look at the logs in the Elastic Load Balancer management console, you will see the following types of entries indicating that the nodes joined the cluster:
2013-02-19 01:06:55,647] INFO - RpcMembershipRequestHandler Received JOIN message from 127.0.0.1:4250(wso2.governance.domain)
[2013-02-19 01:06:55,648] INFO - MembershipManager Application member 127.0.0.1:4250(wso2.governance.domain) joined group wso2.governance.domain
[2013-02-19 01:07:06,652] INFO - DefaultGroupManagementAgent Application member Host:127.0.0.1, Port: 4250, HTTP:9764, HTTPS:9444, Domain: wso2.governance.domain, Sub-domain:null, Active:true joined application cluster
[2013-02-19 01:11:07,423] INFO - RpcMembershipRequestHandler Received JOIN message from 127.0.0.1:4251(wso2.governance.domain)
[2013-02-19 01:11:07,423] INFO - MembershipManager Application member 127.0.0.1:4251(wso2.governance.domain) joined group wso2.governance.domain
[2013-02-19 01:11:18,425] INFO - DefaultGroupManagementAgent Application member Host:127.0.0.1, Port: 4251, HTTP:9765, HTTPS:9445, Domain: wso2.governance.domain, Sub-domain:null, Active:true joined application cluster

11. Point your browser to: https://governance.cluster.wso2.com:8243/carbon

You are now invoking Governance Registry through the load balancer. When there is high load, or if one of the Governance Registry instances becomes unavailable, the load balancer forwards traffic to the other nodes in the cluster.