This site contains the documentation that is relevant to older WSO2 product versions and offerings.
For the latest WSO2 documentation, visit https://wso2.com/documentation/.

Clustering Governance Registry with WSO2 ELB

Owned by Former user (Deleted)
Last updated: Feb 26, 2015
7 min read

The following sections in this topic provide information on how to cluster WSO2 Governance Registry using WSO2 ELB.

Architecture overview

Governance Registry is separated into three registries:

  • Config registry - Stores the configurations and related information of a product
  • Governance registry - Stores governance artifacts and other related information
  • Local registry - Stores information that is local to a node in a cluster

When you cluster Governance Registry, local registries should not be shared and must be kept in different databases, whereas both the Config and Governance registries can be shared across nodes in a cluster. The local registries use the default H2 database that is embedded in Governance Registry, so you only need to configure the Config and Governance registries.

In this example, we use MySQL as the database for the Config and Governance registries and create JDBC mounts between the database and the Governance Registry nodes. WSO2 Elastic Load Balancer is used to handle the distribution of incoming traffic to the nodes. This example uses a single database instance, but a cluster of databases is recommended in a production environment as shown in the illustration below.

Carbon supports a deployment model in which its architecture components are separated as 'management' nodes, which deploy artifacts and change configurations, and 'worker' nodes, which process requests. This approach is useful for service-hosting products like WSO2 ESB and Application Server that perform intensive request processing, but because there is no service artifact deployment in Governance Registry, the nodes are not separated and are all treated identically. So in short, there is no concept of worker/manager separation in Governance Registry clusters.

Create the database instance

  1. Open a terminal and create a database instance called governancedb as follows, replacing wso2greg-4.6.0 with the version of Governance Registry you are using.
    create database governancedb; 
    mysql -u root -p governancedb < wso2greg-4.6.0/dbscripts/mysql.sql
  2. After that create a database instance called wso2apim as follows
    create database wso2apim; 
    mysql -u root -p wso2apim < wso2greg-4.6.0/dbscripts/apimgt/mysql.sql

Create JDBC mounts

Next, you create JDBC mounts for the Config and Governance registry spaces from all the "Read Write" Governance Registry nodes to the governancedb database. For more information on mounts, see Remote Instance and Mount Configuration Details.

  1. Under your Governance Registry directory, navigate to repository/conf/datasources and open the master-datasources.xml file.

  2. In <datasources>, add the following <datasource> entry to define the data source that will be used to connect to the database:

    <datasource> 
  <name>WSO2_CARBON_DB_mount</name> 
  <description>The datasource used for registry and user manager</description> 
  <jndiConfig> 
    <name>jdbc/WSO2CarbonDB_mount</name> 
  </jndiConfig> 
  <definition type="RDBMS"> 
    <configuration> 
      <url>jdbc:mysql://localhost:3306/governancedb</url> 
      <username>root</username> 
      <password>root123</password> 
      <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
      <maxActive>50</maxActive> 
      <maxWait>60000</maxWait> 
      <testOnBorrow>true</testOnBorrow> 
      <validationQuery>SELECT 1</validationQuery> 
      <validationInterval>30000</validationInterval> 
    </configuration> 
  </definition> 
</datasource>

     

  3. Navigate up a level to the /repository/conf directory, and add the following configuration to the registry.xml file to create the mounts, one for the Config space and one for the Governance space:

    <dbConfig name="wso2registry_mount"> 
  <dataSource>jdbc/WSO2CarbonDB_mount</dataSource> 
</dbConfig> 

<remoteInstance url="https://localhost:9443/registry"> 
  <id>instanceid</id> 
  <dbConfig>wso2registry_mount</dbConfig> 
  <readOnly>false</readOnly> 
  <enableCache>true</enableCache> 
  <registryRoot>/</registryRoot>
  <cacheId>root@jdbc:mysql://localhost:3306/governancedb</cacheId>
</remoteInstance> 

<mount path="/_system/config" overwrite="true"> 
  <instanceId>instanceid</instanceId> 
  <targetPath>/_system/config</targetPath> 
</mount> 
<mount path="/_system/governance" overwrite="true"> 
  <instanceId>instanceid</instanceId> 
  <targetPath>/_system/governance</targetPath> 
</mount>

Enable clustering

Next, you enable clustering by modifying the axis2.xml configuration file. Governance Registry uses local instance cache and cache invalidation messages to update the cache across the cluster nodes, and Axis2 cluster messages are used to send those invalidation messages.

  1. Navigate to the repository/conf/axis2 directory and open the axis2.xml file.

  2. Find the following entry and change enable to true:

    <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

  3. If you will run multiple Governance Registry instances on the same server, avoid port conflicts by changing the port <Offset> value in <$GREG_HOME>/repository/conf/carbon.xml on each subsequent node. For example, set it to 1 in the second node and set it to 2 in the third node to increment the port number.

  4. Download the MySQL JDBC driver from http://www.mysql.com/products/connector/ and copy it to <$GREG_HOME>/repository/components/lib.

Configure the load balancer

Next, you configure WSO2 Elastic Load Balancer in front of the Governance Registry nodes to distribute requests to the nodes as needed. If you are using a load balancer other than Elastic Load Balancer, configure it to provide a single URL in front of the Governance Registry nodes. If you need high availability in the load balancing layer, you could also create a cluster of load balancers and include the Governance Registry instances in the virtual IP and server pool.

  1. Download and unzip Elastic Load Balancer.

  2. In the ELB directory that's created from the ZIP file, navigate to repository/conf/, and open the loadbalancer.conf file.

  3. In the governance domains section, add wso2.governance.domain after the three existing domains, so that the governance section now looks like this:

    governance {
  hosts         governance.cloud-test.wso2.com;
  #url_suffix   governance.wso2.com;
        domains   {
            wso2.governance.domain { 
                tenant_range *; 
                group_mgt_port 4000;
                mgt{
                    hosts governance.cluster.wso2.com; 
                }
            }
        }
}

  4. Navigate to the repository/conf/axis2 directory and open the axis2.xml file.

  5. Navigate to the localMemberHost parameter and uncomment it so that it will be exposed to members of the cluster.
    <parameter name="localMemberHost">127.0.0.1</parameter>

  6. Navigate to the localMemberPort parameter and change it as below.
    <parameter name="localMemberPort">5000</parameter>

  7. Update your system's hosts file with the following entry: 
    127.0.0.1 governance.cluster.wso2.com

  8. Start Elastic Load Balancer by navigating to the bin directory and executing wso2server.sh or wso2server.bat.

Configure the Governance Registry nodes

The final step is to configure and start the Governance Registry nodes.

  1. On the first Governance Registry node, navigate to <$GREG_HOME>/repository/conf/axis2 and open the axis2.xml file.

  2. Change the following parameters to the values shown below:

    <parameter name="membershipScheme">wka</parameter>
<parameter name="domain">wso2.governance.domain</parameter> 
<parameter name="localMemberHost">governance.cluster.wso2.com</parameter> 
<parameter name="localMemberPort">4250</parameter>


  3. In the <members> section, verify that there is a member with the details of the load balancer and the Governance Registry node. The following shows the default for Elastic Load Balancer (port 4000) and the Governance Registry node (port 4001). If you are using a different load balancer, update the first member with the correct details.

    <members>
    <member>
        <hostName>127.0.0.1</hostName> 
        <port>4000</port>
    </member>
    <member>
        <hostName>127.0.0.1</hostName>
        <port>4001</port>
    </member>
</members>

  4. Navigate to repository/conf/tomcat and open the catalina-server.xml file.

  5. For the HTTP and HTTPS connectors (the first two connectors in the file), change the proxyPort attribute as follows and move it down into the list of uncommented attributes for that connector: 
    HTTP connector: change proxyPort from 80 to 8280
    HTTPS connector : change proxyPort from 443 to 8243

  6. Navigate to repository/conf and open carbon.xml.

  7. Update the  "HostName" and  "MgtHostName" elements as follows:
    <HostName>governance.cluster.wso2.com</HostName> 
    <MgtHostName>governance.cluster.wso2.com</MgtHostName>

  8. Open the user-mgt.xml file and update the "datasource" property as follows to enable user manager tables to be created in "governancedb":
    <Property name="dataSource">jdbc/WSO2CarbonDB_mount</Property>

  9. Open the master-datasources.xml file and update the "WSO2AM_DB" datasource as follows:

    <datasource>
      <name>WSO2AM_DB</name>
      <description>The datasource used for API Manager database</description>
      <jndiConfig>
                <name>jdbc/WSO2AM_DB</name>
      </jndiConfig>
      <definition type="RDBMS">
         <configuration>
            <url>jdbc:mysql://localhost:3306/wso2apim?autoReconnect=true&amp;relaxAutoCommit=true</url>
            <username>root</username>  
           <password>root</password>  
           <driverClassName>com.mysql.jdbc.Driver</driverClassName>  
           <maxActive>50</maxActive>  
           <maxWait>60000</maxWait>  
           <testOnBorrow>true</testOnBorrow>  
           <validationQuery>SELECT 1</validationQuery>  
           <validationInterval>30000</validationInterval> 
          </configuration>
       </definition>
  </datasource>

  10. Repeat these steps for the other two Governance Registry nodes, changing the value of "localMemberPort" to 4251 and 4252 respectively.

  11. Start each of the Governance Registry instances as described in Running the Product. First time you many need to start servers with -Dsetup argument.

    If you look at the logs in the Elastic Load Balancer management console, you will see the following types of entries indicating that the nodes joined the cluster:
    2013-02-19 01:06:55,647] INFO - RpcMembershipRequestHandler Received JOIN message from 127.0.0.1:4250(wso2.governance.domain) 
    [2013-02-19 01:06:55,648] INFO - MembershipManager Application member 127.0.0.1:4250(wso2.governance.domain) joined group wso2.governance.domain 
    [2013-02-19 01:07:06,652] INFO - DefaultGroupManagementAgent Application member Host:127.0.0.1, Port: 4250, HTTP:9764, HTTPS:9444, Domain: wso2.governance.domain, Sub-domain:null, Active:true joined application cluster 
    [2013-02-19 01:11:07,423] INFO - RpcMembershipRequestHandler Received JOIN message from 127.0.0.1:4251(wso2.governance.domain) 
    [2013-02-19 01:11:07,423] INFO - MembershipManager Application member 127.0.0.1:4251(wso2.governance.domain) joined group wso2.governance.domain 
    [2013-02-19 01:11:18,425] INFO - DefaultGroupManagementAgent Application member Host:127.0.0.1, Port: 4251, HTTP:9765, HTTPS:9445, Domain: wso2.governance.domain, Sub-domain:null, Active:true joined application cluster

  12. Point your browser to: https://governance.cluster.wso2.com:8243/carbon

You are now invoking Governance Registry through the load balancer. When there is high load, or if one of the Governance Registry instances becomes unavailable, the load balancer forwards traffic to the other nodes in the cluster. To setting up different databases you can refer to Setting up Databases.