Clustering ES 2.0.0

This section describes how to set up a WSO2 Enterprise Store cluster and how to front this cluster with an Nginx load balancer. The following sections give you information and instructions on how to set up your cluster.

Clustering deployment pattern

In this pattern there are four WSO2 ES nodes; two nodes are publishers and two nodes are stores. This covers the minimum requirement for high availability of both the publisher and store. In this pattern, we allow access to the admin console through an external load balancer. The following image depicts the sample pattern this clustering deployment scenario will follow.

Setting up the databases

Do the following to setup the databases correctly.

Creating the databases

Do the following steps to create the databases necessary. Note that we use MySQL here as an example, but you can use any suitable database instead.

1. Download and install MySQL Server.

2. Download the MySQL JDBC driver.

3. Unzip the downloaded MySQL driver zipped archive, and copy the MySQL JDBC driver JAR (mysql-connector-java-x.x.xx-bin.jar) into the <PRODUCT_HOME>/repository/components/lib directory of both the manager and worker nodes.

4. Define the host name for configuring permissions for the new database by opening the /etc/hosts file and adding the following line:
   

   <MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com

   You would do this step only if your database is not on your local machine and on a separate server.

5. Enter the following command in a terminal/command window, where username is the username you want to use to access the databases:
   mysql -u username -p
6. When prompted, specify the password that will be used to access the databases with the username you specified.

7. Create the databases using the following commands, where <ES_HOME> is the path to any of the ES instances you installed, and username and password are the same as those you specified in the previous steps:

   create database ES_carbondb;
   use ES_carbondb;
   source [ES_HOME]/dbscripts/mysql.sql;
   grant all on ES_carbondb.* TO root@"xxx.xxx.xxx.xxx" identified by "root";
   
   create database ES_regdb;
   use ES_regdb;
   source [ES_HOME]/dbscripts/mysql.sql;
   grant all on ES_regdb.* TO root@"xxx.xxx.xx.xx" identified by "root";
   
   create database ES_Identitydb;
   use ES_Identitydb;
   source [ES_HOME]/dbscripts/identity/mysql.sql;
   grant all on ES_Identitydb.* TO root@"xxx.xxx.xx.xx" identified by "root";
   
   create database ES_Socialdb;
   use ES_Socialdb;
   source [ES_HOME]/dbscripts/social/mysql.sql;
   grant all on ES_Socialdb.* TO root@"xxx.xxx.xx.xx" identified by "root";
   
   create database ES_Storagedb;
   use ES_Storagedb;
   source [ES_HOME]/dbscripts/storage/mysql.sql;
   grant all on ES_Storagedb.* TO root@"xxx.xxx.xx.xx" identified by "root";
   
   create database ES_umdb;
   use ES_umdb;
   source [ES_HOME]/dbscripts/mysql.sql;
   grant all on ES_umdb.* TO root@"xxx.xxx.xx.xx" identified by "root";

Configuring the datasources

Do the following datasource configurations in all nodes of the cluster.

1. Open the <ES_HOME>/repository/conf/datasources/master-datasources.xml file and make the following additions to configure the datasources.

   CARBON_DB

     <datasource>
              <name>WSO2_CARBON_DB</name>
              <description>The datasource used for registry and user  manager</description>
              <jndiConfig>
                  <name>jdbc/WSO2CarbonDB</name>
              </jndiConfig>
              <definition type="RDBMS">
                  <configuration>
                      <url>jdbc:mysql://xxx.xxx.xx.xx:3306/ES_carbondb</url>
                      <username>root</username>
                      <password>root</password>
                      <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                      <maxActive>50</maxActive>
                      <maxWait>60000</maxWait>
                      <testOnBorrow>true</testOnBorrow>
                      <validationQuery>SELECT 1</validationQuery>
                      <validationInterval>30000</validationInterval>
                  </configuration>
              </definition>
   </datasource>

   USER_DB

   <datasource>
              <name>WSO2UM_DB</name>
              <description>The datasource used for registry and user manager</description>
              <jndiConfig>
                  <name>jdbc/UserDB</name>
              </jndiConfig>
              <definition type="RDBMS">
                  <configuration>
                      <url>jdbc:mysql://xxx.xxx.xx.xx:3306/ES_umdb</url>
                      <username>root</username>
                      <password>root</password>
                      <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                      <maxActive>50</maxActive>
                      <maxWait>60000</maxWait>
                      <testOnBorrow>true</testOnBorrow>
                      <validationQuery>SELECT 1</validationQuery>
                      <validationInterval>30000</validationInterval>
                  </configuration>
              </definition>
   </datasource>

   REGISTRY_DB

   <datasource>
              <name>Registry_DB</name>
              <description>The datasource used for registry and user manager</description>
              <jndiConfig>
                  <name>jdbc/WSO2RegistryDB</name>
              </jndiConfig>
              <definition type="RDBMS">
                  <configuration>
                      <url>jdbc:mysql://xxx.xxx.xx.xx:3306/ES_regdb</url>
                      <username>root</username>
                      <password>root</password>
                      <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                      <maxActive>50</maxActive>
                      <maxWait>60000</maxWait>
                      <testOnBorrow>true</testOnBorrow>
                      <validationQuery>SELECT 1</validationQuery>
                      <validationInterval>30000</validationInterval>
                  </configuration>
              </definition>
   </datasource>

   IDENTITY_DB

   <datasource>
              <name>WSO2_IDENTITY_DB</name>
              <description>The datasource used for registry and user manager</description>
              <jndiConfig>
                  <name>jdbc/WSO2IdentityDB</name>
              </jndiConfig>
              <definition type="RDBMS">
                  <configuration>
                      <url>jdbc:mysql://xxx.xxx.xx.xx:3306/ES_Identitydb</url>
                      <username>root</username>
                      <password>root</password>
                      <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                      <maxActive>50</maxActive>
                      <maxWait>60000</maxWait>
                      <testOnBorrow>true</testOnBorrow>
                      <validationQuery>SELECT 1</validationQuery>
                      <validationInterval>30000</validationInterval>
                  </configuration>
              </definition>
   </datasource>

2. Open the <ES_HOME>/repository/conf/datasources/social-datasources.xml file and make the following additions to configure the datasources for the Social DB. This must be done in all the nodes.

   SOCIAL_DB

   <datasource>
              <name>WSO2_SOCIAL_DB</name>
              <description>The datasource used for registry and user manager</description>
              <jndiConfig>
                  <name>jdbc/WSO2SocialDB</name>
              </jndiConfig>
              <definition type="RDBMS">
                  <configuration>
                      <url>jdbc:mysql://xxx.xxx.xx.xx:3306/ES_Socialdb</url>
                      <username>root</username>
                      <password>root</password>
                      <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                      <maxActive>50</maxActive>
                      <maxWait>60000</maxWait>
                      <testOnBorrow>true</testOnBorrow>
                      <validationQuery>SELECT 1</validationQuery>
                      <validationInterval>30000</validationInterval>
                  </configuration>
              </definition>
   </datasource>

User management datasource configuration

This section provides instructions on how to configure the datasource for the user management DB. Do the following configuration for all nodes.

• Change the datasource property in the <ES_HOME>/repository/conf/user-mgt.xml file.
  <Property name="dataSource">jdbc/WSO2UM_DB</Property>

Mounting the registry

The config and governance DB must be mounted in order to share these resources across the cluster.

• Navigate to the <ES_HOME>/repository/conf/registry.xml file, and add the following configuration to create the registry mounts, one for the config space and one for the governance space. This should be done for all publisher and store nodes.

  <dbConfig name="wso2registry">
      	<dataSource>jdbc/WSO2CarbonDB</dataSource>
  </dbConfig>
  <dbConfig name="sharedregistry">
  	<dataSource>jdbc/WSO2RegistryDB</dataSource>
  </dbConfig>
  
  <remoteInstance url="https://localhost:9443/registry">
  	<id>instanceid</id>
  	<dbConfig>sharedregistry</dbConfig>
  	<readOnly>false</readOnly>
  	<enableCache>true</enableCache>
  	<registryRoot>/</registryRoot>
  	<cacheId>root@jdbc:mysql://xxx.xxx.xx.xx:3306/ES_regdb?autoReconnect=true</cacheId>
  </remoteInstance>
  
  <mount path="/_system/config" overwrite="true">
  	<instanceId>instanceid</instanceId>
  	<targetPath>/_system/configPub</targetPath>
  </mount>
  
  <mount path="/_system/governance" overwrite="true">
  	<instanceId>instanceid</instanceId>
  	<targetPath>/_system/governance</targetPath>
  </mount>

Configuring clustering for the publisher nodes

Do the following changes for all the publisher nodes in the cluster.

1. Make the following configuration changes in the <ES_HOME>/repository/conf/axis2/axis2.xml file. These changes are related to clustering.
   1. Enable clustering for this node by setting enable to true in the clustering tag.
      <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
   2. Set the membershipScheme parameter to wka to enable the well-known address registration method (this node will send cluster initiation messages to WKA members that we define later).
      <parameter name="membershipScheme">wka</parameter> 
   3. Specify the name of the cluster domain that this node is set to join.
      <parameter name="domain">wso2.es.domain</parameter>
   4. Specify the host used to communicate cluster messages. This is the IP address of the publisher node you are configuring.
      <parameter name="localMemberHost">xxx.xxx.xxx.xx1</parameter>
   5. Specify the port used to communicate cluster messages:
      <parameter name="localMemberPort">4250</parameter>

   6. Specify the well known member. Here, the well known member is the other publisher node.

      <members>
      	<member>
      		<hostName>xxx.xxx.xxx.xx2</hostName>
      		<port>4254</port>
      	</member>
      </members>

   7. The subDomain property must be commented out since this setup does not involve setup that supports worker manager separation.

      <parameter name="properties">
      	<property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
      	<property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
      	<!--property name="subDomain" value="mgt"/-->
      </parameter>

2. Make the following configuration changes in the <ES_HOME>/repository/conf/carbon.xml file.
   1. Configure the HostName for each publisher node.
      <HostName>publisher.es-wso2.com</HostName> 

   2. Enable SVN-based deployment synchronization with the AutoCommit property marked as true.

      <DeploymentSynchronizer>
      	<Enabled>true</Enabled>
      	<AutoCommit>true</AutoCommit>
      	<AutoCheckout>true</AutoCheckout>
      	<RepositoryType>svn</RepositoryType>
      	<SvnUrl>xxxxxxxxxxxxxxxxx</SvnUrl>
      	<SvnUser>xxxx</SvnUser>
      	<SvnPassword>xxxx</SvnPassword>
      	<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
      </DeploymentSynchronizer>

3. Map the host names to the IP. Add the following host entries to your DNS, or “/etc/hosts” file (in Linux) in all the nodes of the cluster. You have to map the host names with the IP address of the load balancer.
   <IP-of-Nginx> publisher.es-wso2.com 

Configuring clustering for the store nodes

Do the following changes for all the store nodes in the cluster.

1. Make the following configuration changes in the <ES_HOME>/repository/conf/axis2/axis2.xml file. These changes are related to clustering.
   1. Enable clustering for this node by setting enable to true in the clustering tag.
      <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
   2. Set the membershipScheme parameter to wka to enable the well-known address registration method (this node will send cluster initiation messages to WKA members that we define later).
      <parameter name="membershipScheme">wka</parameter>
   3. Specify the name of the cluster domain that this node is set to join.
      <parameter name="domain">wso2.es.domain</parameter>
   4. Specify the host used to communicate cluster messages. This is the IP address of the publisher node you are configuring.
      <parameter name="localMemberHost">xxx.xxx.xxx.xx3</parameter>
   5. Specify the port used to communicate cluster messages:
      <parameter name="localMemberPort">4251</parameter>

   6. Specify the well known member. Here, the well known member is the other publisher node.

      <members>
      	<member>
      		<hostName>xxx.xxx.xxx.xx4</hostName>
      		<port>4252</port>
      	</member>
      </members>

   7. The subDomain property must be commented out since this setup does not involve setup that supports worker manager separation.

      <parameter name="properties">
      	<property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
      	<property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
      	<!--property name="subDomain" value="mgt"/-->
      </parameter>

2. Make the following configuration changes in the <ES_HOME>/repository/conf/carbon.xml file.
   1. Configure the HostName for each publisher node.
      <HostName>store.es-wso2.com</HostName>

   2. Enable SVN-based deployment synchronization with the AutoCommit property marked as true.

      <DeploymentSynchronizer>
      	<Enabled>true</Enabled>
      	<AutoCommit>true</AutoCommit>
      	<AutoCheckout>true</AutoCheckout>
      	<RepositoryType>svn</RepositoryType>
      	<SvnUrl>xxxxxxxxxxxxxxxxx</SvnUrl>
      	<SvnUser>xxxx</SvnUser>
      	<SvnPassword>xxxx</SvnPassword>
      	<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
      </DeploymentSynchronizer>

3. Modify the <ES_HOME>/repository/conf/security/sso-idp-config.xml file with the AssertionConsumerService URLs of SSOIdentityProviderConfig as indicated below.

   <Issuer>store</Issuer> 
    
   <AssertionConsumerService>https://store.es-wso2.com/store/acs</AssertionConsumerService>
   <SignResponse>true</SignResponse>
   <CustomLoginPage>/store/login.jag</CustomLoginPage>
   
   
   <AssertionConsumerService>https://publisher.es-wso2.com/publisher/acs</AssertionConsumerService>
   <SignResponse>true</SignResponse>
   <CustomLoginPage>/publisher/controllers/login.jag</CustomLoginPage>

4. Configure single sign-on with the Identity Server. To do this, modify the <ES_HOME>/repository/deployment/server/jaggeryapps/publisher/config/publisher.json file with the following. You must configure this for all nodes that require single sign-on.

   "authentication":{
   	"activeMethod":"sso",
   	"methods":{
   		"sso":{
   			"attributes":{
   				"issuer":"publisher", 
   				"identityProviderURL":"https://publisher.es-wso2.com/samlsso",
   				"keyStorePassword":"wso2carbon",
   				"identityAlias":"wso2carbon",
   				"responseSigningEnabled":"true",
   				"acs":"%https.host%/publisher/acs",
   				"keyStoreName":"/repository/resources/security/wso2carbon.jks"
   				}
   			},
   		"basic":{
   			"attributes":{
   			}
   		}
   	}
   }

5. Map the host names to the IP. Add the following host entries to your DNS, or “/etc/hosts” file (in Linux) in all the nodes of the cluster. You have to map the host names with the IP address of the load balancer.
   <IP-of-Nginx> store.es-wso2.com

Configuring Nginx as the load balancer

The load balancer automatically distributes incoming traffic across multiple WSO2 product instances. It enables you to achieve greater levels of fault tolerance in your cluster, and provides the required balancing of load needed to distribute traffic.

Use the following steps to configure NGINX Plus version 1.7.11 as the load balancer for WSO2 products.

1. Install NGINX Plus in a server configured in your cluster.

2. Configure NGINX Plus to direct the requests to the publisher nodes. To do this, create a VHost file (es.pub.conf) in the /etc/nginx/conf.d directory and add the following configurations into it.

   upstream espub {
      	server xxx.xxx.xxx.xx1:9443;
       server xxx.xxx.xxx.xx2:9443;
   }
   
   server {
      	listen   80;
      	server_name publisher.es-wso2.com;
      	rewrite ^/(.*) https://publisher.es-wso2.com/$1 permanent;
   
   
      	proxy_set_header Upgrade $http_upgrade;
      	proxy_set_header Connection "upgrade";
      	proxy_http_version 1.1;
   
   }
   
   server {
     	listen 443;
     	server_name publisher.es-wso2.com;
   
     	ssl on;
     	ssl_certificate /etc/nginx/ssl/es.crt;
     	ssl_certificate_key /etc/nginx/ssl/es.key;
   
      	proxy_set_header Upgrade $http_upgrade;
      	proxy_set_header Connection "upgrade";
      	proxy_http_version 1.1;
   
     	location / {
         	index index.html;
         	proxy_set_header X-Forwarded-Host $host;
         	proxy_set_header X-Forwarded-Server $host;
         	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   
         	proxy_set_header Host $http_host;
         	proxy_read_timeout 5m;
         	proxy_send_timeout 5m;
   
         	proxy_pass https://espub;
   		proxy_redirect https://espub https://publisher.es-wso2.com/;
   
   
     	}
   }

3. Configure NGINX Plus to direct the requests to the store nodes. To do this, create a VHost file (es.store.conf) in the /etc/nginx/conf.d directory and add the following configurations into it.

   upstream esstore {
      	server xxx.xxx.xxx.xx3:9443;
       server xxx.xxx.xxx.xx4:9443;
   }
   
   server {
      	listen   80;
      	server_name publisher.es-wso2.com;
      	rewrite ^/(.*) https://store.es-wso2.com/$1 permanent;
   
   
      	proxy_set_header Upgrade $http_upgrade;
      	proxy_set_header Connection "upgrade";
      	proxy_http_version 1.1;
   
   }
   
   server {
     	listen 443;
     	server_name store.es-wso2.com;
   
     	ssl on;
     	ssl_certificate /etc/nginx/ssl/es.crt;
     	ssl_certificate_key /etc/nginx/ssl/es.key;
   
      	proxy_set_header Upgrade $http_upgrade;
      	proxy_set_header Connection "upgrade";
      	proxy_http_version 1.1;
   
     	location / {
         	index index.html;
         	proxy_set_header X-Forwarded-Host $host;
         	proxy_set_header X-Forwarded-Server $host;
         	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   
         	proxy_set_header Host $http_host;
         	proxy_read_timeout 5m;
         	proxy_send_timeout 5m;
   
         	proxy_pass https://esstore;
   proxy_redirect https://espub https://store.es-wso2.com/;
   
   
     	}
   }

4. The Nginx/etc/hosts node IPs must be mapped with the host name.

   xxx.xxx.xxx.xx1 : publisher.es-wso2.com
   xxx.xxx.xxx.xx2 : publisher.es-wso2.com
   xxx.xxx.xxx.xx3 : store.es-wso2.com
   xxx.xxx.xxx.xx4 : store.es-wso2.com

Start running the servers

The next step is to start running all the ES servers. Your console indicates that members are joining the cluster. See Running the Product for more information on starting up the servers.