Setting up a Cluster in AWS Mode

When WSO2 products are deployed in a clustered mode on Amazon EC2 instances, it is recommended to use the AWS clustering mode. As a best practice, it is recommended to add all nodes in a single cluster to the same AWS security group.

To enable AWS clustering mode, you must edit the clustering section in the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file. When enabling AWS clustering mode, it is imperative that you do the steps in this topic, in addition to the steps you follow in the Setting up a Cluster topic for other configuration files.

Open the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file and do the following changes.

1. Enable clustering for this server. 
   <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
2. Set the membership scheme to aws to enable the AWS registration method. 
   <parameter name="membershipScheme">aws</parameter>
3. Specify the port used to communicate cluster messages. This must be any port number between 5701 and 5800.
   <parameter name="localMemberPort">5701</parameter>

4. Specify the IP of the instance in the <localMemberHost> element. Here's an example:
   <parameter name="localMemberHost">127.0.0.1</parameter>


5. Define the following AWS specific configurations. These are the AWS access key, secret key, security group, region (with or without the zone), tag key, and tag value. The AWS credentials and security group depend on your configurations in the Amazon EC2 instance. The tagKey and tagValue are optional and the rest of the following parameters are mandatory. The region defaults to us-east-1. 

   <parameter name="accessKey">xxxxxxxxxx</parameter>
   <parameter name="secretKey">yyyyyyyyyy</parameter>
   <parameter name="securityGroup">a_group_name</parameter>
   <parameter name="region">us-east-1</parameter> 
   // If you are adding a zone (say zone a), it can be added as us-east-1-a. 
   <parameter name="tagKey">a_tag_key</parameter>  
   <parameter name="tagValue">a_tag_value</parameter>  

   Tip: In order to provide specific permissions to create an access key and secret key for only this AWS clustering attempt, use the following custom policy block. Attach this to the user account that will operate AWS clustering in WSO2 products. The access key and secret key can only be used to list EC2 instance details in the AWS account.

   { "Version": "2012-10-17", 
   	 "Statement":
   	 [
   	 {
   		 "Effect": "Allow",
   		 "Action":
   			 [
   			 "ec2:DescribeAvailabilityZones",
   			 "ec2:DescribeInstances"
   			 ],
   			 "Resource": [ "*" ]
   	 }
   	 ]
   }

   You can use the following link as a reference on how to add the custom IAM policy: http://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_managed-policies.html

   For AWS clustering to work properly, you must set a specific tag key with a tag value for all EC2 instances that belong in the same cluster.

6. Start the server as per the instructions in the Setting up a Cluster topic. If the cluster is set up successfully, you should not see any errors when the server starts up, and also see the following log message.

   [2015-06-23 09:26:41,674]  INFO - HazelcastClusteringAgent Using aws based membership management scheme

   When new members join the cluster, you should see messages similar to the following.

   [2015-06-23 09:27:08,044]  INFO - AWSBasedMembershipScheme Member joined [5327e2f9-8260-4612-9083-5e5c5d8ad567]: /10.0.0.172:5701 

   When members leave the cluster, you should see messages similar to the following.

   [2015-06-23 09:28:34,364]  INFO - AWSBasedMembershipScheme Member left [b2a30083-1cf1-46e1-87d3-19c472bb2007]: /10.0.0.245:5701

<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">  
        <parameter name="AvoidInitiation">true</parameter>  
        <parameter name="membershipScheme">aws</parameter>  
        <parameter name="domain">wso2.carbon.domain</parameter>  
  
        <parameter name="localMemberPort">5701</parameter>  
        <parameter name="accessKey">xxxxxxxxxxxx</parameter>  
        <parameter name="secretKey">yyyyyyyyyyyy</parameter>  
        <parameter name="securityGroup">a_group_name</parameter>  
        <parameter name="region">us-east-1</parameter>  
        <parameter name="tagKey">a_tag_key</parameter>  
        <parameter name="tagValue">a_tag_value</parameter>   
  
        <parameter name="properties">  
            <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>  
            <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>  
            <property name="subDomain" value="worker"/>  
        </parameter>  
 </clustering>

All nodes having the same set of cluster configuration parameters will belong to the same cluster. Hazelcast calls the AWS APIs and gets a set of nodes that satisfy the specified parameters (region, securityGroup, tagKey, tagValue).

When the WSO2 product server starts up, it creates a Hazelcast cluster. At that point, it calls EC2 APIs and gets the list of potential members in the cluster. To call the EC2 APIs, it requires the AWS credentials. This is the only time these credentials are used. AWS APIs are only used on startup to learn about other potential members in the cluster.

Once the EC2 instances are retrieved, Hazelcast will try to connect to potential members that are running on the same port as its localMember port. By default this port is 5701. If that port is open, it will try to do a Hazelcast handshake and add that member if it belongs to the same cluster domain (group). The new member will repeat the process of trying to connect to the next port (i.e., 5702 by default) in increments of 1, until the next port is not reachable.

The following is the pseudocode for this.

for each EC2 instance e
     port = localMemberPort;
     while(canConnect(e, port))
          addMemberIfIPossible(e, port)    // A Hazelcast member is running & in the same domain
          port = port +1

Subsequently, the connections established between members are point to point TCP connections. Member failures are detected through a TCP ping. So, once the member discovery is done, the rest of the interactions in the cluster are same as when the multicast and WKA (Well Known Address) modes are used.

With that facility, you are not required to provide any member IP addresses or hostnames, which may be impossible on an IaaS such as EC2.

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
                   port="9443"
                   proxyPort="443"
                   bindOnInit="false"
                   sslProtocol="TLS"
                   sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2"
                   maxHttpHeaderSize="8192"
                   acceptorThreadCount="2"
                   maxThreads="250"
                   minSpareThreads="50"
                   disableUploadTimeout="false"
                   enableLookups="false"
                   connectionUploadTimeout="120000"
                   maxKeepAliveRequests="200"
                   acceptCount="200"
                   server="WSO2 Carbon Server"
                   clientAuth="false"
                   compression="on"
                   scheme="https"
                   ciphers="SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,SSL_DHE_RSA_WITH_DES_CBC_SHA,
         SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,
         TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA"
                   secure="true"
                   SSLEnabled="true"
                   compressionMinSize="2048"
                   noCompressionUserAgents="gozilla, traviata"
                   compressableMimeType="text/html,text/javascript,application/x-javascript,application/javascript,application/xml,text/css,application/xslt+xml,text/xsl,image/gif,image/jpg,image/jpeg"
                                           keystoreFile="${carbon.home}/repository/resources/security/wso2carbon.jks"
                                           keystorePass="wso2carbon"
                   URIEncoding="UTF-8"/>