Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

When WSO2 products are deployed in a clustered mode on Amazon EC2 instances, it is recommended to use the AWS clustering mode. As a best practice, it is recommended to add all nodes in a single cluster to the same AWS security group.

...

  1. Enable clustering for this server. 
    <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
  2. Set the membership scheme to aws to enable the AWS registration method. 
    <parameter name="membershipScheme">aws</parameter>
  3. Specify the port used to communicate cluster messages. This must be any port number between 5701 and 5800.
    <parameter name="localMemberPort">5701</parameter>
  4. Specify the IP of the instance in the <localMemberHost> element. This must be set to the IP address bound to the network interface used to communicate with other members in the group.  Here's an example:
    <parameter name="localMemberHost">127.0.0.1</parameter>

  5. Define the following AWS specific configurations. These are the AWS access key, secret key, security group, region (with or without the zone), tag key, and tag value. The AWS credentials and security group depend on your configurations in the Amazon EC2 instance. The tagKey and tagValue are optional and the rest of the following parameters are mandatory. The region defaults to us-east-1

    Code Block
    languagexml
    <parameter name="accessKey">xxxxxxxxxx</parameter>
    <parameter name="secretKey">yyyyyyyyyy<>yyyyyyyyyy</parameter>
    <parameter name="securityGroup">a_group_name</parameter>
    // <parameter name="securityGroup">a_group_name</parameter>iamRole">ec2-describe-test-role</parameter>
            
    
    <parameter name="region">us-east-1</parameter> 
    // Make Ifsure you do arenot addingadd aany zone (say zonevalues (e.g., a), it can be added as us-east-1-ato the region parameter.
    
    <parameter name="tagKey">a_tag_key</parameter>  
    <parameter name="tagValue">a_tag_value</parameter>  
    // If you are adding tags, they should be attached to the AWS instances.
    Tip

    Tip: In order to provide specific permissions to create an access key and secret key for only this AWS clustering attempt, use the following custom policy block. Attach this to the user account that will operate AWS clustering in WSO2 products. The access key and secret key can only be used to list EC2 instance details in the AWS accountAWS account. This can be used as an AWS policy for the IAM role as well.

    Code Block
    { "Version": "2012-10-17", 
    	 "Statement":
    	 [
    	 {
    		 "Effect": "Allow",
    		 "Action":
    			 [
    			 "ec2:DescribeAvailabilityZones",
    			 "ec2:DescribeInstances"
    			 ],
    			 "Resource": [ "*" ]
    	 }
    	 ]
    }

    You can use the following link as a reference on how to add the custom IAM policy: http://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_managed-policies.html

    For AWS clustering to work properly, you must set a specific tag key with a tag value for all EC2 instances that belong in the same clustertag key with a tag value for all EC2 instances that belong in the same cluster.

    Tip
    titleIAM Role

    You can use IAM Role instead of IAM user access keys in the AWS clustering scheme. The necessary policies can be associated with the IAM Role and assumed by the EC2 instance running the EI application. This means you can use the iamRole without accessKey and securityKey parameters.

  6. Start the server as per the instructions in the Setting up a Cluster topic. If the cluster is set up successfully, you should not see any errors when the server starts up, and also see the following log message.

    Code Block
    [2015-06-23 09:26:41,674]  INFO - HazelcastClusteringAgent Using aws based membership management scheme

    When new members join the cluster, you should see messages similar to the following.

    Code Block
    [2015-06-23 09:27:08,044]  INFO - AWSBasedMembershipScheme Member joined [5327e2f9-8260-4612-9083-5e5c5d8ad567]: /10.0.0.172:5701 

    When members leave the cluster, you should see messages similar to the following.

    Code Block
    [2015-06-23 09:28:34,364]  INFO - AWSBasedMembershipScheme Member left [b2a30083-1cf1-46e1-87d3-19c472bb2007]: /10.0.0.245:5701
Expand
titleClick here to view a sample of the configurations in the axis2.xml file.
Code Block
languagexml
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">  
        <parameter name="AvoidInitiation">true</parameter>  
        <parameter name="membershipScheme">aws</parameter>  
        <parameter name="domain">wso2.carbon.domain</parameter>  
  domain</parameter>  
        <parameter name="iamRole">ec2-describe-test-role</parameter>
        <parameter name="localMemberPort">5701</parameter>  
        <parameter name="accessKey">xxxxxxxxxxxx</parameter>  
        <parameter name="secretKey">yyyyyyyyyyyy</parameter>  
        		<parameter name="securityGroup">a_group_name</parameter>  
        <parameter name="region">us-east-1</parameter>  
        <parameter name="tagKey">a_tag_key</parameter>  
        <parameter name="tagValue">a_tag_value</parameter>   
  
        <parameter name="properties">  
            <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>  
            <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>  
            <property name="subDomain" value="worker"/>  
        </parameter>  
 </clustering>

...

Note

Note: This scheme of trying to establish connections with open Hazelcast ports from one EC2 instance to another does not violate any AWS security policies because the connection establishment attempts are made from nodes within the same security group to ports that are allowed within that security group.that security group.

Note

Note: To enable logs in Hazelcast, set the Hazelcast log level to FINEST in the <PRODUCT_HOME>/repository/conf/etc/logging-bridge.properties file:

com.hazelcast.level = FINEST

Note

Note: When configuring this for WSO2 API Manager 2.0.0 and more recent versions, you must configure ciphers for this in the <APIM_HOME>/repository/conf/tomcat/catalina_server.xml file. This is done so that the AWS ELB can communicate with the WSO2 API Manager. The following is a sample configuration.

Code Block
languagexml
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
                   port="9443"
                   proxyPort="443"
                   bindOnInit="false"
                   sslProtocol="TLS"
                   sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2"
                   maxHttpHeaderSize="8192"
                   acceptorThreadCount="2"
                   maxThreads="250"
                   minSpareThreads="50"
                   disableUploadTimeout="false"
                   enableLookups="false"
                   connectionUploadTimeout="120000"
                   maxKeepAliveRequests="200"
                   acceptCount="200"
                   server="WSO2 Carbon Server"
                   clientAuth="false"
                   compression="on"
                   scheme="https"
                   ciphers="SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,SSL_DHE_RSA_WITH_DES_CBC_SHA,
         SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,
         TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA"
                   secure="true"
                   SSLEnabled="true"
                   compressionMinSize="2048"
                   noCompressionUserAgents="gozilla, traviata"
                   compressableMimeType="text/html,text/javascript,application/x-javascript,application/javascript,application/xml,text/css,application/xslt+xml,text/xsl,image/gif,image/jpg,image/jpeg"
                                           keystoreFile="${carbon.home}/repository/resources/security/wso2carbon.jks"
                                           keystorePass="wso2carbon"
                   URIEncoding="UTF-8"/>
Note

Note: The WSO2 server needs to access the AWS endpoint ec2.<region>-<zone>.amazonaws.com. For example, ec2.us-east-1-a.amazonaws.com. Therefore, if you need a proxy to connect to the Internet, set the proxy values in wso2server.sh -east-1.amazonaws.com. Therefore, if you need a proxy to connect to the Internet, set the proxy values in wso2server.sh as follows:
-Dhttps.proxyHost=<Host>

-Dhttps.proxyPort=<Port>

Be sure to use https.proxyHost instead of http.proxyHost
When you set the proxy parameters in wso2server.sh, each and every http/s call goes through the proxy. To avoid this, set the -Dhttp.nonProxyHosts as follows:
-
Dhttps
Dhttp.
proxyHost=<Host>
-Dhttps.proxyPort=<Port>
Be sure to use https.proxyHost instead of http.proxyHostnonProxyHosts="localhost|127.0.0.1|.*.local
Note

Note: You might have to import the Amazon endpoint certificates to the client truststore of the WSO2 servers.