Setting Up Deployment Pattern 2

This page guides you through setting up deployment pattern 2, which is a HA clustered deployment of WSO2 Identity Server with WSO2 Identity Analytics. For more information about deployment pattern 2 and its high level architecture, see Deployment Patterns - Pattern 2. 

You can install multiple instances of WSO2 products in a cluster to ensure that if one instance becomes unavailable or is experiencing high traffic, another instance will seamlessly handle the requests. For complete information on clustering concepts, see Clustering Overview in the Common Product Administration Guide.

Creating a cluster of WSO2 Identity Server instances involves a standard two node cluster for high availability. To ensure that the instances share governance registry artifacts, you must create a JDBC mount.

At a high level, use the following options to cluster Identity Server with a minimum of two nodes. The first section includes instructions on setting up databases. The second section involves setting up a standard two node cluster, the third section involves setting up the Identity Server dashboard in a clustered environment and the third section includes additional configurations if you need to set up a load balancer to front your cluster.

Setting up the databases

Each Carbon-based product uses a database to store information such as user management details and registry data. All nodes in the cluster must use one central database for config and governance registry mounts. By default, each WSO2 product is shipped with an embedded H2 database that works for all types of data. 

You can create the following databases and associated datasources. This is NOT mandatory and you can choose to not create these databases if you wish and simply have a single database to handle all these concerns.

WSO2_USER_DB

<PRODUCT_HOME>/dbscripts/
<PRODUCT_HOME>/dbscripts/identity/

<PRODUCT_HOME>/dbscripts/

<PRODUCT_HOME>/dbscripts/identity/

<PRODUCT_HOME>/dbscripts/

<PRODUCT_HOME>/dbscripts/

<PRODUCT_HOME>/dbscripts/

<PRODUCT_HOME>/dbscripts/

<PRODUCT_HOME>/dbscripts/

To understand this concept further, see the following diagram.

For more information on the concept of sharing governance and config registry databases across the cluster, see the topic on Sharing Databases in a Cluster in the WSO2 Product Administration Guide.

Do the following configurations to implement the database setup.

Creating the databases

Create the databases that you require using the RDBMS of your choice and run the relevant script for the RDBMS of your choice according to the table above. For instructions on how to do this, see Setting up the Physical Database in the WSO2 Product Administration Guide.

Configuring the datasources

Configure the datasources for the databases that you created above in both the WSO2 Identity Server nodes of your cluster. For instructions on how to configure the datasources for the databases you created, see Changing the Carbon Database in the WSO2 Product Administration Guide.

<datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration">     <providers>
        <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider>
    </providers>
    <datasources>
        <datasource>
            <name>REGISTRY_LOCAL1</name>
            <description>The datasource used for registry- local</description>
            <jndiConfig>
                <name>jdbc/WSO2CarbonDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL1?autoReconnect=true</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
        <datasource>
            <name>REGISTRY_DB</name>
            <description>The datasource used for registry- config/governance</description>
            <jndiConfig>
                <name>jdbc/WSO2RegistryDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
         <datasource>
            <name>WSO2_USER_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2UMDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
         <datasource>
            <name>IDENTITY_DB</name>
            <description>The datasource used for identity data</description>
            <jndiConfig>
                <name>jdbc/WSO2IDENTITYDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/IDENTITY_DB</url>
                    <username>regadmin</username>
                    <password>regadmin</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
   </datasources>
</datasources-configuration>

<datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration"> 
     <providers> 
        <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider> 
    </providers> 
    <datasources> 
        <datasource> 
            <name>REGISTRY_LOCAL2</name> 
            <description>The datasource used for registry- local</description> 
            <jndiConfig> 
                <name>jdbc/WSO2CarbonDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_LOCAL2?autoReconnect=true</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
        <datasource> 
            <name>REGISTRY_DB</name> 
            <description>The datasource used for registry- config/governance</description> 
            <jndiConfig> 
                <name>jdbc/WSO2RegistryDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/REGISTRY_DB?autoReconnect=true</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
         <datasource> 
            <name>WSO2_USER_DB</name> 
            <description>The datasource used for registry and user manager</description> 
            <jndiConfig> 
                <name>jdbc/WSO2UMDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/WSO2_USER_DB</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
         <datasource> 
            <name>IDENTITY_DB</name> 
            <description>The datasource used for identity data</description> 
            <jndiConfig> 
                <name>jdbc/WSO2IDENTITYDB</name> 
            </jndiConfig> 
            <definition type="RDBMS"> 
                <configuration> 
                    <url>jdbc:mysql://carbondb.mysql-wso2.com:3306/IDENTITY_DB</url> 
                    <username>regadmin</username> 
                    <password>regadmin</password> 
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
                    <maxActive>50</maxActive> 
                    <maxWait>60000</maxWait> 
                    <testOnBorrow>true</testOnBorrow> 
                    <validationQuery>SELECT 1</validationQuery> 
                    <validationInterval>30000</validationInterval> 
                </configuration> 
            </definition> 
        </datasource> 
   </datasources> 
</datasources-configuration>

Clustering Identity Server for high availability

The following diagram represents the typical two node WSO2 Identity Server cluster in active-active mode. Note that this representation does not include the load balancer.

Figure: Identity Server cluster for high availability

Do the following configurations to set this up.

1. Install Identity Server on each node.

2. Do the following changes to the <IS_HOME>/repository/conf/axis2/axis2.xml file for both nodes.

   1. Enable clustering on node 1 and node 2 by setting the clustering element to true:  
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

   2. Specify the name of the cluster this node will join.
<parameter name="domain">wso2.is.domain</parameter>

   3. Use the well known address (WKA) based clustering method. In WKA-based clustering, we need to have a subset of cluster members configured in all the members of the cluster. At least one well known member has to be operational at all times.
      <parametername="membershipScheme">wka</parameter>

   4. Configure the localMemberHost and localMemberPort entries. These must be different port values for the two nodes if they are on the same server to prevent any conflicts.

      <parametername="localMemberHost">127.0.0.1</parameter>
      <parametername="localMemberPort">4000</parameter>

   5. Under the members section, add the hostName and port for each WKA member. As we have only two nodes in our sample cluster configuration, we will configure both nodes as WKA nodes.

      <members>
          <member>
            <hostName>127.0.0.1</hostName>
            <port>4000</port>
          </member>
          <member>
            <hostName>127.0.0.2</hostName>
            <port>4010</port>
          </member>
      </members>

      Note: You can also use IP address ranges for the hostName. For example,  192.168.1.2-10. This should ensure that the cluster eventually recovers after failures. One shortcoming of doing this is that you can define a range only for the last portion of the IP address. You should also keep in mind that the smaller the range, the faster the time it takes to discover members since each node has to scan a lesser number of potential members.

3. Configure caching.

   From WSO2 Identity Server 5.2.0 onwards, distributed caching is disabled and it is not recommended to use this due to many practical issues that are related to configuring and running distributed caching properly. WSO2 Identity Server employs Hazelcast as the primary method of implementing cluster messages while using distributed caching in a simple setup.

   About Clustering

   For information on clustering, see Clustering WSO2 Products.

   About Caching

   • Why caching

     Caching is an additional layer on top of databases. It enables to keep the recently used data that are fetched from the database in local memory, so that for subsequent data requests instead of fetching from the database the data can be served from the local memory. Caching has certain advantages and disadvantages that you need to evaluate when deciding on your caching strategy.

   • Advantages

   • • The load on the underlying database or LDAP is reduced as data is served from already fetched data in memory.

     • Improved performance due to the reduced number of database calls for repetitive data fetching.

   • Disadvantages

   • • Coherency problems may occur when the data change is not immediately reflected on cached data if one node or an external system updates the database.

     • Data in memory can become stale yet be served, e.g., serving data from memory while its corresponding record in the database is deleted.

   Caching in WSO2 Identity Server

   Historically WSO2 Identity Server used distributed caching to utilize the above-mentioned advantages as well as to minimize the coherence problem. However, in newer deployment patterns where the network is not tightly controlled, distributed caching fail in unexpected ways. Hence, we no longer recommend using distributed caching. Instead, it is recommended to have local caches (if required) and cache invalidation messages (if required) by considering the information given below.

   • The ForceLocalCache property

     When Hazelcast clustering is enabled certain caches act as distributed caches. The ForceLocalCache property within the <cache> section in the carbon.xml file in the <IS_HOME>/repository/conf directory is there to mark that all the caches should act like local caches even in a clustered setup. (This is by default set to true)

     <ForceLocalCache>true</ForceLocalCache>

     Cache invalidation uses Hazelcast messaging to distribute the invalidation message over the cluster and invalidate the caches properly.  This is used to minimize the coherence problem in a multi-node setup.

   • Typical clustered deployment cache scenarios

     Scenario	Local Caching	Distributed Caching	Hazelcast Clustering	Distributed Invalidation	Description
     1. All caches are local with distributed cache invalidation	Enabled	Not Applicable	Enabled	Enabled	This is the recommended approach. Hazelcast messaging invalidates the caches.
     2. All caches are local without distributed cache invalidation	Enabled	Not Applicable	Disabled	Disabled	Invalidation clears only the caches in specific nodes. Other caches are cleared at cache expiration.  Hazelcast communication is not used. As the decisions take time to propagate over nodes (default cache timeout is 15 minutes), there is a security risk in this method. To reduce the risk, reduce the default cache timeout period. To learn how to reduce the default cache timeout period, see Configuring Cache Layers - timeout.
     3. No caching	Disabled	Disabled	Disabled	Disabled	The data are directly acquired from the database.  Eliminates the security risks caused due to not having cache invalidation. This method will create a performance degradation due to the lack of caching.
     4. Certain caches are disabled while the remaining are local	Enabled for the available local caches	Not Applicable	Enabled	Enabled	To reduce the security risk created in the second scenario and to improve performance in comparison with the third scenario, disable the security-related caches and sustain the performance-related caches as local caches.  This requires identification of these caches depending on the use case.
     5. Distributed caching enabled	Disabled—the ForceLocalCache property is set to false.	Enabled	Enabled	Not Applicable	This scenario is only recommended if the network has tight tolerance where the network infrastructure is capable of handling high bandwidth with very low latency.  Typically this applies only when you deploy all the nodes in a single server rack having fiber-optic cables. In any other environments, this implementation will cause cache losses. Thus, this implementation is not recommended for general use.

4. Change the datasource name to jdbc/WSO2UMDB in user-mgt.xml (located in <IS_HOME>/repository/conf/) and identity.xml (located in <IS_HOME>/repository/conf/identity) of both node1 and node2.

   user-mgt.xml

   <UserManager>
     <Realm>
     <Configuration>
     ...
     <Property name="dataSource">jdbc/WSO2UMDB</Property>
     </Configuration>
     ...
     </Realm>
   </UserManager>

   identity.xml

   <JDBCPersistenceManager>
      	 <DataSource>
      		<Name>jdbc/WSO2IDENTITYDB</Name>
      	 </DataSource>
      	    	 <!-- <SkipDBSchemaCreation>false</SkipDBSchemaCreation> -->
   </JDBCPersistenceManager>

5. Copy the JDBC driver (in this case MySQL driver) to the <IS_HOME>/repository/component/lib directory of both nodes. To do this, download the MySQL Java connector JAR from here and place it in the <IS_HOME>/repository/components/lib directory.

6. Point all cluster nodes to same user store (to share one LDAP directory). By default, WSO2 Identity Server is started with an embedded LDAP which comes with the product. Disable the embedded LDAP of node 2 by modifying embedded-ldap.xml which can be found in the <IS_HOME>/repository/conf/identity directory.

   <EmbeddedLDAP>
   	<Property name="enable">false</Property>
   <--------------------->
   <EmbeddedLDAP>

   Point node 2 to the default user store of node1. You need to configure the connection URL in user-mgt.xml of node2 as given below (default port is 10389). By default, the connection URL given in the file is  ldap://localhost:${Ports.EmbeddedLDAP.LDAPServerPort}.

   <UserStoreManager class="org.wso2.carbon.user.core.ldap.ReadWriteLDAPUserStoreManager">
       <------------>
   	<Property name="ConnectionURL">ldap://[IP_of_node1]:10389</Property>
   	<------------>
   </UserStoreManager >

   If you are using some other external user store, make sure you point both nodes to that external user store.

7. If both nodes will be running on the same server, set the port offset to avoid port conflicts. 

8. Start the nodes. Use the -Dsetup option (e.g., sh wso2server.sh -Dsetup) on both nodes.
9. Verify in the registry browser that the governance collection is shown with the symlink icon.
   1. Login to the management console.
   2. Navigate to, Home > Registry     > Browse
      

Setting up the dashboard

Follow the steps given below to set up the dashboard for the WSO2 Identity Server in a clustered environment.

1. Change the service provider configuration for the dashboard in the <IS_HOME>/repository/conf/identity/sso-idp-config.xml file.

   <AssertionConsumerServiceURLs><AssertionConsumerServiceURL>https://is.wso2.com/dashboard/acs</AssertionConsumerServiceURL></AssertionConsumerServiceURLs>
   <DefaultAssertionConsumerServiceURL>https://is.wso2.com/dashboard/acs</DefaultAssertionConsumerServiceURL>

2. Configure the  proxyHost and proxyHTTPSPort in the  <IS_HOME>/repository/deployment/server/jaggeryapps/dashboard/conf/site.json file with your IP or hostname and the port.

   "proxyHost" : "is.wso2.com",
   "proxyHTTPSPort" : "443",

3. Configure the  proxyHost and proxyHTTPSPort in the  <WSO2IS_HOME>/repository/deployment/server/webapps/shindig/WEB-INF/web.xml file with your IP or hostname and the port.

   shindig.host=is.wso2.com
   shindig.port=443

4. Optionally, configure the <IS_HOME>repository/conf/datasources/master-datasources.xml file to set up the user dashboard.

    Click here to view a sample datasource configuration.

   <datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration">
      <providers>
         <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider>
      </providers>
      <datasources>
         <datasource>
            <name>USER_DB</name>
            <description>The datasource used for users and authorization management</description>
            <jndiConfig>
               <name>jdbc/UserDB</name>
            </jndiConfig>
            <definition type="RDBMS">
               <configuration>
                  <url>jdbc:mysql://localhost:3306/USER_DB</url>
                  <username>root</username>
                  <password>root</password>
                  <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                  <maxActive>50</maxActive>
                  <maxWait>60000</maxWait>
                  <testOnBorrow>true</testOnBorrow>
                  <validationQuery>SELECT 1</validationQuery>
                  <validationInterval>30000</validationInterval>
               </configuration>
            </definition>
         </datasource>
         <datasource>
            <name>IDENTITY_DB</name>
            <description>The datasource used for WSO2 Identity Server specific data management</description>
            <jndiConfig>
               <name>jdbc/IdentityDB</name>
            </jndiConfig>
            <definition type="RDBMS">
               <configuration>
                  <url>jdbc:mysql://localhost:3306/IDENTITY_DB</url>
                  <username>root</username>
                  <password>root</password>
                  <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                  <maxActive>50</maxActive>
                  <maxWait>60000</maxWait>
                  <testOnBorrow>true</testOnBorrow>
                  <validationQuery>SELECT 1</validationQuery>
                  <validationInterval>30000</validationInterval>
               </configuration>
            </definition>
         </datasource>
         <datasource>
            <name>REG_DB</name>
            <description>The datasource used for registry- config/governance</description>
            <jndiConfig>
               <name>jdbc/RegistryDB</name>
            </jndiConfig>
            <definition type="RDBMS">
               <configuration>
                  <url>jdbc:mysql://localhost:3306/REG_DB?autoReconnect=true</url>
                  <username>root</username>
                  <password>root</password>
                  <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                  <maxActive>50</maxActive>
                  <maxWait>60000</maxWait>
                  <testOnBorrow>true</testOnBorrow>
                  <validationQuery>SELECT 1</validationQuery>
                  <validationInterval>30000</validationInterval>
               </configuration>
            </definition>
         </datasource>
         <datasource>
            <name>LOCAL_REG_DB_1</name>
            <description>The datasource used for local registry</description>
            <jndiConfig>
               <name>jdbc/WSO2CarbonDB</name>
            </jndiConfig>
            <definition type="RDBMS">
               <configuration>
                  <url>jdbc:h2:repository/database/WSO2CARBON_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000</url>
                  <username>wso2carbon</username>
                  <password>wso2carbon</password>
                  <driverClassName>org.h2.Driver</driverClassName>
                  <maxActive>50</maxActive>
                  <maxWait>60000</maxWait>
                  <testOnBorrow>true</testOnBorrow>
                  <validationQuery>SELECT 1</validationQuery>
                  <validationInterval>30000</validationInterval>
                  <defaultAutoCommit>false</defaultAutoCommit>
               </configuration>
            </definition>
         </datasource>
      </datasources>
   </datasources-configuration>

Setting up the cluster with a load balancer (Nginx)

If you need to set up the above WSO2 Identity Server cluster with Nginx, you can follow the instructions given below (you must do this after setting up the cluster following the above instructions). When clustering WSO2 Identity Server with a load balancer, make sure to enable sticky sessions. This is required for the management console and the dashboard to work and if we disable temporary session data persistence in the <IS_HOME>/repository/conf/identity/identity.xml file. 

For more information on sticky sessions, see Sticky Sessions with Manager Nodes. The following is the deployment diagram with the load balancer.

Configuring Nginx

Use the following steps to configure NGINX Plus version 1.7.11 or NGINX community version 1.9.2 as the load balancer for the WSO2 Identity Server.

1. Install NGINX Plus or Nginx community version in a server configured in your cluster.

2. Configure Nginx to direct the HTTP requests to the two IS nodes via the HTTP 80 port using the  http://is.wso2.com/. To do this, create a VHost file (is.http.conf) in the /etc/nginx/conf.d  directory and add the following configurations into it.

   Nginx Community Version and NGINX Plus

   upstream wso2.is.com {
           server xxx.xxx.xxx.xx3:9763;
           server xxx.xxx.xxx.xx4:9763;
   }
   
   server {
           listen 80;
           server_name is.wso2.com;
           location / {
                  proxy_set_header X-Forwarded-Host $host;
                  proxy_set_header X-Forwarded-Server $host;
                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                  proxy_set_header Host $http_host;
                  proxy_read_timeout 5m;
                  proxy_send_timeout 5m;
                  proxy_pass http://wso2.is.com;
    
   			   proxy_http_version 1.1;
           	   proxy_set_header Upgrade $http_upgrade;
           	   proxy_set_header Connection "upgrade";
           }
   }

3. Configure NGINX Plus or Nginx community version to direct the HTTPS requests to the two IS nodes via the HTTPS 443 port using  https://is.wso2.com/. To do this, create a VHost file (is.https.conf) in the /etc/nginx/conf.d directory and add the following configurations into it.

   Note: The configurations for Nginx community version and NGINX Plus are different here since the community version does not support the sticky directive.

4. Configure Nginx to access the Management Console as  https://mgt.is.wso2.com/carbon  via HTTPS 443 port. This is to direct requests to the manager node. To do this, create a VHost file ( mgt.is.https.conf) in the /etc/nginx/conf.d directory and add the following configurations into it.

   Nginx Community Version and NGINX Plus

   server {
   	listen 443;
   	server_name mgt.is.wso2.com;
   	ssl on;
   	ssl_certificate /etc/nginx/ssl/mgt.crt;
   	ssl_certificate_key /etc/nginx/ssl/mgt.key;
   
   	location / {
                  proxy_set_header X-Forwarded-Host $host;
                  proxy_set_header X-Forwarded-Server $host;
                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                  proxy_set_header Host $http_host;
                  proxy_read_timeout 5m;
                  proxy_send_timeout 5m;
   			   proxy_pass https://xxx.xxx.xxx.xx2:9443/;
    
   			   proxy_http_version 1.1;
   			   proxy_set_header Upgrade $http_upgrade;
   			   proxy_set_header Connection "upgrade";
       	}
   	error_log  /var/log/nginx/mgt-error.log ;
              access_log  /var/log/nginx/mgt-access.log;
   }

5. Restart the Nginx server.
   $sudo service nginx restart


   Tip: You do not need to restart the server if you are simply making a modification to the VHost file. The following command should be sufficient in such cases.
   $sudo service nginx reload  

   Also, import the ssl certificate used in the nginx to the wso2is-5.3.0/repository/resources/security/client-truststore.jks of the Identity Server nodes.

Create SSL certificates

Create SSL certificates for both the Identity Server nodes using the instructions that follow.

1. Create the Server Key.
   $sudo openssl genrsa -des3 -out server.key 1024
2. Certificate Signing Request.
   $sudo openssl req -new -key server.key -out server.csr
3. Remove the password.
   $sudo cp server.key server.key.org 
$sudo openssl rsa -in server.key.org -out server.key
4. Sign your SSL Certificate.
   $sudo openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

While creating keys, enter the hostname (is.wso2.com or mgt.is.wso2.com) as the common name.

Identity Server node 1 configuration

Configure the Identity Server node 1 using the following steps.

1. Go to the <IS_HOME>/repository/conf/tomcat/catalina-server.xml file and add the proxy port as 443.

   <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
             port="9443"
             proxyPort="443"
             ........
   
   <!--
           optional attributes:
   
           proxyPort="80"
   -->
   
   <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
             port="9763"
             proxyPort="80"

2. Configure deployment synchronizer in carbon.xml (autocommit=false in node 2). See Configuring SVN-Based Deployment Synchronizer for more details on how to configure this.

   Note: This is optional and you only need to do this if you configure a secondary user store manager through the management console.

   <deploymentsynchronizer>
    	<enabled>true</enabled>
    	<autocommit>true</autocommit>
    	<autocheckout>true</autocheckout>
    	<repositorytype>svn</repositorytype>
    	<svnurl>http://svnexample.wso2.com/svn/test</svnurl>
    	<svnuser>wso2</svnuser>
    	<svnpassword>wso2123</svnpassword>
    	<svnurlappendtenantid>true</svnurlappendtenantid>
   </deploymentsynchronizer>

3. In the <IS_HOME>/repository/conf/carbon.xml directory, define the hostname for your server.

   <HostName>wso2.is.com</HostName>
   
   <MgtHostName>wso2.is.com</MgtHostName>

   This hostname is used by the IS cluster. It must be specified in the /etc/hosts file as:

   127.0.0.1   wso2.is.com

Identity Server node 2 configuration

Follow all the configuration steps that were done in node 1 for node 2 as well. 

Running the cluster

1. Start Nginx and the Identity Server nodes.

2. Now you can access the management console using the following URL: https://wso2.is.com/carbon/

Minimum High Availability Deployment for WSO2 IS Analytics

This section explains how to configure WSO2 Identity Server Analytics in a distributed setup. You can configure alerts to monitor these APIs and detect unusual activity, manage locations via geo location statistics and to carry out detailed analysis of logs relating to the APIs. WSO2 IS Analytics is powered by WSO2 DAS. The following diagram indicates the minimum deployment pattern used for high availability.

WSO2 Identity Server Analytics supports a deployment scenario that has focus on high availability (HA) along with HA processing. To enable HA processing, you should have two WSO2 IS Analytics servers in a cluster.

For this deployment, both nodes should be configured to receive all events. To achieve this, clients can either send all the requests to both the nodes or each request to any one of the two nodes (i.e., using load balancing or failover mechanisms). If clients send all the requests to both nodes, the user has to specify that events are duplicated in the cluster (i.e., the same event comes to all the members of the cluster). Alternatively, if a client sends a request to one node, internally it sends that particular request to the other node as well. This way, even if the clients send requests to only one node, both IS Analytics nodes receive all the requests. 

In this scenario, one IS Analytics node works in active mode and the other works in passive mode. However, both nodes process all the data.

If the active node fails, the other node becomes active and receives all the requests.

When the failed node is up again, it fetches  all the internal states of the current active node via synching.

The newly arrived node then becomes the passive node and starts processing all the incoming messages to keep its state synched with the active node so that it can become active if the current active node fails.

Prerequisites

Before you configure a minimum high availability IS Analytics cluster, the following needs to be carried out.

1. Download the WSO2 IS Analytics distribution. Click DOWNLOAD ANALYTICS in the WSO2 Identity and Access Management page.
2. Take the following steps to install WSO2 IS Analytics. Sicne this procedure is identical to installing WSO2 Data Analytics Server (DAS), these steps take you to the DAS documentation for details.
   1. Ensure that you have met the Installation Prerequisites.
   2. Go to the installation instructions relevant to your operating system:
      
      • Installing on Linux
      • Installing on Windows
      • Installing as a Windows Service
      • Installing as a Linux Service
3. Follow the steps below to set up MySQL.

   1. Download and install MySQL Server.

   2. Download the MySQL JDBC driver.

   3. Unzip the downloaded MySQL driver zipped archive, and copy the MySQL JDBC driver JAR (mysql-connector-java-x.x.xx-bin.jar) into the <IS Analytics_HOME>/repository/components/lib directory of all the nodes in the cluster.

   4. Enter the following command in a terminal/command window, where username is the username you want to use to access the databases.
      mysql -u username -p 
   5. When prompted, specify the password that will be used to access the databases with the username you specified.

   6. Create two databases named userdb and regdb.

      About using MySQL in different operating systems

      For users of Microsoft Windows, when creating the database in MySQL, it is important to specify the character set as latin1. Failure to do this may result in an error (error code: 1709) when starting your cluster. This error occurs in certain versions of MySQL (5.6.x) and is related to the UTF-8 encoding. MySQL originally used the latin1 character set by default, which stored characters in a 2-byte sequence. However, in recent versions, MySQL defaults to UTF-8 to be friendlier to international users. Hence, you must use latin1 as the character set as indicated below in the database creation commands to avoid this problem. Note that this may result in issues with non-latin characters (like Hebrew, Japanese, etc.). The following is how your database creation command should look.

      mysql> create database <DATABASE_NAME> character set latin1;

      For users of other operating systems, the standard database creation commands will suffice. For these operating systems, the following is how your database creation command should look.

      mysql> create database <DATABASE_NAME>;

   7. Execute the following script for the two databases you created in the previous step.
      mysql> source <IS Analytics_HOME>/dbscripts/mysql.sql; 


      From WSO2 Carbon Kernel 4.4.6 onwards there are two MySQL DB scripts available in the product distribution. Click here to identify as to which version of the MySQL script to use.

       Click here to view the commands for performing steps f and g

      mysql> create database userdb;
      mysql> use userdb;
      mysql> source <IS Analytics_HOME>/dbscripts/mysql.sql;
      mysql> grant all on userdb.* TO username@localhost identified by "password";
       
       
      mysql> create database regdb;
      mysql> use regdb;
      mysql> source <IS Analytics_HOME>/dbscripts/mysql.sql;
      mysql> grant all on regdb.* TO username@localhost identified by "password";

   8. Configure the datasource in the <IS Analytics_HOME>/repository/conf/analytics/analytics-conf.xml file as shown in the code extract below. As it is possible to maintain the data in one database, you can point all three datasources to a single database.
      
      Alternatively, if you want to separate the data logically, create the following two databases in MySQL and point to the respective database as shown in the extract below. 

      • WSO2_ANALYTICS_EVENT_STORE_DB

      • WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB

      <analytics-record-store name="EVENT_STORE">
         <implementation>org.wso2.carbon.analytics.datasource.rdbms.RDBMSAnalyticsRecordStore</implementation>
         <properties>
            <property name="datasource">WSO2_ANALYTICS_EVENT_STORE_DB</property>
            <property name="category">read_write_optimized</property>
         </properties>
      </analytics-record-store>
      <analytics-record-store name="EVENT_STORE_WO">
         <implementation>org.wso2.carbon.analytics.datasource.rdbms.RDBMSAnalyticsRecordStore</implementation>
         <properties>
            <property name="datasource">WSO2_ANALYTICS_EVENT_STORE_DB</property>
            <property name="category">write_optimized</property>
         </properties>
      </analytics-record-store>
      <analytics-record-store name="PROCESSED_DATA_STORE">
         <implementation>org.wso2.carbon.analytics.datasource.rdbms.RDBMSAnalyticsRecordStore</implementation>
         <properties>
            <property name="datasource">WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB</property>
            <property name="category">read_write_optimized</property>
         </properties>
      </analytics-record-store>

Required configurations

When configuring the minimum high availability cluster following setups should be done for both nodes.

1. Do the following database-related configurations.

   1. Follow the steps below to configure the <IS Analytics_HOME>/repository/conf/datasources/master-datasources.xml file as required. 

      Note that you can point all these datasources to a single database as it is not technically neccessary to separate the data into different databases. However, if required, you can have separate databases as well.
      
      The steps given below demonstrate the flow assuming you have created separate databases for each. If you are using a single database instead, simply point the datasources indicated below to a single database.

      1. Enable all the nodes to access the users database by configuring a datasource to be used by user manager as shown below. 

         <datasource>
         	<name>WSO2UM_DB</name>
         	<description>The datasource used by user manager</description>
         	<jndiConfig>
           	<name>jdbc/WSO2UM_DB</name>
         	</jndiConfig>
         	<definition type="RDBMS">
           	<configuration>
             	<url>jdbc:mysql://[MySQL DB url]:[port]/userdb</url>
             	<username>[user]</username>
             	<password>[password]</password>
             	<driverClassName>com.mysql.jdbc.Driver</driverClassName>
             	<maxActive>50</maxActive>
             	<maxWait>60000</maxWait>
             	<testOnBorrow>true</testOnBorrow>
             	<validationQuery>SELECT 1</validationQuery>
             	<validationInterval>30000</validationInterval>
           	</configuration>
         	</definition>
         </datasource>

      2. Enable the nodes to access the registry database by configuring the WSO2REG_DB data source as follows.

         <datasource>
         	<name>WSO2REG_DB</name>
         	<description>The datasource used by the registry</description>
         	<jndiConfig>
           	<name>jdbc/WSO2REG_DB</name>
         	</jndiConfig>
         	<definition type="RDBMS">
           	<configuration>
             	<url>jdbc:mysql://[MySQL DB url]:[port]/regdb</url>
             	<username>[user]</username>
             	<password>[password]</password>
             	<driverClassName>com.mysql.jdbc.Driver</driverClassName>
             	<maxActive>50</maxActive>
             	<maxWait>60000</maxWait>
             	<testOnBorrow>true</testOnBorrow>
             	<validationQuery>SELECT 1</validationQuery>
             	<validationInterval>30000</validationInterval>
           	</configuration>
         	</definition>
         </datasource>

         For detailed information about registry sharing strategies, see the library article Sharing Registry Space across Multiple Product Instances.

   2. Point to your database WSO2_ANALYTICS_EVENT_STORE_DB and WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB in the <IS Analytics_HOME>/repository/conf/datasources/analytics-datasources.xml file as shown below.

      <datasources-configuration>
          <providers>
              <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider>
          </providers>
      
          <datasources>
              <datasource>
                  <name>WSO2_ANALYTICS_EVENT_STORE_DB</name>
                  <description>The datasource used for analytics record store</description>
                  <definition type="RDBMS">
                      <configuration>
                          <url>jdbc:mysql://[MySQL DB url]:[port]/WSO2_ANALYTICS_EVENT_STORE_DB</url>
                          <username>[username]</username>
                          <password>[password]</password>
                          <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                          <maxActive>50</maxActive>
                          <maxWait>60000</maxWait>
                          <testOnBorrow>true</testOnBorrow>
                          <validationQuery>SELECT 1</validationQuery>
                          <validationInterval>30000</validationInterval>
                          <defaultAutoCommit>false</defaultAutoCommit>
                      </configuration>
                  </definition>
              </datasource>
              <datasource>
                  <name>WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB</name>
                  <description>The datasource used for analytics record store</description>
                  <definition type="RDBMS">
                      <configuration>
                          <url>jdbc:mysql://[MySQL DB url]:[port]/WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB</url>
                          <username>[username]</username>
                          <password>[password]</password>
                          <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                          <maxActive>50</maxActive>
                          <maxWait>60000</maxWait>
                          <testOnBorrow>true</testOnBorrow>
                          <validationQuery>SELECT 1</validationQuery>
                          <validationInterval>30000</validationInterval>
                          <defaultAutoCommit>false</defaultAutoCommit>
                      </configuration>
                  </definition>
              </datasource>
          </datasources>
      </datasources-configuration>

      For more information, see Datasources in DAS documentation.

   3. To share the user store among the nodes, open the <IS Analytics_HOME>/repository/conf/user-mgt.xmlfile and modify the dataSource property of the <configuration> element as follows.

      <configuration> 
      ...
          <Property name="dataSource">jdbc/WSO2UM_DB</Property>
      </configuration>

      The datasource name specified in this configuration should be the same as the datasource used by user manager that you configured in sub step a, i.

   4. In the <IS Analytics_HOME>/repository/conf/registry.xml file, add or modify the dataSource attribute of the <dbConfig name="govregistry"> element as follows.

      <dbConfig name="govregistry">
      	<dataSource>jdbc/WSO2REG_DB</dataSource>
      </dbConfig>
      <remoteInstance url="https://localhost:9443/registry"> 
      	<id>gov</id>
      	<cacheId>user@jdbc:mysql://localhost:3306/regdb</cacheId>
      	<dbConfig>govregistry</dbConfig>
      	<readOnly>false</readOnly>
      	<enableCache>true</enableCache>
      	<registryRoot>/</registryRoot>
      </remoteInstance>
      <mount path="/_system/governance" overwrite="true">
      	<instanceId>gov</instanceId>
      	<targetPath>/_system/governance</targetPath>
      </mount>
      <mount path="/_system/config" overwrite="true">
      	<instanceId>gov</instanceId>
      	<targetPath>/_system/config</targetPath>
      </mount>

      Do not replace the following configuration when adding in the mounting configurations. The registry mounting configurations mentioned in the above steps should be added in addition to the following.

      <dbConfig name="wso2registry">
          <dataSource>jdbc/WSO2CarbonDB</dataSource>
      </dbConfig>

2. Update the <IS Analytics_HOME>/repository/conf/axis2/axis2.xml file as follows to enable Hazlecast clustering for both nodes. 
   

   1. Set clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" to true as shown below to enable Hazlecast clustering.

      <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

   2. Enable wka mode on both nodes as shown below.  For more information on wka mode, read About membership schemes.

      <parameter name="membershipScheme">wka</parameter>

   3. Add both the nodes as well known members in the cluster under the members tag in each node as shown in the example below.

      <members>
          <member>
              <hostName>[node1 IP]</hostName>
              <port>[node1 port]</port>
          </member>
          <member>
              <hostName>[node2 IP]</hostName>
              <port>[node2 port]</port>
          </member>
      </members>

   4. For each node, enter the respective server IP address as the value for the localMemberHost property as shown below.

      <parameter name="localMemberHost">[Server_IP_Address]</parameter>

    Click here to view the complete clustering section of the axis2.xml file. with the changes mentioned above.

       <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
                   enable="true">
   
           <!--
              This parameter indicates whether the cluster has to be automatically initalized
              when the AxisConfiguration is built. If set to "true" the initialization will not be
              done at that stage, and some other party will have to explictly initialize the cluster.
           -->
           <parameter name="AvoidInitiation">true</parameter>
   
           <!--
              The membership scheme used in this setup. The only values supported at the moment are
              "multicast" and "wka"
   
              1. multicast - membership is automatically discovered using multicasting
              2. wka - Well-Known Address based multicasting. Membership is discovered with the help
                       of one or more nodes running at a Well-Known Address. New members joining a
                       cluster will first connect to a well-known node, register with the well-known node
                       and get the membership list from it. When new members join, one of the well-known
                       nodes will notify the others in the group. When a member leaves the cluster or
                       is deemed to have left the cluster, it will be detected by the Group Membership
                       Service (GMS) using a TCP ping mechanism.
           -->
           
           <parameter name="membershipScheme">wka</parameter>
   
           <!--<parameter name="licenseKey">xxx</parameter>-->
           <!--<parameter name="mgtCenterURL">http://localhost:8081/mancenter/</parameter>-->
   
           <!--
            The clustering domain/group. Nodes in the same group will belong to the same multicast
            domain. There will not be interference between nodes in different groups.
           -->
           <parameter name="domain">wso2.carbon.domain</parameter>
   
           <!-- The multicast address to be used -->
           <!--<parameter name="mcastAddress">228.0.0.4</parameter>-->
   
           <!-- The multicast port to be used -->
           <parameter name="mcastPort">45564</parameter>
   
           <parameter name="mcastTTL">100</parameter>
   
           <parameter name="mcastTimeout">60</parameter>
   
           <!--
              The IP address of the network interface to which the multicasting has to be bound to.
              Multicasting would be done using this interface.
           -->
           <!--
               <parameter name="mcastBindAddress">10.100.5.109</parameter>
           -->
           <!-- The host name or IP address of this member -->
   
           <parameter name="localMemberHost">[node IP]</parameter>
   
           <!--
               The bind adress of this member. The difference between localMemberHost & localMemberBindAddress
               is that localMemberHost is the one that is advertised by this member, while localMemberBindAddress
               is the address to which this member is bound to.
           -->
           <!--
           <parameter name="localMemberBindAddress">[node IP]</parameter>
           -->
   
           <!--
           The TCP port used by this member. This is the port through which other nodes will
           contact this member
            -->
           <parameter name="localMemberPort">[node port]</parameter>
   
           <!--
               The bind port of this member. The difference between localMemberPort & localMemberBindPort
               is that localMemberPort is the one that is advertised by this member, while localMemberBindPort
               is the port to which this member is bound to.
           -->
           <!--
           <parameter name="localMemberBindPort">4001</parameter>
           -->
   
           <!--
           Properties specific to this member
           -->
           <parameter name="properties">
               <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
               <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
               <property name="subDomain" value="worker"/>
           </parameter>
   
           <!--
           Uncomment the following section to load custom Hazelcast data serializers.
           -->
           <!--
           <parameter name="hazelcastSerializers">
               <serializer typeClass="java.util.TreeSet">org.wso2.carbon.hazelcast.serializer.TreeSetSerializer
               </serializer>
               <serializer typeClass="java.util.Map">org.wso2.carbon.hazelcast.serializer.MapSerializer</serializer>
           </parameter>
           -->
   
           <!--
              The list of static or well-known members. These entries will only be valid if the
              "membershipScheme" above is set to "wka"
           -->
           <members>
               <member>
                   <hostName>[node1 IP]</hostName>
                   <port>[node1 port]</port>
               </member>
               <member>
                   <hostName>[node2 IP]</hostName>
                   <port>[node2 port]</port>
               </member>
           </members>
   
           <!--
           Enable the groupManagement entry if you need to run this node as a cluster manager.
           Multiple application domains with different GroupManagementAgent implementations
           can be defined in this section.
           -->
           <groupManagement enable="false">
               <applicationDomain name="wso2.as.domain"
                                  description="AS group"
                                  agent="org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent"
                                  subDomain="worker"
                                  port="2222"/>
           </groupManagement>
       </clustering>

3. Configure the  <IS Analytics_HOME>/repository/conf/event-processor.xml file as follows to cluster IS Analytics in the Receiver.

   1. Enable the HA mode by setting the following property.

      <mode name="HA" enable="true">

   2. Disable the Distributed mode by setting the following property.

      <mode name="Distributed" enable="false">

   3. For each node, enter the respective server IP address under the HA mode Config section as shown in the example below.

      When you enable the HA mode for WSO2 IS Analytics, the following are enabled by default:

      • State persistence: If there is no real time use case that requires any state information after starting the cluster, you should disable event persistence by setting the persistence attribute to false in the <IS Analytics_HOME>/repository/conf/event-processor.xml file as shown below.

        <persistence enable="false">
            <persistenceIntervalInMinutes>15</persistenceIntervalInMinutes>
            <persisterSchedulerPoolSize>10</persisterSchedulerPoolSize>
            <persister class="org.wso2.carbon.event.processor.core.internal.persistence.FileSystemPersistenceStore">
                <property key="persistenceLocation">cep_persistence</property>
            </persister>
        </persistence>

        When state persistence is enabled for WSO2 IS Analytics, the internal state of IS Analytics is persisted in files. These files are not automatically deleted. Therefore, if you want to save space in your IS Analytics pack, you need to delete them manually.

        These files are created in the <IS Analytics_HOME>/cep_persistence/<tenant-id> directory. This directory has a separate sub-directory for each execution plan. Each execution plan can have multiple files. The format of each file name is <TIMESTAMP>_<EXECUTION_PLAN_NAME> (e.g, 1493101044948_MyExecutionPlan). If you want to clear files for a specific execution plan, you need to leave the two files with the latest timestamps and delete the rest.

      • Event synchronization: However, if you set the event.duplicated.in.cluster=true property for an event receiver configured in a node, IS Analytics does not perform event synchronization for that receiver.

          <!-- HA Mode Config -->
          <mode name="HA" enable="true">
             ...
              <eventSync>
                  <hostName>[Server_IP_Address]</hostName>

    Click here to view the complete event-processor.xml file with the changes mentioned above.

   <eventProcessorConfiguration>
       <mode name="SingleNode" enable="false">
           <persistence enable="false">
               <persistenceIntervalInMinutes>15</persistenceIntervalInMinutes>
               <persisterSchedulerPoolSize>10</persisterSchedulerPoolSize>
               <persister class="org.wso2.carbon.event.processor.core.internal.persistence.FileSystemPersistenceStore">
                   <property key="persistenceLocation">cep_persistence</property>
               </persister>
           </persistence>
       </mode>
   
       <!-- HA Mode Config -->
       <mode name="HA" enable="true">
           <nodeType>
               <worker enable="true"/>
               <presenter enable="false"/>
           </nodeType>
           <checkMemberUpdateInterval>10000</checkMemberUpdateInterval>
           <eventSync>
               <hostName>172.18.1.217</hostName>
               <port>11224</port>
               <reconnectionInterval>20000</reconnectionInterval>
               <serverThreads>20000</serverThreads>
               <!--Size of TCP event publishing client's send buffer in bytes-->
               <publisherTcpSendBufferSize>5242880</publisherTcpSendBufferSize>
               <!--Character encoding of TCP event publishing client-->
               <publisherCharSet>UTF-8</publisherCharSet>
               <publisherBufferSize>1024</publisherBufferSize>
               <publisherConnectionStatusCheckInterval>30000</publisherConnectionStatusCheckInterval>
               <!--Number of events that could be queued at receiver before they are synced between CEP/DAS nodes-->
               <receiverQueueSize>1000000</receiverQueueSize>
               <!--Max total size of events that could be queued at receiver before they are synced between CEP/DAS nodes-->
               <receiverQueueMaxSizeMb>10</receiverQueueMaxSizeMb>
               <!--Number of events that could be queued at publisher to sync output between CEP/DAS nodes-->
               <publisherQueueSize>1000000</publisherQueueSize>
               <!--Max total size of events that could be queued at publisher to sync output between CEP/DAS nodes-->
               <publisherQueueMaxSizeMb>10</publisherQueueMaxSizeMb>
           </eventSync>
           <management>
               <hostName>172.18.1.217</hostName>
               <port>10005</port>
               <tryStateChangeInterval>15000</tryStateChangeInterval>
               <stateSyncRetryInterval>10000</stateSyncRetryInterval>
           </management>
           <presentation>
               <hostName>0.0.0.0</hostName>
               <port>11000</port>
               <!--Size of TCP event publishing client's send buffer in bytes-->
               <publisherTcpSendBufferSize>5242880</publisherTcpSendBufferSize>
               <!--Character encoding of TCP event publishing client-->
               <publisherCharSet>UTF-8</publisherCharSet>
               <publisherBufferSize>1024</publisherBufferSize>
               <publisherConnectionStatusCheckInterval>30000</publisherConnectionStatusCheckInterval>
           </presentation>
       </mode>
   
       <!-- Distributed Mode Config -->
       <mode name="Distributed" enable="false">
           <nodeType>
               <worker enable="true"/>
               <manager enable="true">
                   <hostName>0.0.0.0</hostName>
                   <port>8904</port>
               </manager>
               <presenter enable="false">
                   <hostName>0.0.0.0</hostName>
                   <port>11000</port>
               </presenter>
           </nodeType>
           <management>
               <managers>
                   <manager>
                       <hostName>localhost</hostName>
                       <port>8904</port>
                   </manager>
                   <manager>
                       <hostName>localhost</hostName>
                       <port>8905</port>
                   </manager>
               </managers>
               <!--Connection re-try interval to connect to Storm Manager service in case of a connection failure-->
               <reconnectionInterval>20000</reconnectionInterval>
               <!--Heart beat interval (in ms) for event listeners in "Storm Receivers" and "CEP Publishers" to acknowledge their
               availability for receiving events"-->
               <heartbeatInterval>5000</heartbeatInterval>
               <!--Storm topology re-submit interval in case of a topology submission failure-->
               <topologyResubmitInterval>10000</topologyResubmitInterval>
           </management>
           <transport>
               <!--Port range to be used for events listener servers in "Storm Receiver Spouts" and "CEP Publishers"-->
               <portRange>
                   <min>15000</min>
                   <max>15100</max>
               </portRange>
               <!--Connection re-try interval (in ms) for connection failures between "CEP Receiver" to "Storm Receiver" connections
               and "Storm Publisher" to "CEP Publisher" connections-->
               <reconnectionInterval>20000</reconnectionInterval>
               <!--Size of the output queue of each "CEP Receiver" which stores events to be published into "Storm Receivers" .
               This must be a power of two-->
               <cepReceiverOutputQueueSize>8192</cepReceiverOutputQueueSize>
               <!--Size of the output queue of each "Storm Publisher" which stores events to be published into "CEP Publisher" .
               This must be a power of two-->
               <stormPublisherOutputQueueSize>8192</stormPublisherOutputQueueSize>
               <!--Size of TCP event publishing client's send buffer in bytes-->
               <tcpEventPublisherSendBufferSize>5242880</tcpEventPublisherSendBufferSize>
               <!--Character encoding of TCP event publishing client-->
               <tcpEventPublisherCharSet>UTF-8</tcpEventPublisherCharSet>
               <!--Size of the event queue in each storm spout which stores events to be processed by storm bolts -->
               <stormSpoutBufferSize>10000</stormSpoutBufferSize>
               <connectionStatusCheckInterval>20000</connectionStatusCheckInterval>
           </transport>
           <presentation>
               <presentationOutputQueueSize>1024</presentationOutputQueueSize>
               <!--Size of TCP event publishing client's send buffer in bytes-->
               <tcpEventPublisherSendBufferSize>5242880</tcpEventPublisherSendBufferSize>
               <!--Character encoding of TCP event publishing client-->
               <tcpEventPublisherCharSet>UTF-8</tcpEventPublisherCharSet>
               <connectionStatusCheckInterval>20000</connectionStatusCheckInterval>
           </presentation>
           <statusMonitor>
               <lockTimeout>60000</lockTimeout>
               <updateRate>60000</updateRate>
           </statusMonitor>
           <stormJar>org.wso2.cep.storm.dependencies.jar</stormJar>
           <distributedUIUrl></distributedUIUrl>
           <memberUpdateCheckInterval>20000</memberUpdateCheckInterval>
       </mode>
   </eventProcessorConfiguration>

   The following node types are configured for the HA deployment mode in the <IS Analytics_HOME>/repository/conf/event-processor.xml file.

   • eventSync: Both the active and the passive nodes in this setup are event synchronizing nodes as explained in the introduction. Therefore, each node should have the host and the port on which it is operating specified under the <eventSync> element.

     Note that the eventSync port is not automatically updated to the port in which each node operates via port offset.

   • management: In this setup, both the nodes carry out the same tasks, and therefore, both nodes are considered manager nodes. Therefore, each node should have the host and the port on which it is operating specified under the <management> element.

     Note that the management port is not automatically updated to the port in which each node operates via port offset.

   • presentation: You can optionally specify only one of the two nodes in this setup as the presenter node. The dashboards in which processed information is displayed are configured only in the presenter node. Each node should have the host and the port on which the assigned presenter node is operating specified under the <presentation> element. The host and the port as well as the other configurations under the <presentation> element are effective only when the presenter enable="false property is set under the <!-- HA Mode Config --> section.

4. Update the <IS Analytics_HOME>/repository/conf/analytics/spark/spark-defaults.conf file as follows to use the Spark cluster embedded within IS Analytics.

   • Keep the carbon.spark.master configuration as local. This instructs Spark to create a Spark cluster using the Hazelcast cluster.

   • Enter 2 as the value for the carbon.spark.master.count configuration. This specifies that there should be two controllers in the Spark cluster. One controller serves as an active controller and the other serves as a stand-by controller. 

   The following example shows the <IS Analytics_HOME>/repository/conf/analytics/spark/spark-defaults.conf file with changes mentioned above.

   carbon.spark.master local
   carbon.spark.master.count 2

   For more information, see Spark Configurations in DAS documentation.

   Important: If the path to <IS Analytics_HOME> is different in the two nodes, please do the following. 

5. In order to share the C-Apps deployed among the nodes, configure the SVN-based deployment synchronizer. For detailed instructions, see Configuring SVN-Based Deployment Synchronizer.

   IS Analytics Minimum High availability Deployment set up does not use a manager and a worker. For the purpose of configuring the deployment synchronizer, you can add the configurations relevant to the manager for the node of your choice, and add the configurations relating to the worker for the other node.

   If you do not configure the deployment synchronizer, you are required to deploy any C-App you use in the IS Analytics Minimum High Availability Deployment set up to both the nodes.

6. If the physical IS Analytics server has multiple network interfaces with different IPs, and if you want Spark to use a specific Interface IP, open either the <IS Analytics_HOME>/bin/load-spark-env-vars.sh file (for Linux) or <IS Analytics_HOME>/bin/load-spark-env-vars.bat file (for Windows), and add the following parameter to configure the Spark IP address. 

   export  SPARK_LOCAL_IP=<IP_Address>

Starting the cluster

Once you complete the configurations mentioned above, start the two IS Analytics nodes. If the cluster is successfully configured, the following CLI logs are generated.

• The following is displayed in the CLIs of both nodes, and it indicates that the registry mounting is successfully done.

  [2016-01-28 14:20:53,596]  INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} -  Configured Registry in 107ms
  [2016-01-28 14:20:53,631]  INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} -  Connected to mount at govregistry in 7ms
  [2016-01-28 14:20:53,818]  INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} -  Connected to mount at govregistry in 0ms

• A CLI log similar to the following is displayed for the first node you start to indicate that it has successfully started.

  [2016-01-28 14:32:40,283]  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} -  Using wka based membership management scheme
  [2016-01-28 14:32:40,284]  INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} -  Added member: Host:10.100.0.46, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
  [2016-01-28 14:32:40,284]  INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} -  Added member: Host:10.100.0.46, Remote Host:null, Port: 4001, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true
  [2016-01-28 14:32:41,665]  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} -  Hazelcast initialized in 1379ms
  [2016-01-28 14:32:41,728]  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} -  Local member: [9c7619a9-8460-465d-8fd0-7eab1c464386] - Host:10.100.0.46, Remote Host:null, Port: 4000, HTTP:9763, HTTPS:9443, Domain: wso2.carbon.domain, Sub-domain:worker, Active:true
  [2016-01-28 14:32:41,759]  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} -  Elected this member [9c7619a9-8460-465d-8fd0-7eab1c464386] as the Coordinator node
  [2016-01-28 14:32:41,847]  INFO {org.wso2.carbon.event.processor.manager.core.internal.HAManager} -  CEP HA Snapshot Server started on 0.0.0.0:10005
  [2016-01-28 14:32:41,850]  INFO {org.wso2.carbon.event.processor.manager.core.internal.HAManager} -  Became CEP HA Active Member
  [2016-01-28 14:32:41,885]  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} -  Cluster initialization completed

• Once you start the second node, a CLI log similar to the following will be displayed for the first node to indicate that another node has joined the cluster.

  [2016-01-28 14:34:13,252]  INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} -  Member joined [504bceff-4a08-46fe-83e6-b9561d3fff81]: /10.100.0.46:4001
  [2016-01-28 14:34:15,963]  INFO {org.wso2.carbon.event.processor.manager.commons.transport.client.TCPEventPublisher} -  Connecting to 10.100.0.46:11224
  [2016-01-28 14:34:15,972]  INFO {org.wso2.carbon.event.processor.manager.core.internal.EventHandler} -  CEP sync publisher initiated to Member '10.100.0.46:11224'

• A CLI log similar to the following is displayed for the second node once it joins the cluster.

  [2016-01-28 14:34:27,086]  INFO {org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor} -  Spark Master map size after starting masters : 2

Testing the HA deployment

The HA deployment you configured can be tested as follows.

1. Access the Spark UIs of the active controller and the stand-by controller using < node ip>:8081 in each node.
   • Information relating to the active controller is displayed as shown in the example below.
     
   • Information relating to the stand-by controller is displayed as shown in the example below.
     
2. Click the links under Running Applications in the Spark UI of the active controller to check the Spark application UIs of those applications. A working application is displayed as shown in the following example.
   
3. Click the Environment tab of a Spark application UI to check whether all the configuration parameters are correctly set. You can also check whether the class path variables in this tab can be accessed manually.
   
4. Check the Spark UIs of workers to check whether they have running executors. If a worker UI does not have running executors or if it is continuously creating executors, it indicates an issue in the Spark cluster configuration. The following example shows a worker UI with a running executor.
   
5. Check the symbolic parameter, and check if you could manually access it via a cd <directory> command in the CLI.
6. Log into the IS Analytics Management Console and navigate to Main => Manage => Batch Analytics => Console to open the Interactive Analytics Console. Run a query in this console.