The following diagram indicates the minimum deployment pattern used for high availability.
EI Analytics supports a deployment scenario that has focus on high availability (HA) along with HA processing. To enable HA processing, you should have two EI Analytics servers in a cluster.
For this deployment, both the EI Analytics nodes should be configured to receive all events. To achieve this, clients can either send all the requests to both the nodes or each request to any one of the two nodes (i.e., using load balancing or failover mechanisms). If clients send all the requests to both nodes, the user has to specify that events are duplicated in the cluster (i.e., the same event comes to all the members of the cluster). Alternatively, if a client sends a request to one node, internally it sends that particular request to the other node as well. This way, even if the clients send requests to only one node, both EI Analytics nodes receive all the requests.
In this scenario, one EI Analytics node works in active mode and the other works in passive mode. However, both nodes process all the data.
If the active node fails, the other node becomes active and receives all the requests.
When the failed node is up again, it fetches all the internal states of the current active node via synching.
The newly arrived node then becomes the passive node and starts processing all the incoming messages to keep its state synched with the active node so that it can become active if the current active node fails.
Warning: Some of the requests may be lost during the time the passive node switches to the active mode.
Prerequisites
Before you configure a minimum high availability EI Analytics cluster, the following needs to be carried out.
- Download and install WSO2 EI from here.
- Follow the steps below to set up MySQL.
Download and install MySQL Server.
Download the MySQL JDBC driver.
Unzip the downloaded MySQL driver zipped archive, and copy the MySQL JDBC driver JAR (
mysql-connector-java-x.x.xx-bin.jar
) into the<EI_HOME>/lib
directory of all the nodes in the cluster.- Enter the following command in a terminal/command window, where
username
is the username you want to use to access the databases.
mysql -u username -p
- When prompted, specify the password that will be used to access the databases with the username you specified.
Create two databases named
userdb
andregdb.
About using MySQL in different operating systems
For users of Microsoft Windows, when creating the database in MySQL, it is important to specify the character set as latin1. Failure to do this may result in an error (error code: 1709) when starting your cluster. This error occurs in certain versions of MySQL (5.6.x) and is related to the UTF-8 encoding. MySQL originally used the latin1 character set by default, which stored characters in a 2-byte sequence. However, in recent versions, MySQL defaults to UTF-8 to be friendlier to international users. Hence, you must use latin1 as the character set as indicated below in the database creation commands to avoid this problem. Note that this may result in issues with non-latin characters (like Hebrew, Japanese, etc.). The following is how your database creation command should look.
mysql> create database <DATABASE_NAME> character set latin1;
For users of other operating systems, the standard database creation commands will suffice. For these operating systems, the following is how your database creation command should look.
mysql> create database <DATABASE_NAME>;
Execute the following script for the two databases you created in the previous step.
mysql> source <EI_HOME>/wso2/analytics/dbscripts/mysql.sql;
- Create the following databases in MySQL.
WSO2_ANALYTICS_EVENT_STORE_DB
WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB
Required configurations
When configuring the minimum high availability cluster following setups should be done for both nodes.
- Do the following database-related configurations.
Follow the steps below to configure the <EI_HOME>/wso2/analytics
/conf/datasources/master-datasources.xml
file as requiredEnable the all the nodes to access the users database by configuring a datasource to be used by user manager as shown below.
<datasource> <name>WSO2UM_DB</name> <description>The datasource used by user manager</description> <jndiConfig> <name>jdbc/WSO2UM_DB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://[MySQL DB url]:[port]/userdb</url> <username>[user]</username> <password>[password]</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource>
Enable the nodes to access the registry database by configuring the
WSO2REG_DB
data source as follows.<datasource> <name>WSO2REG_DB</name> <description>The datasource used by the registry</description> <jndiConfig> <name>jdbc/WSO2REG_DB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://[MySQL DB url]:[port]/regdb</url> <username>[user]</username> <password>[password]</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource>
For detailed information about registry sharing strategies, see the library article Sharing Registry Space across Multiple Product Instances.
Point to
WSO2_ANALYTICS_EVENT_STORE_DB
andWSO2_ANALYTICS_PROCESSED_DATA_STORE_DB
in the<EI_HOME>/wso2/analytics/conf/datasources/analytics-datasources.xml
file as shown below.<datasources-configuration> <providers> <provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider> </providers> <datasources> <datasource> <name>WSO2_ANALYTICS_EVENT_STORE_DB</name> <description>The datasource used for analytics record store</description> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://[MySQL DB url]:[port]/WSO2_ANALYTICS_EVENT_STORE_DB</url> <username>[username]</username> <password>[password]</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> <defaultAutoCommit>false</defaultAutoCommit> </configuration> </definition> </datasource> <datasource> <name>WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB</name> <description>The datasource used for analytics record store</description> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://[MySQL DB url]:[port]/WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB</url> <username>[username]</username> <password>[password]</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> <defaultAutoCommit>false</defaultAutoCommit> </configuration> </definition> </datasource> </datasources> </datasources-configuration>
For more information, see Datasources in DAS documentation.
Datasources are configured for WSO2 EI Analytics in the way they are configured for WSO2 Data Analytics Server.
To share the user store among the nodes, open the
<EI_HOME>/wso2/analytics/conf/user-mgt.xml
file and modify thedataSource
property of the<configuration>
element as follows.<configuration> ... <Property name="dataSource">jdbc/WSO2UM_DB</Property> </configuration>
The datasource name specified in this configuration should be the same as the datasource used by user manager that you configured in sub step a, i.
In the
<EI_HOME>/wso2/analytics/conf/registry.xml
file, add or modify thedataSource
attribute of the<dbConfig name="govregistry">
element as follows.<dbConfig name="govregistry"> <dataSource>jdbc/WSO2REG_DB</dataSource> </dbConfig> <remoteInstance url="https://localhost:9443/registry"> <id>gov</id> <cacheId>user@jdbc:mysql://localhost:3306/regdb</cacheId> <dbConfig>govregistry</dbConfig> <readOnly>false</readOnly> <enableCache>true</enableCache> <registryRoot>/</registryRoot> </remoteInstance> <mount path="/_system/governance" overwrite="true"> <instanceId>gov</instanceId> <targetPath>/_system/governance</targetPath> </mount> <mount path="/_system/config" overwrite="true"> <instanceId>gov</instanceId> <targetPath>/_system/analyticsconfig</targetPath> </mount>
Do not replace the following configuration when adding in the mounting configurations. The registry mounting configurations mentioned in the above steps should be added in addition to the following.
<dbConfig name="wso2registry"> <dataSource>jdbc/WSO2CarbonDB</dataSource> </dbConfig>
Change the
hostname
to the actual hostname in the<EI_HOME>/wso2/analytics/repository/deployment/server/jaggeryapps/portal/configs/designer.json
file as follows.This step avoids the host verification errors that can occur when multiple WSO2 product instances publish statistics.
}, "host": { "hostname": "localhost", "port": "", "protocol": "" }
- Update the
<EI_HOME>/wso2/analytics/conf/axis2/axis2.xml
file as follows to enable Hazlecast clustering for both nodes.
Set
clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
to true as shown below to enable Hazlecast clustering.<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
Enable the wka mode for both nodes as shown below. See About Membership Schemes for more information on wka mode.
<parameter name="membershipScheme">wka</parameter>
Add both the EI Analytics nodes as well known members in the cluster under the
members
tag in each node as shown in the example below.<members> <member> <hostName>[node1 IP]</hostName> <port>[node1 port]</port> </member> <member> <hostName>[node2 IP]</hostName> <port>[node2 port]</port> </member> </members>
For each node, enter the respective server IP address as the value forthe
localMemberHost
property as shown below.<parameter name="localMemberHost">[Server_IP_Address]</parameter>
Configure the
<EI_HOME>/wso2/analytics/conf/event-processor.xml
file as follows to cluster ESB Analytics in the Receiver.Enable the
HA
mode by setting the following property.<mode name="HA" enable="true">
Disable the
Distributed
mode by setting the following property.<mode name="Distributed" enable="false">
For each node, enter the respective server IP address under the
HA mode
Config section as shown in the example below.When you enable the HA mode for WSO2 EI Analytics, state persistence is enabled by default. If there is no real time use case that requires any state information after starting the cluster, you should disable event persistence by setting the
persistence
attribute tofalse
in the<EI_HOME>/wso2/analytics/conf/event-processor.xml
file as shown below.<persistence enable="false"> <persistenceIntervalInMinutes>15</persistenceIntervalInMinutes> <persisterSchedulerPoolSize>10</persisterSchedulerPoolSize> <persister class="org.wso2.carbon.event.processor.core.internal.persistence.FileSystemPersistenceStore"> <property key="persistenceLocation">cep_persistence</property> </persister> </persistence>
<!-- HA Mode Config --> <mode name="HA" enable="true"> ... <eventSync> <hostName>[Server_IP_Address]</hostName>
When you enable persistance for the WSO2 EI Analytics profile, the internal state of EI Analytics is persisted in files. These files are not automatically deleted. Therefore, if you want to save space in your EI Analytics pack, you need to delete them manually.
These files are created in the
<EI_HOME>/wso2/analytics/cep_persistence/<tenant-id>
directory. This directory has a separate sub-directory for each execution plan. Each execution plan can have multiple files. The format of each file name is<TIMESTAMP>_<EXECUTION_PLAN_NAME>
(e.g,1493101044948_MyExecutionPlan
). If you want to clear files for a specific execution plan, you need to leave the two files with the latest timestamps and delete the rest.
The following node types are configured for the HA deployment mode in the
<EI-Analytics_HOME>/repository/conf/event-processor.xml
file.eventSync
: Both the active and the passive nodes in this setup are event synchronizing nodes as explained in the introduction. Therefore, each node should have the host and the port on which it is operating specified under the<eventSync>
element.management
: In this setup, both the nodes carry out the same tasks, and therefore, both nodes are considered manager nodes. Therefore, each node should have the host and the port on which it is operating specified under the<management>
element.presentation
: You can optionally specify only one of the two nodes in this setup as the presenter node. The dashboards in which processed information is displayed are configured only in the presenter node. Each node should have the host and the port on which the assigned presenter node is operating specified under the<presentation>
element. The host and the port as well as the other configurations under the<presentation>
element are effective only when thepresenter enable="false
property is set under the<!-- HA Mode Config -->
section.
Update the
<EI_HOME>/wso2/analytics/conf/analytics/spark/spark-defaults.conf
file as follows to use the Spark cluster embedded within EI Analytics.- Keep the
carbon.spark.master
configuration aslocal
. This instructs Spark to create a Spark cluster using the Hazelcast cluster. Enter
2
as the value for thecarbon.spark.master.count
configuration. This specifies that there should be two masters in the Spark cluster. One master serves as an active master and the other serves as a stand-by master.
The following example shows the
<EI_HOME>/wso2/analytics/conf/analytics/spark/spark-defaults.conf
file with changes mentioned above.carbon.spark.master local carbon.spark.master.count 2
For more information, see Spark Configurations in DAS documentation.
Important: If the path to
<EI_HOME>/wso2/analytics
is different in the two nodes, do the following.- Keep the
In order to share the C-Apps deployed among the nodes, configure the SVN-based deployment synchronizer. For detailed instructions, see WSO2 Clustering Guide - Configuring SVN-Based Deployment Synchronizer.
EI Analytics Minimum High availability Deployment set up does not use a manager and a worker. For the purpose of configuring the deployment synchronizer, you can add the configurations relevant to the manager for the node of your choice, and add the configurations relating to the worker for the other node.
If you do not configure the deployment synchronizer, you are required to deploy any C-App you use in the EI Analytics Minimum High Availability Deployment set up to both the nodes.
Starting the cluster
Once you complete the configurations mentioned above, start the two EI Analytics nodes. If the cluster is successfully configured, the following CLI logs are generated.
The following is displayed in the CLIs of both nodes, and it indicates that the registry mounting is successfully done.
[2016-01-28 14:20:53,596] INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} - Configured Registry in 107ms [2016-01-28 14:20:53,631] INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} - Connected to mount at govregistry in 7ms [2016-01-28 14:20:53,818] INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} - Connected to mount at govregistry in 0ms
A CLI log similar to the following is displayed for the first node you start to indicate that it has successfully started.
[2016-01-28 14:32:40,283] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Using wka based membership management scheme [2016-01-28 14:32:40,284] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:10.100.0.46, Remote Host:null, Port: 4000, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true [2016-01-28 14:32:40,284] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:10.100.0.46, Remote Host:null, Port: 4001, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true [2016-01-28 14:32:41,665] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Hazelcast initialized in 1379ms [2016-01-28 14:32:41,728] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Local member: [9c7619a9-8460-465d-8fd0-7eab1c464386] - Host:10.100.0.46, Remote Host:null, Port: 4000, HTTP:9763, HTTPS:9443, Domain: wso2.carbon.domain, Sub-domain:worker, Active:true [2016-01-28 14:32:41,759] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Elected this member [9c7619a9-8460-465d-8fd0-7eab1c464386] as the Coordinator node [2016-01-28 14:32:41,847] INFO {org.wso2.carbon.event.processor.manager.core.internal.HAManager} - EI-ANALYTICS HA Snapshot Server started on 0.0.0.0:10005 [2016-01-28 14:32:41,850] INFO {org.wso2.carbon.event.processor.manager.core.internal.HAManager} - Became EI-ANALYTICS HA Active Member [2016-01-28 14:32:41,885] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Cluster initialization completed
Once you start the second node, a CLI log similar to the following will be displayed for the first node to indicate that another node has joined the cluster.
[2016-01-28 14:34:13,252] INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - Member joined [504bceff-4a08-46fe-83e6-b9561d3fff81]: /10.100.0.46:4001 [2016-01-28 14:34:15,963] INFO {org.wso2.carbon.event.processor.manager.commons.transport.client.TCPEventPublisher} - Connecting to 10.100.0.46:11224 [2016-01-28 14:34:15,972] INFO {org.wso2.carbon.event.processor.manager.core.internal.EventHandler} - EI-ANALYTICS sync publisher initiated to Member '10.100.0.46:11224'
A CLI log similar to the following is displayed for the second node once it joins the cluster.
[2016-01-28 14:34:27,086] INFO {org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor} - Spark Master map size after starting masters : 2
Following are some exceptions you may view in the start up log when you start the cluster.
When you start the passive node of the HA cluster, the following errors are displayed.
This is because the artifacts are yet to be deployed in the passive node even though it has received the sync message from the active node. This error is no longer displayed once the start up for the passive node is complete.
When the Apache Spark Cluster is not properly instantiated, the following errors are displayed.
All the nodes in the Spark cluster should be started in order to stop this exception from occurring.