Versions Compared

Old Version 7

changes.mady.by.user Former user

Saved on

compared with

New Version 8

changes.mady.by.user Former user

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Sample Error Log
Code Block
TID: [-1] [] [2018-02-13 04:49:40,345] ERROR {org.wso2.carbon.analytics.dataservice.core.indexing.IndexNodeCoordinator} -  Error in processing staging index data: Error in processing index staging entry: [-1234:NEW_CUSTOMERS_BY_QUARTER_PRODUCT_AND_POD] does not exist {org.wso2.carbon.analytics.dataservice.core.indexing.IndexNodeCoordinator}
org.wso2.carbon.analytics.datasource.commons.exception.AnalyticsException: Error in processing index staging entry: [-1234:NEW_CUSTOMERS_BY_QUARTER_PRODUCT_AND_POD] does not exist
	at org.wso2.carbon.analytics.dataservice.core.indexing.IndexNodeCoordinator.processStagingEntry(IndexNodeCoordinator.java:961)
	at org.wso2.carbon.analytics.dataservice.core.indexing.IndexNodeCoordinator.access$600(IndexNodeCoordinator.java:69)
	at org.wso2.carbon.analytics.dataservice.core.indexing.IndexNodeCoordinator$StagingDataIndexWorker.run(IndexNodeCoordinator.java:997)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
OccurenceWhen the INSERT OVERWRITE operation is carried out.
Possible Reasons

The corruption of meta information and indexing that could have resulted from one of the following:

  • Deployments or the removal of deployments in the nodes of the DAS setup.
  • Configuration changes in the DAS nodes of the setup/
  • Creating/deleting tables in the event store or the process store.
Recommended Action

To resolve this issue, reindex your data following the steps below:

  1. Shut down all the DAS nodes in your deployment.

  2. Truncate the following tables.

    Info

    Table names are encoded in characters that cannot be read by humans for all database types other than HBase and Cassandra. Therefore, if the database type is not HBase or Cassandra, you need to use the encoded versions of these table names depending on the database type when performing this step.


    • ANX_X1000__INDEX_STAGING_DATA_*
    • IDX_X1000__INDEX_STAGING_DATA_*
     

  3. Drop all the tables with names that start with ANX and IDX such as the examples given below.

    Info

    Table names are encoded in characters that cannot be read by humans for all database types other than HBase and Cassandra. Therefore, this step is relevant only if the database type used is HBase or Cassandra because the table names given below can only exist for ese two database types


    • ANX_X1000__INDEX_STAGING_DATA_*
    • IDX_X1000__INDEX_STAGING_DATA_*
  4. For each node, remove all the index data stored in the <DAS_HOME>/repository/data directory.
  5. In the <DAS_HOME>/repository/conf/analytics/local-shard-allocation-config.conf file, change the mode for all the shards from NORMAL to INIT. This must be done for each node.
  6. Restart the DAS nodes.

Out Of Memory

Sample Production Head Heap Dump Analysis Dominator Tree

OccurenceThis occurs when WSO2 DAS receives a large bulk of data to be indexed.
Possible Reasons

The indexing operation of the bulk of data handled maybe using almost all the available memory based on the taxanomy cache size specified in the <DAS_HOME>/repository/conf/analytics/analytics-config.xml file.

e.g., If the available memory is 3.6 GB and 2.9 GB of that is used by the indexing operation as shown in the sample given above.

Recommended Action

To overcome this out of memory issue, increase the taxonomy cache size by following the procedure given below.

  1. Install the latest WUM update.

  2. Add the following configuration under analytics-dataservice-configuration in the <DAS_HOME>/repository/conf/analytics/analytics-config.xml file (if it does not already exist).

    Info

    When ths configuration is not added in the analytics-config.xml file, the default values shown below apply.

    Code Block
    languagexml
    <taxonomy-writer-cache>
     <cacheType>DEFAULT</cacheType>
     <!--<LRUType>STRING</LRUType>-->
     <!--<cacheSize>4096</cacheSize>-->
</taxonomy-writer-cache>

  3. To save the large taxanomy indexes in the file system of DAS instead of in the memory, change the value of the cacheType parameter to LRU as shown below.
    <cacheType>LRU</cacheType>

    Info

    When the value specified for the cacheType parameter is DEFAULT, taxanomy indexes are saved in the DAS memory.

  4. Uncomment the LRUType parameter.
  5. Uncomment the cacheSize parameter. Then increase the value for this parameter as required.
  6. Restart the server.

...