Page Comparison

ESB Analytics stores data and performs various analysis operations on them according to defned defined analytic queries. Thereby, as the volume of the data stored grows over time, the analysis and summarization jobs will also consume more time. Hence, purging data allows you to increase the performance of ESB Analytics by reducing the time taken to execute the analytics scripts as well as optimising optimizing the disk usage.

Warning
Decide on a retention period based on the number of days you need to retain data for analysis and purge data that are older than that in the below tables.

ORG_WSO2_ESB_ANALYTICS_STREAM_EVENT
ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERSECOND
ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERSECONDALL
ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMINUTE
ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMINUTEALL
ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERHOUR
ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMONTH
ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERDAY
ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERSECOND
ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERMINUTE
ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERHOUR
ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERDAY
ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERMONTH

There are two three ways to purge data in ESB Analytics:

Table of Contents

maxLevel	3
minLevel	3

Using the Carbon App

For EI versions 6.1.0-6.3.0, purging analytics data can be managed via the org.wso2.analytics.ei.realtime Carbon application. This application is deployed by default if you get the WUM update timestamp: 1565079455315.

For more information about WUM-updating WSO2 products, see Getting Continuous Updates.

Using the per-table method

...

Open the <EI_HOME>/wso2/analytics/conf/analytics/analytics-config.xml file.

Change the content under the <analytics-data-purging> property as shown below:

Note
Change the value of the `<data-retention-days>` property as per your requirement. However, do not purge data that is less than 2 days old as it can result in a data loss.

Code Block

language	xml

<analytics-data-purging>
    <purging-enable>true</purging-enable>
    <cron-expression>0 0 12 * * ?</cron-expression>
    <purge-include-tables>
         <table>ORG_WSO2_ESB_ANALYTICS_STREAM_EVENT</table>
    	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERSECOND</table>
    	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERSECONDALL</table>
    	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMINUTE</table>
    	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMINUTEALL</table>
    	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERHOUR</table>
    	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMONTH</table>
    	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERDAY</table>
    	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERSECOND</table>
    	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERMINUTE</table>
    	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERHOUR</table>
    	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERDAY</table>
    	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERMONTH</table>
      </purge-include-tables>
    <data-retention-days>2</data-retention-days>
  </analytics-data-purging>

The properties of the above configuration file are shown below.

Property Description

<purging-enable>

Change the value to true if you want to enable data purging.

<cron-expression>

The cron expression to define how you want to schedule the data purging operation. For example, the following cron expression will configure the archive job to run at 12:00 PM (noon) every day : 0 0 12 * * ? For more information on cron expressions, go to Oracle Documentation.

<purge-include-table-patterns>

Specify the tables of which you want to purge data. By default, it is configured to perform data purging on all tables as follows:

<table>.*</table> However, you can specify the required tables by defining a regular expression or a table name within the <table> property. Define one tag per each regular expression if you want to specify multiple tables.

<data-retention-days>

Define the value as to keep data of only the last 'n' no of days back in the selected table. For example, the default value 365 will purge all data stored before 1 year.

Info
You can purge all records by setting a minus value (e.g. -1) for `<data-retention-days>`.

Save your changes.

Versions Compared

Old Version 3

New Version 4

Key

Using the Carbon App

Using the per-table method