Purging Analytics Data

ESB Analytics stores data and performs various analysis operations on them according to defined analytic queries. Thereby, as the volume of the data stored grows over time, the analysis and summarization jobs also consumes more time. Hence, purging data allows you to increase the performance of ESB Analytics by reducing the time taken to execute the analytics scripts as well as optimizing the disk usage.

• ORG_WSO2_ESB_ANALYTICS_STREAM_EVENT

• ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERSECOND

• ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERSECONDALL

• ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMINUTE

• ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMINUTEALL

• ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERHOUR

• ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMONTH

• ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERDAY

• ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERSECOND

• ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERMINUTE

• ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERHOUR

• ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERDAY

• ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERMONTH

There are three ways to purge data in ESB Analytics:

Using the Carbon App

For EI versions 6.1.0-6.3.0, purging analytics data can be managed via the org.wso2.analytics.ei.realtime Carbon application. This application is deployed by default if you get the WUM update timestamp: 1565079455315.

For more information about WUM-updating WSO2 products, see Getting Continuous Updates.

Using the per-table method

 Follow the steps below to schedule data purging via the per-table method.

1. Log in to the Management Console of the Analytics profile.
2. In the Main tab, click Data Explorer to open the Data Explorer page.

3. Select the required table in the Table Name field, and click Schedule Data Purging as shown below. 

   The Schedule Data Purging option is displayed only for users of whom the assigned role has the Delete permission under Record enabled.

   
   This opens the Schedule Data Purging dialog box.

4. In the Schedule Data Purging dialog box, set the time and days within which you want to purge data as shown below, and click Save.

   Do not purge data that is less than 2 days old as it can result in a data loss.

   

   The fields of the above screen are described below.

   Field	Description
   Enable Data Purging	Whether you want to enable data purging or not.
   Schedule Time (Either cron string or HH:MM)	Enter the time on which you want to purge data via cron expression or by defining the time in the following format: HH:MM. For example, the following cron expression will configure the data purging job to run at  12:00 PM (noon) every day :  0 0 12 * * ?  For more information on cron expressions, go to Oracle Documentation.
   Purge Record Older Than (Days)	Define the value as to keep data of only the last 'n' no of days back in the selected table. For example, if you give 1 as the value, the system will purge all data stored before yesterday.

5. Click Save and close the dialog box.

6. Repeat these steps for all the required tables to schedule them to be cleared regularly.
   

   Change the value of the <data-retention-days> property as per your requirement. However, the following values are recommended to retain and purge data in an optimum way.

   • You do not need to enable data purging for table defined for months and years.

   • In the <data-retention-days> property of tables for days, you must specify 45 days in order to retain data analyzed based on days for an acceptable time period.

   • For other tables created for shorter time granularities (e.g., seconds, minutes, and hours), you can specify a retention period of two days.

Removing scheduled data purging operations

Follow the steps below to remove a data purging operation that you already scheduled.

1. Log in to the Management Console as an admin user, if you are not already logged in.
2. Click Main, and then click Data Explorer.
3. Select the required table in the Table Name field, and then click Schedule Data Purging to open the Schedule Data Purging dialog box.
4. Clear the Enable Data Purging check box as shown below.
   
5. Click Save, and close the dialog box.

Using the global method

1. Open the <EI_HOME>/wso2/analytics/conf/analytics/analytics-config.xml file.

2. Change the content under the <analytics-data-purging> property as shown below:

   Change the value of the <data-retention-days> property as per your requirement. However, do not purge data that is less than 2 days old as it can result in a data loss.

   <analytics-data-purging>
       <purging-enable>true</purging-enable>
       <cron-expression>0 0 12 * * ?</cron-expression>
       <purge-include-table-patterns>
         <table>
            <table>ORG_WSO2_ESB_ANALYTICS_STREAM_EVENT</table>
       	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERSECOND</table>
       	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERSECONDALL</table>
       	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMINUTE</table>
       	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMINUTEALL</table>
       	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERHOUR</table>
       	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMONTH</table>
       	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERDAY</table>
       	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERSECOND</table>
       	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERMINUTE</table>
       	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERHOUR</table>
       	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERDAY</table>
       	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERMONTH</table>
         </table>
         </purge-include-table-patterns>
       <data-retention-days>2</data-retention-days>
     </analytics-data-purging>

   The properties of the above configuration file are shown below.

   Property	Description
   <purging-enable>	Change the value to true if you want to enable data purging.
   <cron-expression>	The cron expression to define how you want to schedule the data purging operation. For example, the following cron expression will configure the archive job to run at  12:00 PM (noon) every day :  0 0 12 * * ?  For more information on cron expressions, go to Oracle Documentation.
   <purge-include-table-patterns>	Specify the tables of which you want to purge data. By default, it is configured to perform data purging on all tables as follows: <table>.*</table> However, you can specify the required tables by defining a regular expression or a table name within the <table> property. Define one tag per each regular expression if you want to specify multiple tables.
   <data-retention-days>	Define the value as to keep data of only the last 'n' no of days back in the selected table.   For example, the default value 365 will purge all data stored before 1 year. You can purge all records by setting a minus value (e.g. -1) for <data-retention-days>.

3. Save your changes.