Purging Data

WSO2 DAS stores data in the Data Access Layer (DAL) and performs various analysis operations on them according to defined analytic queries. Thereby, as the volume of the data stored grows over time, the analysis and summarization jobs will also eventually consume more time. Then you can apply data purging to reduce the time taken to execute the analytics scripts, as well as the disk usage, since usually it is not necessary to analyze all data to produce the final result. 

You can perform data purging in WSO2 DAS based on the following methods.

Per table data purging

Follow the steps below to purge data of a selected table via the Management Console.

1. Log in to the management console as an admin user using the following URL: https://<DAS_HOST>:<DAS_PORT>/carbon/ 
2. Click Main, and then click Data Explorer.

3. Select the required event table for the Table Name parameter, and click Schedule Data Purging as shown below to open the Schedule Data Purging dialog box.

   The Schedule Data Purging option is displayed only for users of whom the assigned role has the Delete permission under Record enabled. For instructions on setting this permission to users, see WSO2 DAS-Specific User Permissions .

   

4. Enter information as follows in the Schedule Data Purging dialog box to schedule the data purging task.
   
   The fields of the above screen are described below.

   Field	Description
   Enable Data Purging	Whether you want to enable data purging or not.
   Schedule Time (Either cron string or HH:MM)	Enter the time on which you want to purge data via cron expression or by defining the time in the following format: HH:MM. For example, the following cron expression will configure the data purging job to run at  12:00 PM (noon) every day :  0 0 12 * * ?  For more information on cron expressions, go to Oracle Documentation.
   Purge Record Older Than (Days)	Define the value as to keep data of only the last 'n' no of days back in the selected table. For example, if you give 1 as the value, the system will purge all data stored before yesterday.

5. Click Save, and close the dialog box.

Removing scheduled data purging operations

Follow the steps below to remove a data purging operation that you have already scheduled.

1. Log in to the Management Console as an admin user, if you are not already logged in.
2. Click Main, and then click Data Explorer.
3. Select the required table in the Table Name field, and then click Schedule Data Purging to open the Schedule Data Purging dialog box.
4. Clear the Enable Data Purging check box as shown below.
   
5. Click Save, and close the dialog box.

Global data purging

You can perform data purging as a global operation which will affect all tenants. Follow the steps below to perform global data purging.

1. Navigate to <DAS_HOME>/repository/conf/analytics/ analytics-config.xml file.

2. Change the configurations within the  <analytics-data-purging> property  as shown below.

   <analytics-data-purging>
   <!-- Below entry will indicate purging is enable or not. If user wants to enable data purging for cluster then this property need to be enable in all nodes -->
   <purging-enable>true</purging-enable>
   <cron-expression>0 0 12 * * ?</cron-expression>
   <!-- Tables that need include to purging. Use regex expression to specify the table name that need include to purging.-->
   <purge-include-tables>
   <table>.*</table>
   <!--<table>.*jmx.*</table>-->
   </purge-include-tables>
   <!-- All records that insert before the specified retention time will be eligible to purge -->
   <data-retention-days>365</data-retention-days>
   </analytics-data-purging>

   The properties of the above configuration file are shown below.

   Property	Description
   <purging-enable>	Change the value to true if you want to enable data purging.
   <cron-expression>	The cron expression to define how you want to schedule the data purging operation. For example, the following cron expression will configure the archive job to run at  12:00 PM (noon) every day :  0 0 12 * * ?  For more information on cron expressions, go to Oracle Documentation.
   <purge-include-tables>	Specify the tables of which you want to purge data. By default, it is configured to perform data purging on all tables as follows: <table>.*</table> However, you can specify the required tables by defining a regular expression or a table name within the <table> property. Define one tag per each regular expression if you want to specify multiple tables.
   <data-retention-days>	Define the value as to keep data of only the last 'n' no of days back in the selected table.   For example, the default value 365 will purge all data stored before 1 year.

Disabling data purging in a clustered mode

In a clustered mode, you can disable the scheduled data purging tasks being operated in a particular node (e.g. a node which is used for database tasks) using a startup parameter. Start the particular node by executing the following command in the CLI, to disable data purging in it:  sh <DAS_HOME>/bin/wso2server.sh -DdisableDataPurging=true

Else, you can permanently use the startup parameter as a system property by adding the following line to the  <DAS_HOME>/bin/wso2server.sh file.