Purging Analytics Data

Data purging is an option to remove historical data in WSO2 Identity Server Analytics. This is important since it is not possible to delete tables or table data in WSO2 Identity Server Analytics. By purging data, you can achieve high performance in IS data analysis without removing analyzed summary data. Only data from the stream data generated by identity server is purged. This data is contained in the following tables:

• ORG_WSO2_IS_ANALYTICS_STREAM_OVERALLSESSION
• ORG_WSO2_IS_ANALYTICS_STREAM_PROCESSEDOVERALLAUTHENTICATION
• ORG_WSO2_IS_ANALYTICS_STREAM_SESSIONINFO


There are two ways to purge data in IS Analytics:

Using the Management Console

To schedule data purging via the Management Console, follow the steps below:

1. Log in to the WSO2 IS Analytics Management Console.
2. In the Main tab, click Data Explorer (Manage > Interactive Analysis) to open the Data Explorer page.
   
3. Select the required table in the Table Name field, and click Schedule Data Purging as shown below. 
   
   This opens the Schedule Data Purging dialog box.
4. In the Schedule Data Purging dialog box, set the time and days within which you want to purge data as shown below. Then click Save.
   
5. Repeat these steps for all 3 tables mentioned above to schedule all of them to be cleared regularly.

Using the global method

1. Open the <IS_HOME>/repository/conf/analytics/analytics-config.xml file.

2. Change the contents under the <analytics-data-purging> property as shown below:

   <analytics-data-purging>
         <purging-enable>true</purging-enable>
         <cron-expression>0 0 12 * * ?</cron-expression>
         <purge-include-tables>
            <table>.*</table>
         </purge-include-tables>
         <data-retention-days>365</data-retention-days>
   </analytics-data-purging>

3. Save your changes.

Verify purging

To verify if the automatic purging task runs successfully, do the following.

• Check the server log at the relevant time at which the task is scheduled to run. You should see a log similar to the one shown below.

  TID: [-1234] [] [2019-10-31 00:00:03,245]  INFO {org.wso2.carbon.analytics.dataservice.core.tasks.AnalyticsGlobalDataPurgingTask} -  Records persist before Sun Oct 27 23:59:59 MDT 2019[1572242399999] going to purge from ORG_WSO2_APIMGT_STATISTICS_PERHOURRESPONSE in tenant[-1234] {org.wso2.carbon.analytics.dataservice.core.tasks.AnalyticsGlobalDataPurgingTask}

• Check the difference in the database table size to check if it has reduced once the purging task has run. You can run a database query similar to the MYSQL query shown below to check the size of the event store table.

  SELECT table_name AS "Table", round(((data_length + index_length) / 1024 / 1024), 2) as size FROM information_schema.TABLES WHERE table_schema = "<analytics_eventsore_database_name>" ORDER BY size DESC;

  For instructions on how to retrieve the relevant table names related to purging, see Retrieving the Actual Name of a Table in the WSO2 Data Analytics Server documentation.