Page Comparison

Data purging is an option to remove historical data in the WSO2 API Management Manager Analytics profile. This is important as because it is not possible to delete tables or table data in the WSO2 API Manager Analytics profile. By purging data, you can achieve high performance in data analysis without removing analyzed summary data. This data is contained in the Anchorstat-tablesstat-tablesfollowing tables:

• ORG_WSO2_APIMGT_STATISTICS_WORKFLOW
• ORG_WSO2_APIMGT_STATISTICS_PERMINUTEREQUEST
• ORG_WSO2_APIMGT_STATISTICS_PERHOURREQUEST
• ORG_WSO2_APIMGT_STATISTICS_PERDAYREQUEST
• ORG_WSO2_APIMGT_STATISTICS_PERMINUTERESPONSE
• ORG_WSO2_APIMGT_STATISTICS_PERHOURRESPONSE
• ORG_WSO2_APIMGT_STATISTICS_PERDAYRESPONSE
• ORG_WSO2_APIMGT_STATISTICS_PERMINUTEEXECUTIONTIMES
• ORG_WSO2_APIMGT_STATISTICS_PERHOUREXECUTIONTIMES
• ORG_WSO2_APIMGT_STATISTICS_PERDAYEXECUTIONTIMES
• ORG_WSO2_APIMGT_STATISTICS_THROTTLE
• ORG_WSO2_APIMGT_STATISTICS_FAULT
• ORG_WSO2_CARBON_IDENTITY_OAUTH_TOKEN_ISSUANCE
• LOGANALYZER

You can perform data purging in the Analytics profile using either one of the following methods.

Purging Data of a Specific Table

Follow the instructions below to purge data of a selected table via the Management Console.

...

Select the required table in the Table Name field, and click Schedule Data Purging as shown below. 

...

In the Schedule Data Purging dialog box, set the time and days within which you want to purge data as shown below. Then click Save.

Image Removed

...

Removing scheduled data purging operations

Follow the instructions below to remove a data purging operation that you have already scheduled.

1. Sign in to the WSO2 API-M Analytics Management Console as an admin user if you have not already signed in.
   https://<APIM-ANALYTICS_HOST>:<APIM-ANALYTICS_PORT>/carbon/
2. Click Main, and then click Data Explorer.
3. Select the required table in the Table Name field, and then click Schedule Data Purging. The Schedule Data Purging dialog box appears.
4. Clear the Enable Data Purging check box as shown below.
   Image Removed
5. Click Save, and close the dialog box.

Purging data globally

...

Change the contents under the <analytics-data-purging> property as shown below:

<analytics-data-purging>
    <purging-enable>true</purging-enable>
    <cron-expression>0 0 12 * * ?</cron-expression>
    <purge-include-tables>
         <table>ORG_WSO2_APIMGT_STATISTICS_WORKFLOW</table>
    	 <table>ORG_WSO2_APIMGT_STATISTICS_PERMINUTEREQUEST</table>
    	 <table>ORG_WSO2_APIMGT_STATISTICS_PERHOURREQUEST</table>
    	 <table>ORG_WSO2_APIMGT_STATISTICS_PERDAYREQUEST</table>
    	 <table>ORG_WSO2_APIMGT_STATISTICS_PERMINUTERESPONSE</table>
    	 <table>ORG_WSO2_APIMGT_STATISTICS_PERHOURRESPONSE</table>
    	 <table>ORG_WSO2_APIMGT_STATISTICS_PERDAYRESPONSE</table>
    	 <table>ORG_WSO2_APIMGT_STATISTICS_PERMINUTEEXECUTIONTIMES</table>
    	 <table>ORG_WSO2_APIMGT_STATISTICS_PERHOUREXECUTIONTIMES</table>
    	 <table>ORG_WSO2_APIMGT_STATISTICS_PERDAYEXECUTIONTIMES</table>
    	 <table>ORG_WSO2_APIMGT_STATISTICS_THROTTLE</table>
    	 <table>ORG_WSO2_APIMGT_STATISTICS_FAULT</table>
    	 <table>LOGANALYZER</table>
    	 <table>ORG_WSO2_CARBON_IDENTITY_OAUTH_TOKEN_ISSUANCE</table>
      </purge-include-tables>
    <data-retention-days>2</data-retention-days>
  </analytics-data-purging>

The properties of the above configuration file are shown below.

...

<purging-enable>

...

<cron-expression>

...

<purge-include-tables>

...

<table>.*</table> However, you can specify the required tables by defining a regular expression or a table name within the <table> property. Define one tag for each regular expression if you want to specify multiple tables.

...

<data-retention-days>

...

Disabling data purging in a clustered mode

In a clustered mode you can disable the scheduled data purging tasks being operated in a particular node (e.g., a node which is used for database tasks) using a start up parameter. Start the API Analytics of that particular node by executing the following command to disable data purging:  

sh <API-M_ANALYTICS_HOME>/bin/wso2server.sh -DdisableDataPurging=true

Otherwise, you can permanently use the start up parameter as a system property by adding the following line of code to the <API-M_ANALYTICS_HOME>/bin/wso2server.sh file.

-DdisableDataPurging=true \

while [ "$status" = "$START_EXIT_STATUS" ]
do
    $JAVACMD \
    -Xbootclasspath/a:"$CARBON_XBOOTCLASSPATH" \
    $JVM_MEM_OPTS \
    -XX:+HeapDumpOnOutOfMemoryError \
    -XX:HeapDumpPath="$CARBON_HOME/repository/logs/heap-dump.hprof" \
    $JAVA_OPTS \
    -Dcom.sun.management.jmxremote \
    -classpath "$CARBON_CLASSPATH" \
    -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \
    -Djava.io.tmpdir="$CARBON_HOME/tmp" \
    -Dcatalina.base="$CARBON_HOME/lib/tomcat" \
    -Dwso2.server.standalone=true \
    -Dcarbon.registry.root=/ \
    -Djava.command="$JAVACMD" \
    -Dcarbon.home="$CARBON_HOME" \
    -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \
    -Dcarbon.config.dir.path="$CARBON_HOME/repository/conf" \
    -Djava.util.logging.config.file="$CARBON_HOME/repository/conf/etc/logging-bridge.properties" \
    -Dcomponents.repo="$CARBON_HOME/repository/components/plugins" \
    -Dconf.location="$CARBON_HOME/repository/conf"\
    -Dcom.atomikos.icatch.file="$CARBON_HOME/lib/transactions.properties" \
    -Dcom.atomikos.icatch.hide_init_file_path=true \
    -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false \
    -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true \
    -Dcom.sun.jndi.ldap.connect.pool.authentication=simple  \
    -Dcom.sun.jndi.ldap.connect.pool.timeout=3000  \
    -Dorg.terracotta.quartz.skipUpdateCheck=true \
    -Djava.security.egd=file:/dev/./urandom \
    -Dfile.encoding=UTF8 \
    -Djava.net.preferIPv4Stack=true \
    -Dcom.ibm.cacheLocalHost=true \
    -DworkerNode=false \
	-DdisableDataPurging=true \
    -Dorg.apache.cxf.io.CachedOutputStream.Threshold=104857600 \
    $NODE_PARAMS \
    org.wso2.carbon.bootstrap.Bootstrap $*
    status=$?

Purging alert related data

Details of all the alerts that are generated by WSO2 API Manager Analytics can be viewed via the WSO2 API-M Analytics Dashboard because all information related alerts are persisted in the WSO2 API-M Analytics related database. However, if you do not wish to maintain the WSO2 API-M Analytics alert related data that is older than a given time period, you can purge the following tables based on a data retention period of your choice that is no less than 2 days; however, note that this will delete summarized historical data related to WSO2 API-M Analytics alerts and you will not be able to view the deleted data via the WSO2 API-M Analytics Dashboard. The latter mentioned alert related data is contained in the following tables:

...

Purging alert related tables

Data Purging is enabled by default for all the aggregation tables. However, you can purge data in certain alert tables, namely APIMIpAccessSummary and APIMIpAccessAlertCount tables, which have the potential to grow quickly like aggregation tables. Follow the instructions below to purge the data in the latter mentioned alert tables.

1. Configure WSO2 API Manager (WSO2 API-M) with WSO2 API-M Analytics, and start WSO2 API-M Analytics, WSO2 API Manager, and the WSO2 API-M Analytics Dashboard node in the latter mentioned order. For more information, see Integrating API Manager Analytics.
2. Access the Business Rules via the WSO2 API-M Analytics dashboard.
   https://<HOSTNAME>:<DASHBOARD_PORT>/business-rules
   For example:  https://localhost:9643/business-rules
3. Click the edit link that corresponds to the APIM DATA PURGING business rule.
   Image Added
   

4. Click Save and Deploy to enable data purging in the APIMIpAccessSummary and APIMIpAccessAlertCount tables.
   When the above business rule is enabled, it will purge data from APIMIpAccessSummary and APIMIpAccessAlertCount tables every time the trigger is triggered. The default time interval for the trigger is 1 years.
   Image Added

   Info
   If you need to purge the data immediately or cannot wait for the next trigger, then you need to reduce the default time interval to trigger data purging by changing the Time Interval For Trigger value to a smaller value (e.g., one minute). You can set the time interval for the triggerin seconds, minutes, hours, days, months, and years.