Archiving Operation Data to Improve Performance

WSO2 IoT Server stores the operation details of each device for audit purposes. When large amounts of operation data get accumulated, it affects the performance of the server as all these details are stored in the database. Therefore, WSO2 IoT Server introduces a method to archive the operation details of the devices. It allows you to archive the operation data and completely remove the data from the archival table after a specific time period. This feature is disabled by default.

Note!

To comply with the General Data Protection Regulations (GDPR), you need to make sure that you delete user specific details if a user requests the right to be forgotten. Therefore, if you have enabled operation data archival, make sure to delete the required data from the archival table too.

Let's take a look at the steps you need to follow to enabled the archiving operations.

Configure the archival datasource. By default, the archival data is stored in the jdbc/DM_ARCHIVAL_DS datasource.
Therefore, make sure to update the datasource in the <IOTS_HOME>/conf/datasources/cdm-datasources.xml file with the database details. The default configurations connect to the H2 database, which should not be used in production.

The default configurations:

<datasource>
   <name>DM_ARCHIVAL_DS</name>
   <description>The archival datasource used for CDM</description>
   <jndiConfig>
      <name>jdbc/DM_ARCHIVAL_DS</name>
   </jndiConfig>
   <definition type="RDBMS">
      <configuration>
         <url>jdbc:h2:repository/database/WSO2DM_ARCHIVAL_DS;DB_CLOSE_ON_EXIT=FALSE</url>
         <username>wso2carbon</username>
         <password>wso2carbon</password>
         <driverClassName>org.h2.Driver</driverClassName>
         <maxActive>50</maxActive>
         <maxWait>60000</maxWait>
         <testOnBorrow>true</testOnBorrow>
         <validationQuery>SELECT 1</validationQuery>
         <validationInterval>30000</validationInterval>
      </configuration>
   </definition>
</datasource>

Open the <IOTS_HOME>/conf/cdm-config.xml file and configure the following properties:

Adding data to the archival table. (These configurations are within the `<ArchivalTask>` property)
`Enabled`	By default, the data archival task is disabled. Assign `true` as the value to enable the archival process.
`TaskClass`	The class that is used to archive the operation data.
`CronExpression`	Defines how often you want the task the run using a cron expression. By default, the data archival task is set to run every day at 12 AM.
`RetentionPeriod`	Define the time period after which you want the data to be sent to the archival table. By default, the data is sent to the archival table after 30 days.
`ExecutionBatchSize`	If a large set of data sent to the archival table, it takes a long time to send the data. Therefore, define the maximum number of data that should be sent to the table at a given time. This ensures that the data are sent in batches.
Deleting the data that was added to the archival table. (These configurations are within the `<PurgingTask>` property that is within the `<ArchivalTask>` property )
`Enabled`	By default, the data deleting task is disabled. Assign `true` as the value to enable it.
`TaskClass`	The class that is used to delete the data that was archived.
`CronExpression`	Defines how often you want the task the run using a cron expression. By default, the data deleting task is set to run every day at 3 AM.
`RetentionPeriod`	Define the time period after which you want the data to be completely deleted from the archival table. By default, the data is sent to delete after 365 days.