General Data Protection Regulation for WSO2 IoT Server

WSO2 IoT Server consists of three profiles (IoT Server Core, Message Broker, and Analytics) that can persist a user's PII in various sources, namely log files and RDBMSs. Organizations that use WSO2 IoT Server has a legal obligation to remove all instances of a user's PII from the system if the relevant user requests the right to be forgotten. For example, consider a situation where an employee resigns from the organization and, thereby, requests the organization to remove all instances of one's PII from the organization's system. You can fulfill this requirement by anonymizing the user's PII in the system, or (in some cases) by completely removing such PII from the system.

See the topics given below for instructions on how to remove PII from each profile of WSO2 IoT Server.

What is GDPR?

The General Data Protection Regulation (GDPR) is a new legal framework that was formalized by the European Union (EU) in 2016. It comes into effect from 28, May 2018. GDPR requires any organization that processes Personally Identifiable Information (PII) of individuals who live in Europe to be compliant with the regulations. Organizations that fail to demonstrate GDPR compliance are subjected to financial penalties. Click here for more information.

Do you want to learn more about GDPR?

If you are new to GDPR, we recommend that you take a look at our tutorial series on Creating a Winning GDPR Strategy.

Part 1 - Introduction to GDPR
Part 2 - 7 Steps for GDPR Compliance
Part 3 - Identity and Access Management to the Rescue
Part 4 - GDPR Compliant Consent Design

For more resources on GDPR, see the white papers, case studies, solution briefs, webinars, and talks published on our WSO2 GDPR homepage. You can also find the original GDPR legal text here.

The PII in WSO2 IoT Server

The following information is considered PII in WSO2 IoT Server:

User details, such as username, date of birth, address, and mobile number.
Mobile device details, such as mobile device identifier, device locations, mobile number, and mobile application details.

How WSO2 IoT Server persists a user's PII

Each profile of WSO2 IoT Server persists user information in various different sources as explained below.

IoT Server Core	The IoT Server Core profile can persist PII in various log files, (carbon logs, audit logs, API logs, and device-specific logs), the database, and the device management console..
Message Broker	The Message Broker profile does not persist PII in any way and it has a short-term memory where the data gets deleted after a short time.
Analytics	The Analytics profile of WSO2 IoT Server uses event streams, which contain user information (PII) in its schemas. This data is stored in two separate RDBMS dedicatedfortheAnalytics profile.

Removing PII from the IoT Server Core profile

You can use the Identity Anonymization Tool to remove references to personally identifiable information (PII) from logs in the IoT Server Core profile.

Let's take a look at how to remove the data:

Step 1: Removing the devices and device details from WSO2 IoT Server

Before you begin

Click here to get the access token.

Encode the client credentials as follows:
Generate the Client ID and the Secret ID.
Encode the client credentials as follows:
Generate the access token using the following command:

The access token you generated expires in an hour. After it expires you can generate a new access token using the refresh token. For more information, see Generating a new access token from the refresh token.

Delete all the device related details of a specific user from the IoT Server databases:
curl -v -k -X DELETE -H 'Content-Type:application/json' -H 'authorization: Bearer <ACCESS TOKEN>' 'https://<IOTS_HOST>:8243/api/device-mgt/v1.0/admin/users/1.0.0/<USERNAME>/devices'
Enter the value of the access token you generated as the value for the<ACCESS_TOKEN>.
Enter the IP or hostname the server is running on as the value for the<IOTS_HOST>.
Enter the username of the user who requested for the details to be forgotten as the value for the<USERNAME>.
Example: Deleting all the devices and device details that are enrolled under Alex from the databases and the device management console.
curl -v -k -X DELETE -H 'Content-Type:application/json' -H 'authorization: Bearer 38093d75-b00e-3f8a-8594-346a11705e75' 'https://localhost:8243/api/device-mgt/v1.0/admin/users/1.0.0/alex/devices'
After running the command, you see that the devices owned by Alex removed from the device management console.
Delete details of a specific device from the WSO2 IoT Server database and the device management console.
curl -v -k -X DELETE -H 'Content-Type:application/json' -H 'authorization: Bearer <ACCESS TOKEN>' 'https://<IOTS_HOST>:8243/api/device-mgt/v1.0/admin/users/1.0.0/type/android/id/<DEVICE_ID>'
Enter the value of the access token you generated as the value for the<ACCESS_TOKEN>.
Enter the IP or hostname the server is running on as the value for the<IOTS_HOST>.
Enter the device type, such as android, ios, or windows, as the value for the<DEVICE_TYPE>.
Enter the device ID of the device as the value for <DEVICE_ID>.
Example: Deleting the device details that are related to the given device ID from the device management console and the database.
curl -v -k -X DELETE -H 'Content-Type:application/json' -H 'authorization: Bearer d302f5cf-ab11-3029-8543-f5d30d36ea30' 'https://localhost:8243/api/device-mgt/v1.0/admin/users/1.0.0/type/android/id/3cf425c8b814ad98'

Step 2: Removing the user from the Device Management Console

You need to remove the user from the device management console before removing the user details from the log files and RDBMS.

Follow the instructions below to edit the user details:

Sign in to the WSO2 IoTS device management console: https://<IOTS_HOST>:9443/devicemgt
- Enter WSO2 IoT Server's host or IP as the value for <IOTS_HOST>. The default host is localhost.
- The default port is 9443. If you port offset WSO2 IoT Server's core profile, make sure to offset the port here too.
  For example, if you port offset the WSO2 IoT Server's core profile by 2, the port needs to be 9445.
Click the icon.
Click User Management.
Click the remove icon on the user you wish to remove.

Click REMOVE to confirm that you want to remove the user.

Step 3: Removing the data from the logs and other RDBMS of IoT Server

Now, you need to remove the user and device details from the logs and the device_enroll, policy, and device_group databases using the identity-anonymization-tool tool. WSO2 IoT Server is shipped with the Identity Anonymization Tool, which can anonymize a user's PII in log files and RDBMSs by replacing all occurrences of the deleted user with either a randomly generated UUID value or a specified pseudonym. This tool is stored in the <IOTS_HOME>/wso2/tools/identity-anonymization-tool directory.

Note!

Note that this tool is designed to run in offline mode (i.e., the server should be shut down or run on another machine) in order to prevent unnecessary load to the server. If this tool runs in online mode (i.e., when the server is running), the H2 databases may go into the DB lock situation.
If you have configured a database other than the default H2 database, copy the relevant driver to the <IOTS_HOME>/wso2/tools/identity-anonymization-tool/lib directory.
Only removes a deleted user's PII from archived log files, and not the live log files that are connected to the system.

Before you start removing PII stored by the WSO2 IoT Server's core profile, be sure that the relevant user has been inactive in the system for a sufficient amount of time. This will ensure that all of the user's PII contained in log files are successfully archived. You can then follow the instructions given below to remove the user's PII references from the archived log files.

Open the <IOTS_HOME>/wso2/tools/identity-anonymization-tool/conf/datasources directory and configure the following files.
- Open the apim-datasources.xml file and add the <IOTS_HOME> path for the <url> property instead of ${APIM_HOME}.
  Example:
```
<url>jdbc:h2:documents/iots-server/repository/database/
```
- Open the iot-datasources.xml file and add the <IOTS_HOME> path for the <url> property instead of {IoT_HOME} .
  Example:
```
<url>jdbc:h2:documents/iots-server/repository/database/WSO2DM_DB;DB_CLOSE_ON_EXIT=FALSE;AUTO_SERVER=TRUE;LOCK_TIMEOUT=60000</url>
```
- Open the u m-datasources.xml file and add the <IOTS_HOME> path for the <url> property instead of ${APIM_HOME} .
  Example:
```
<url>jdbc:h2:documents/iots-server/repository/database/
```
Navigate to the <IOTS_HOME>/bin directory via the terminal.
```
cd <IOTS_HOME>/bin
```
Execute one of the following commands:
- Deleting the username from the logs and RDBMS.
```
-------On Linux/Mac OS------------------- 
./forget-me.sh -U <USERNAME>

-------On Windows-------------------------
forget-me.bat -U <USERNMAE>
```
  All references to the user are removed from WSO2 IoT Server. You can view the generated reports inside the <IOTS_HOME>/repository/components/tools/forget-me/conf directory.
- Deleting the device Identifier from the logs and RDBMSs.
```
-------On Linux/Mac OS------------------- 
./forget-me.sh -U <DEVICE_ID>

-------On Windows-------------------------
forget-me.bat -U <DEVICE_ID>
```
  All references to the device ID are removed from WSO2 IoT Server. You can view the generated reports inside the <IOTS_HOME>/repository/components/tools/forget-me/conf directory.
  This will result in the following:
  - Copies will be created of all the log files specified in the <IOTS_HOME>/wso2/tool/identity-anonymization-tool/config.json file.
    The copy of the log files that are created in the <IOTS_HOME>/repository/logs directory takes the following format: anon-<time_stamp>-<original_log_name>.log For example, anon-1520946791793-audit.log.
  - The PII will be anonymized in the copies. The log files will display the user information as a pseudonym.
Delete original (archived) log files
Note that the PII is not removed from the original log files. It is the responsibility of the organization to remove the original log files that contain the user's PII.

Step 4: Removing the PII from the archival tables

WSO2 IoT Server stores the operation details of each device for audit purposes. When large amounts of operation data get accumulated, it affects the performance of the server as all these details are stored in the database. Therefore, WSO2 IoT Server introduces a method to archive the operation details of the devices. It allows you to archive the operation data and completely remove the data from the archival table after a specific time period. This feature is disabled by default.

Note!

If you have enabled operation data archival, you need to make sure that you delete user specific details when a user requests the right to be forgotten or you can configure the data purging task to delete the data from the archival table without keeping them for 365 days or a specified time period.

By default, the archival data is stored in the jdbc/DM_ARCHIVAL_DS datasource that is configured in the <IOTS_HOME>/conf/datasources/cdm-datasources.xml file. The default configurations connect to the H2 database, which should not be used in production.

For more information, see Archiving Operation Data to Improve Performance in the WSO2 IoT Server Documentation.
For more information, see Archiving Operation Data to Improve Performance in the WSO2 EMM Documentation.

Removing PII from the Analytics profile

Shown below is an example data stream used by the Analytics profile. Note that the username, email and the date of birth are personally identifiable information (PII) of the user.

Stream Name	Attribute List
`org.wso2.gdpr.students`	`username` `email` `dateOfBirth`
`org.wso2.gdpr.students.marks`	`username` `marks`

These PII references can be removed from the Analytics database by using the Identity Anonymization Tool. Follow the steps given below.

Add the relevant drivers for your Analytics-specific databases to the <IOTS_HOME>/wso2/tools/identity-anonymization-tool/lib directory. For example, if you have changed your Analytics databases from the default H2 instances to MySQL, copy the MySQL driver to this given directory.
Create a folder named streams in the <IOTS_HOME>/wso2/tools/identity-anonymization-tool/conf/ directory.
Create a new file named streams.json with the content shown below, and store it in the <IOTS_HOME>/wso2/tools/identity-anonymization-tool/conf/streams directory that you created in the previous step. This file holds the details of the streams and the attributes with PII that we need to remove from the database.
```
{
    "streams": [
        {
            "streamName": "org.wso2.gdpr.students",
            "attributes": ["username", "email", "dateOfBirth"],
            "id": "username"
        },
        {
            "streamName": "org.wso2.gdpr.students.marks",
            "attributes": ["username"],
            "id": "username"
        }
    ]
}
```
The above configuration includes the following:
- Stream Name: The name of the stream.
- Attributes: The list of attributes that contain PII.
- id: The ID attribute, which holds the value that needs to be anonymized (replaced with a pseudonym).

Update the config.json file ( stored in the <IOTS_HOME>/wso2/tools/identity-anonymization-tool/conf/ directory) as shown below.

{
    "processors": [
        "analytics-streams"
    ],
    "directories": [
        {
            "dir": "analytics-streams",
            "type": "analytics-streams",
            "processor": "analytics-streams"
        }
    ]
}

Navigate to the <IOTS_HOME>/bin directory via the terminal.
Run the tool using the following command:
```
./forgetme.sh -U <USERNAME> -carbon <IOTS_HOME>/wso2/analytics
```
This will result in the following:
- Copies will be created of all the log files specified in the <IOTS_HOME>/wso2/tool/identity-anonymization-tool/config.json file.
  The following is the format of the log copy that is created in the <IOTS_HOME>/repository/logs directory: anon-<time_stamp>-<original_log_name>.log. For example, anon-1520946791793-audit.log.
- The PII will be anonymized in the copies. The log files will display the user information as a pseudonym.
Delete original (archived) log files
Note that the PII is not removed from the original log files. It is the responsibility of the organization to remove the original log files that contain the user's PII.

Running the tool in standalone mode

This tool can run standalone and therefore, cater to multiple products. This means that if you are using multiple WSO2 products and need to delete the user's identity from all products at once, you can do so by running the tool in standalone mode.
For information on how to build and run the Forget-Me tool, see Removing References to Deleted User Identities in WSO2 Products in the WSO2 Administration Guide.