Upgrading from WSO2 DAS 3.0.1

This section explains how to upgrade to DAS 3.1.0 from DAS 3.0.1. For more information on release versions, see the Release Matrix.

Preparing to upgrade

The following prerequisites should be completed before upgrading.

• Make a backup of the DAS 3.0.1 database and copy the <DAS_HOME_3.0.1> directory in order to backup the product configurations.
• Download any CAR files that are deployed in the DAS server and save them. Then delete them from DAS 3.0.1. This removes the scheduled tasks relating to the deployed artifacts and allows migration to be carried out smoothly. For more information, see Packaging Artifacts as a C-App Archive.
• Download WSO2 DAS 3.1.0 from http://wso2.com/products/data-analytics-server/.
• Make the required changes in the datasource files in the <DAS_HOME>/repository/conf/datasources  directory to make sure that they are configured as required. For more information, see Datasources.

Migrating Analytics tables

The table schema internally used by DAS 3.1.0 is different to that of DAS 3.0.x. Therefore, this meta data transformation should be done before using the Analytics tables in DAS 3.0.x in DAS 3.1.0 using the following procedure. The stored data itself is backward compatible. Therefore, it does not go through all the data. Instead, a constant time metadata conversion in tables is carried out.

1. Navigate to the <DAS_HOME>/bin directory.
2. Issue one of the following commands as appropriate.
   
   • On Windows:   analytics-backup.bat -migrateTableSchemaV30To31

   • On Linux/Solaris/Mac OS:   ./analytics-backup.sh -migrateTableSchemaV30To31

Migrating the configurations

Once you have completed the database upgrade, You can proceed with the configuration changes as explained below. The configurations can be directly migrated from DAS 3.0.1 to DAS 3.1.0.

To connect DAS 3.1.0 to the upgraded database, configure the following files:

1. Go to the <DAS_HOME>/repository/conf directory and update the datasource references in the user-mgt.xml and registry.xml files to match the updated configurations in the master-datasources.xml file that you made in the above step.
   For detailed instructions to configure the  user-mgt.xml file, see User Management.
   For detailed instructions to configure the  registry.xml file, see Registry.
2. Compare the configuration files in the < DAS_HOME>/repository/conf/data-bridge directory for DAS 3.0.1, and 3.1.0.
3. Compare the configuration files in the <DAS_HOME>/repository/conf/analytics directory for DAS 3.0.1, and 3.1.0. Change the default values in DAS 3.1.0 to the same values you have specified in DAS 3.0.1.

4. Change the default values in <DAS_3.1.0_HOME>/repository/conf/event-processor.xml file to the same values you have specified in the <DAS_3.0.1_HOME>/repository/conf/event-processor.xml file.

   If you enable the HA mode for WSO2 DAS by setting the <mode name="HA" enable="true"> property in the event-processor.xml file, state persistence is enabled by default. If there is no real time use case that requires any state information after starting the cluster, you should disable event persistence by setting the persistence attribute to false in the same file as shown below.

   <persistence enable="false">
       <persistenceIntervalInMinutes>15</persistenceIntervalInMinutes>
       <persisterSchedulerPoolSize>10</persisterSchedulerPoolSize>
       <persister class="org.wso2.carbon.event.processor.core.internal.persistence.FileSystemPersistenceStore">
           <property key="persistenceLocation">cep_persistence</property>
       </persister>
   </persistence>

5. Check for any other configurations that were done for DAS 3.0.1 based on your solution and update the configuration files in DAS 3.1.0 accordingly. For example, configurations related to external user stores, caching, mounting, transports etc.

Migrating index information

Each node stores its index data in the <DAS_HOME>/repository/data directory. There are two possible approaches to migrate this data. Click on the relevant tab based on the approach you want to follow.

Migrating artifacts

All artifacts can be migrated from DAS 3.0.1 to DAS 3.1.0 via Composite Application Archive (CAR) files. For detailed instructions package artifacts as C-App archives, see Packaging Artifacts as C-App Archive .

You should manually deploy Composite Application Archive (CAR) files that you have in DAS 3.0.1 to DAS 3.1.0.

Additional steps are required to migrate Spark scripts and dashboard elements and explained in the following sub sections.

Updating Spark scripts

The CarbonJDBC provider was rewritten for DAS 3.1.0. Therefore, the following additional input parameters are required to be added to the Spark Scripts migrated to DAS 3.1.0 for which Carbon JDBC is used as the provider.

• schema (Required): The schema to be used within Spark for the specifies table.
• primaryKeys (optional): Any database level unique keys that are either already specified or needed for the table.

The following example shows the difference in the query format between DAS 3.0.1 and DAS 3.1.0.

For more information about improvements made to the CarbonJDBC provider, see Creating the Table Using Carbon JDBC as the Provider.

Updating execution plans

WSO2 DAS 3.1.0 uses Siddhi 3.1.2 whereas DAS 3.0.1 uses Siddhi 3.0.5.

All Siddhi versions older than Siddhi 3.1.2 add a length window of size 1 for the initial stream of join queries. From Siddhi 3.1.2 onwards, a length window of size 0 is added. Therefore, once you migrate execution plans from DAS 3.0.1 to DAS 3.1.0, you need to manually add a length window of size 1 for the join queries as shown below (i.e. if you were relying on a length window of size 1 to be added by Siddhi in CEP 4.1.0).

e.g., The following query is included in an execution plan in DAS 3.0.1.

from TempStream[temp > 30.0]#window.time(1 min) as T join RegulatorStream[isOn == false] as R
  on T.roomNo == R.roomNo
select T.roomNo, R.deviceID, 'start' as action
insert into RegulatorActionStream;

When this execution plan is migrated to DAS 3.1.0, #window.length(1) property should be manually added to it as shown below.

from TempStream[temp > 30.0]#window.time(1 min) as T  join RegulatorStream[isOn == false]#window.length(1) as R
  on T.roomNo == R.roomNo
select T.roomNo, R.deviceID, 'start' as action
insert into RegulatorActionStream;

Migrating dashboard elements

The Analytics Dashboard functionality in DAS 3.1.0 is powered by WSO2 Dashboard Server 2.1.0 whereas the same functionality in DAS 3.0.1 and earlier versions is powered by WSO2 Gadget Server 1.4.0. This section explains how to migrate the gadgets and dashboards from DAS 3.0.1 to 4.2.0.

There are two methods that can be used to migrate gadgets and dashboards between the two DAS versions.

• Deploying the gadgets and dashboards in a CAR file
• Migrating the dashboards and gadgets pointing your product home directory.

In both methods, the following two files are used to execute the migration.

Click on the relevant tab depending on the method you want to follow.

Validating path changes

DAS 3.1.0 has a different directory structure to DAS 3.0.1 that affects the path to the gadget store as shown below.

DAS 3.0.1: <DAS_HOME>/repository/deployment/server/jaggeryapps/portal/store/<Tenant_Name>/gadget
DAS 3.1.0: <DAS_HOME>/repository/deployment/server/jaggeryapps/portal/store/<Tenant_Name>/<StoreType [fs/es]>/gadget

Therefore, if a gadget has relative paths included in its index.xml file (i.e. in the <DAS_HOME>/repository/deployment/server/jaggeryapps/portal/store/<Tenant_Name>/<StoreType [fs/es] >/gadget/<GADGET_NAME> path), you should check whether they are updated with the change in the directory structure. This check should be carried out after running the migration script.

e.g., A path is given as follows in the index.xml file in DAS 3.0.1.

<script language="javascript" type="text/javascript" src="../../../../js/igviz.js"></script>

The same link should be given as follows in DAS 3.1.0 due to the change in the directory structure (i.e. an additional ../ should be added for the new directory added to the structure).

<script language="javascript" type="text/javascript" src="../../../../../js/igviz.js"></script>

Updating layout configurations

The dashboard layout configurations are defined in the index.xml file. This file differs as follows based on the DAS version.

{
    "blocks": [
        { "id": "a", "row": 1, "col": 1, "size_x": 4, "size_y": 3 },
        { "id": "b", "row": 1, "col": 5, "size_x": 4, "size_y": 3 },
        { "id": "c", "row": 1, "col": 9, "size_x": 4, "size_y": 3 },
        { "id": "d", "row": 4, "col": 1, "size_x": 4, "size_y": 3 },
        { "id": "e", "row": 4, "col": 5, "size_x": 4, "size_y": 3 },
        { "id": "f", "row": 4, "col": 9, "size_x": 4, "size_y": 3 }
    ]
}

{
    "blocks": [
        { "id": "a", "x": 0, "y": 0, "width": 4, "height": 3 },
        { "id": "b", "x": 4, "y": 0, "width": 4, "height": 3 },
        { "id": "c", "x": 8, "y": 0, "width": 4, "height": 3 },
        { "id": "d", "x": 0, "y": 3, "width": 4, "height": 3 },
        { "id": "e", "x": 4, "y": 3, "width": 4, "height": 3 },
        { "id": "f", "x": 8, "y": 3, "width": 4, "height": 3 }
    ]
}

The layouts migrated from DAS 3.0.1 to DAS 3.1.0 will have the format in which layouts are saved in DAS 3.0.1. To make them compatible with DAS 3.1.0

1. Replace "size_x" and "size_y" attributes with "width" and "height" respectively.
2. Replace "row" and "col" attributes with "x" and "y" respectively, and reduce their values by 1 to match the 0 based indexing.

e.g., { "id": "a", "row": 1, "col": 1, "size_x": 4, "size_y": 3 } should be replaced with { "id": "a", "x": 0, "y": 0, "width": 4, "height": 3 }.

Testing the upgrade

Verify that all the required scenarios are working as expected. This confirms that the upgrade is successful.