Configuring Received Data

This section describes the additional configurations that can be done to data collected by DAS before they are processed.

Setting a custom timestamp value for incoming events

When an event is received by WSO2 DAS, the timestamp of that event is set as a parameter value. This value often needs to be overridden in many situations including the following examples.

• Incremental data processing
• Timestamp based Analytics

In order to override the timestamp value assigned at the time an event is received, the client that publishes events to WSO2 DAS should set an attribute named _timestamp of the Long type as a payload attribute in each event.

Querying for the timestamp in a Spark script

You can query for events by the timestamp in an Analytics script. In order to do this, you need to add a _timestamp long option in the table schema.

e.g., If there is a table named TestTable1 in WSO2 DAS with the name STRING, b INT schema, the timestamps of the events in the table can be queried as follows.

CREATE TEMPORARY TABLE table1 USING CarbonAnalytics OPTIONS (tableName “TestTable1”, schema “a1 STRING, b1 INT, _timestamp LONG”) ;
SELECT a1, _timestamp FROM table1;

Additionally, you can use the _timestamp option to insert a custom value for the timestamp parameter of an event. 

e.g., If you need to insert a set of values from the table1 table to another table named table2, you can write a query as follows.

CREATE TEMPORARY TABLE table2 USING CarbonAnalytics OPTIONS (tableName “TestTable2”, schema “a2 STRING, _timestamp LONG”) ;

The following query inserts the results of the SELECT  a1, _timestamp FROM table1 query into the TestTable2 table.

INSERT INTO TABLE table2 SELECT  a1, _timestamp FROM table1;

INSERT INTO TABLE table2 SELECT  a1, -1 as _timestamp FROM table1;