Page Comparison

Before you begin, 

1. Install Oracle Java SE Development Kit (JDK) version 1.7* or 1.8 and set the JAVA_HOME environment variable.
2. Download WSO2 DAS.
3. Start the DAS by going to <DAS_HOME>/bin using the command-line and executing wso2server.bat  (for Windows) or  wso2server.sh  (for Linux.) 

Batch and interactive analytics

You can perform batch analytics when event streams are configured to be persisted for later batch processing scenarios such as data aggregation, summarization etc. WSO2 DAS batch analytics engine is powered by Apache Spark, which accesses the underlying data storage and executes programs to process the event data. The DAS provides an SQL-like query language to create the jobs through scripts that need to be executed.

Step 1: Persist event stream

Anchor
persist persist

In this step, the org.wso2.event.sensor.stream and org.wso2.event.sensor.filtered.stream event streams that you previously created are persisted so that the data received by them are stored in the databases configured for WSO2 DAS by default.
 

1. Log in the the WSO2 DAS Management Console if you are not already logged in.
2. In the Main   tab, click Streams   to open the   Available Event Streams   page.
3. Click Edit for the event stream org.wso2.event.sensor.stream. This opens the Edit Event Stream page.
4. Click Next [Persist Event].
5. Select the Persist Event Stream check box.
6. Select the Persist Attribute check box for all the available attributes.
7. Select the Index Column check box for the sensorid attribute.
8. Click Save Event Stream to save the changes.
9. Similarly, persist the org.wso2.event.sensor.filtered.stream event stream as shown below.
    

Step 2: Simulate events

In this step, multiple events are simulated to be stored in the data base for batch analytics.

1. Download and save this file in a preferred location in your machine.
2. Log into the DAS management console using the following URL: https://<DAS_HOST>:<DAS_PORT>/carbon/.
3. Click Tools, and then click Event Simulator.
4. Select org.wso2.event.sensor.stream in the  Event Stream Name  field.
    
5. Click Choose File, and then browse and upload the CSV file you downloaded and saved. Click Upload and refresh the page to view the uploaded file.
6. Click Configure to open the Event Mapping Configuration dialog box. Enter a comma ( , ) in the Field delimiter field, and click Configure.
   
   
7. Click Play to start sending the events in the file. Click OK in the message that appears to confirm that the system has started sending events from the file. The events sent are logged in the CLI as shown below.
    

Step 3: Create and execute Spark scripts

In this exercise, a Spark query is written to process the data received by WSO2 DAS and stored in the databases.

1. Log in to the DAS Management Console, if you are not already logged in.
2. Click Main, and then click Scripts in the Batch Analytics menu to open the Available Analytics Scripts page.
3. Click Add New Analytics Script.

4. Enter BATCH_ANALYTICS_SCRIPT in the  Script Name parameter. Enter the following Spark SQL script in the  Spark SQL Queries parameter as shown below.
    
   

   Code Block
   language sql
   CREATE TEMPORARY TABLE sensorData USING CarbonAnalytics OPTIONS (tableName "ORG_WSO2_EVENT_SENSOR_STREAM", schema "humidity FLOAT, sensorValue DOUBLE"); create temporary table highSensorVal using CarbonAnalytics OPTIONS (tableName "highSensorVal", schema "humidity FLOAT, sensorValue DOUBLE"); insert overwrite table highSensorVal select humidity, sensorValue from sensorData WHERE sensorValue > 100; Select * from highSensorVal

     The above script does the following:   

   • Loads data from the DAS Data Access Layer (DAL), and registers temporary tables in the Spark environment.

   • Performs batch processing by fetching the sensor values greater than 100 from sensorData table.

   • Writes back to a new DAL table named highSensorVal.

   • Executes the following query:
     Select * from highSensorVal

    

5. Click Add to save the script. Click OK in the message that appears to confirm that the script was successfully saved.

6. Click Execute for the script.
   
   The following is displayed to indicate that the queries were successfully executed.
   

You can obtain faster results by executing adhoc queries on the indexed attributes through an interactive Web console named the Interactive Analytics Console. Follow the procedure below to perform a batch analytics operation using the Interactive Analytics Console.

1. Log into the DAS Management Console if you are not already logged in.
2. In the Main tab, click Console to open the Interactive Analytics Console.
3. Enter the following query in the console and press the Enter key.
   Select * from highSensorVal
   
   The output of the query is displayed as follows. 
   

Step 4: Search for data

This step involves performing interactive analytics for data received by DAS and stored in the configured databases. Searching is supported by the indexing functionality of DAS powered by Apache Lucene. In this step, we search for a specific record by the sensorid attribute. This is possible because the sensorid attribute was persisted as an index column when you persisted the event stream.

1. Log into the DAS Management Console if you are not already logged in.
2. In the Main tab, click Data Explorer to open the Data Explorer page.
3. In the Table Name parameter, select ORG_WSO2_EVENT_SENSOR_STREAM.
4. Select the By Query option, and enter meta_sensorid:501 in the data field displayed.
   
    
5. Click Search. The following records are displayed in the Results section.