Scheduling Batch Analytics Scripts

Apache Spark is used as the core analytics engine in DAS 3.0.0. For information on writing Spark queries to analyze the collected data, see Spark Query Language.

Analytics scripts

Analytics scripts are used when you have to execute a set of Spark queries in a sequence. Also, you can schedule a Analytics script, to trigger it to execute the query automatically in a given period of time. (E.g. fire at 12 (noon) every day, or fire at every minute starting at 2 p.m. and ending at 2:59 p.m. every day etc.). You need to configure this scheduled time using a cron expression. For more information about cron expressions, see Cron Trigger Tutorial.

You can add/edit/delete scripts, and also you can provide your schedule time for your script to execute as described below.

Adding a new script

Follow the steps below to add a new Spark script.

Log in to the WSO2 DAS Management Console.
In the Main tab, click Scripts to open the Available Analytics Scripts page.

Click Add New Analytics Script to open the Add New Analytics Script page. Then enter the following details related to your script as shown in the example below.

Script Name MyFirstAnalyticsScript

Spark SQL Queries

CREATE TEMPORARY TABLE Log USING CarbonAnalytics OPTIONS(TABLENAME "LOG", SCHEMA "server_name STRING, ip STRING, tenant INTEGER, sequence LONG, summary STRING");

INSERT INTO TABLE Log SELECT "das1", "192.168.1.1", 1, 12, "DAS 1";
INSERT INTO TABLE Log SELECT "das2", "192.168.1.2", 1, 20, "DAS 2";
INSERT INTO TABLE Log SELECT "esb", "192.168.1.41", 1, 17, "ESB";
INSERT INTO TABLE Log SELECT "apim_gateway1", "192.168.1.101", 1, 39, "API Manager Gateway 1";
INSERT INTO TABLE Log SELECT "apim_gateway2", "192.168.1.102", 1, 23, "API Manager Gateway 2";

SELECT ip FROM Log;
SELECT server_name, count(*) FROM Log GROUP BY server_name;
SELECT COUNT(*) FROM Log WHERE summary LIKE '%DAS%';
SELECT substr(summary, 1, 5) FROM Log;
SELECT LAST(ip) FROM Log;

Cron Expression

0 * * * * ?

This cron expression defines the schedule time of the script to execute it in every minute. From the time you save the script, the script will be executed at the beginning of every minute. (E.g:.10:21:00, 10:22:00, 10:23:00,..)

Click Add to save the script. As a result, the script is displayed in the Available Analytics Scripts page as shown below.

When a Spark script is created and saved, it is stored in the registry. It can be accessed using the /_system/config/repository/components/org.wso2.carbon.analytics.spark registry path. For more information about the registry, see Registry.

Editing a script

Follow the steps below to edit an existing Analytics script.

Log in to the WSO2 DAS Management Console.
In the Main tab, click Scripts to open the Available Analytics Scripts page.
Click Edit for the script you want to edit.
Change the content of the script as required. You can update the scheduling information as well.

When you do not enter any value for the scheduling time, then your script is not scheduled to execute. However, if you want to ensure that your script is valid, click Execute. This will execute the queries that you give in the queries window.
For example, you can edit the script created above to unschedule the scheduled time as shown below.
Click Update to save the changes as shown above.

Deleting a script

Follow the steps below to delete an Analytics script.

Log in to the WSO2 DAS Management Console.
In the Main tab, click Scripts to open the Available Analytics Scripts page.
Click Delete for the script you want to delete.
Click Yes in the dialog box which appears to confirm deletion.

If you delete the script you cannot undo that operation, and it will be completely removed from the system. Also, deleting the script will delete the scheduled task associated with it.

Executing a script

You can execute the script manually when you are adding/editing the script, without using any scheduled task. This will trigger the execution of the script content provided in the queries window at that moment. Also, you can execute the script content out of the edit mode as shown below. During this operation, WSO2 DAS fetches the script content and gives it to Spark to execute all the queries in the script. Once the execution is completed the results are displayed.

Follow the steps below to execute the script content.

Log in to the WSO2 DAS Management Console.
In the Main tab, click Scripts to open the Available Analytics Scripts page.
Click Execute, to execute the MyFirstAnalyticsScript query.

The following is displayed to indicate that the query is executed.