Versions Compared

Old Version 139

changes.mady.by.user Former user

Saved on

compared with

New Version 140

changes.mady.by.user Former user

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This guide provides a quick introduction This guide provides a quick introduction to using WSO2 Data Analytics Server (DAS). 

...

Panel

Deploying the sample C-App

You can deploy artifacts (i.e. event streams, event receivers, Spark scripts, event publishers, and dashboards etc.) as composite Carbon Applications (C-Apps) in WSO2 DAS. This guide uses the SMART_HOME.car file as the toolbox which contains all the artifacts required for this guide in a single package. For more information on C-Apps, see Carbon Application Deployment for DAS. Follow the steps below to deploy and use a sample C-App in WSO2 DAS.

  1. Log in to the DAS management console using the following URL: https://<DAS_HOST>:<DAS_PORT>/carbon/
  2. Click  Main , and then click  Add  in the Carbon Applications menu.
  3. Click  Choose File, and upload the <DAS_HOME>/samples/capps/Smart_Home.car file as shown below.
    adding the new C-App
  4.   Click  Main , then click Carbon Applications, and then click List view, to see the uploaded Carbon application as shown below.  
    list of all available C-Apps

       

 

Tip

You can use the Event Flow feature of WSO2 DAS to visualize how the components that you created above are connected with each other. Also, you can use it for verification purposes i.e. to validate the flow of the events within the DAS as shown below.

Image Modified


Publishing events 

Once you develop the complete Event Flow, you can test the flow by publishing the events to the DAS. There are several methods of publishing to DAS. This guide uses the event simulation method using the Event Simulator.

Event Simulator is a tool that can be used for monitoring and debugging event streams. You can use this tool to simulate events by creating event(s) with values assigned to event stream attributes. Follow the steps below to perform event simulation to publish data to WSO2 DAS.

  • Log in to the DAS Management Console, if you are not already logged in.

  • Click Tools, and then click Event Simulator.

  • Download the sample.csv file which contains a set of events records collected from household ‘smart plug’ sensors.

  • In the Send multiple events option, click Choose File.

  • Select the sample.csv file which you downloaded, and click Upload.

  • Click OK in the pop up message which indicates successful uploading of the CSV file, and refresh the page to view the uploaded file which is displayed as shown below.
    Image Removed 

    • Click Configure, and enter the details as shown below.

    Tip

    Select SMARTHOME_DATA:1.0.0 for Select the target event stream, and type a comma in the provided text field for Field delimeter.

    event mapping configurationImage Removed
    ParameterValue
    File Namesample.csv
    Select the target event streamSMARTHOME_DATA:1.0.0
    Field delimiter,
    Delay between events in milliseconds1000

  • Click Configure, and then click OK in the message which pops up. As a result of this configuration, the Play button appears for the sample.csv file in the Event Stream Simulator page as shown below.
    Image Removed 

    • Click Play to start simulating the events in the uploaded file to publish events as shown below

    In this section, the events are published via a log file.

    Navigate to <DAS_HOME>/samples/smart-home/ directory in a new CLI tab, and execute the following command to run the data publisher: ant

    Info

    This reads the <DAS_HOME>/samples/smart-home/resources/access.log file, and sends each log line as an event to the event stream which is deployed through the Smart_Home.car file.

    Viewing the output

    Follow the steps below to view the presentation of the output in the Analytics Dashboard.

    1. Log in to the Management console, if you are not already logged in.

    2. Click  Main , and then click  Analytics Dashboard in the Dashboard menu. 

    3. Log in to the Analytics Dashboard, using admin/admin credentials. 

    4. Click the DASHBOARDS button in the top menu. You view the dashboard deployed by the C-App as shown below.

      Image Removed

    5. Click the View button of the corresponding Dashboardin to the Analytics Dashboard, using admin/admin credentials. 

    6. Click the Home DASHBOARDS button in the top menu of the Smart Home dashboard. You view the usage statistics output of the processed events in a line chart dashboard deployed by the C-App as shown below.

      usage statisticsImage Removed Image Added

    7. Click the Plug_Usage in the top menu View button of the Power_Dashboard. You view the plug usage data output of the processed events in a bar chart as shown below. plug usage outputImage Removed corresponding Dashboard. The following charts are displayed.
      Image Added
      Image Added Image Added

     

    Tip

     Follow the steps below to undeploy the C-App, which you already uploaded in this section before proceeding to the next sections.

    1. Log in to the DAS Management Console using admin/admin credentials, if you are not already logged in.
    2. Click Main, then click Carbon Applications, and then click List view, to see the uploaded Carbon application.
    3. Click on the Delete option to delete the Carbon application as shown below.
      Image Modified
    4. Refresh the Web browser screen, and check if the SMART_HOME.car file has beed removed from the list of all available C-Apps. 

    ...

    Panel


    Anchor
    Aggregating data
    Aggregating data
    Collecting data

    The first step of WSO2 DAS workflow is to collect data. In the data collection process, first you need to create the event stream definition. An e vent is a unit of data collection, and an e vent stream is a sequence of events of a particular type which consists of a set of unique attributes.

    WSO2 DAS exposes a single API for external data sources to publish data events to it, and provides configurable options to either process the data event stream inflow (in memory) for realtime analytics, persist (in data storage) for batch analytics, and index for interactive analytics.

    Anchor
    Creating the event stream
    Creating the event stream
    Creating the event stream

    The first step in collecting data is defining an event stream by creating the event stream definition. The defined stream provides the structure required to process the events. For more information on event streams, see  Event Streams . Follow the steps below to create the event stream.

    1. Log in to the DAS Management Console using the following URL and admin/admin credentials: https://10.100.5.72:9443/carbon/
    2. Click Main, and then click Streams.

    3. Click Add Event Stream, and enter the details as shown below.
      creating a new event stream
      Event Stream Details

      ParameterValue
      Event Stream NameSMARTHOME_DATA
      Event Stream Version1.0.0

       

      Payload Data Attributes  

      Info

      Click Add to add the attribute after entering the attribute name and attribute type. 


      AttributeAttribute Type
      idstring
      valuefloat
      propertybool
      plug_idint
      household_idint
      house_idint
    4. Click Add Event Stream. The new event stream is added to the list of all available event streams as shown below.
      list of all available event streams
     
    Anchor
    Persisting the event stream
    Persisting the event stream
    Persisting the event stream

    Events received by the DAS can be processed either in realtime and/or in batch mode. You need to persist the event information, to process the events in batch mode. However, if you process the events in realtime you do not need to persist them. For persisted events, configurable options are provided to index the data.

    WSO2 DAS introduces a pluggable architecture which allows you to persist data events into any Relational Data Storage (Oracle, MSSQL, MySQL etc.) or NoSQL storages (Apache HBase and Apache Cassandra). Multi data event storage is also possible. F or an example, events can be stored in a NoSQL storage while the processed data events can be stored in a relational data storage.

    Follow the steps below to persist received events.

    1. For persisted events, configurable options are provided to index the data which is required for Interactive Analytics later in the example.

      Log in to the DAS Management Console, if you are not already logged in.
    2. Click Main, and then click Streams.
    3. Click Edit option of the corresponding event stream as shown below. 
      editing the created event stream
    4. Click Next[Persist Event].
    5. Select the Persist Event Stream check box. As a result, the Persist Attribute check boxes are selected for all the attributes. Then select the Index Column checkbox for the house_id attribute as shown below. 
      persisting the event
    6. Click Save Event Stream.
    7. Click Yes in the pop-up message as shown below. 
      pop up message of persisting an event
    8. You view the persisted event stream added to the list of all available event streams as shown below. 
      persisted event added to the list of all available event streams
    Anchor
    Creating the event receiver
    Creating the event receiver
    Creating the event receiver

    Once you define the event stream and configure how it should be used, you need to create event receivers to connect WSO2 DAS with different data sources.

    WSO2 DAS supports event retrieval from many transport protocols and different formats. For information on supported transport protocols and type formats, see Configuring Event Receivers. Follow the steps below to create an event receiver of the WSO2Event type for this guide.

    Info

    WSO2Event event receiver is used to receive events in the WSO2Event format via Thrift or binary protocols. For more information, see WSO2Event Event Receiver.


    1. Log in to the DAS Management Console, if you are not already logged in.
    2. Click Main, and then click Receivers.

    3. Click Add Event Receiver, and enter the details as shown below.

      add a new event receiver

      ParameterValue
      Event Receiver NameDATA_RECEIVER
      Input Event Adapter Typewso2event
      Event StreamSMARTHOME_DATA:1.0.0
      Message Formatwso2event


    4. Click Add Event Receiver. You view the new event receiver added to the list of all available event receivers as shown below. 

      event receivers list
    Creating another event stream

    The SMARTHOME_DATA event stream you have already created serves as the input stream in this scenario. Events of this stream need to be forwarded to another stream once they are processed in order to be published. Follow the steps below to add the output event stream to which the processed data is forwarded.

    1. Log in to the DAS Management Console if you are not already logged in.
    2. Click Main, and then click Streams.
    3. Click  Add Event Stream , and enter the details as shown below.
       

      Event Stream Details

      ParameterValue
      Event Stream NameusageStream
      Event Stream Version1.0.0

       

      Payload Data Attributes 

      Info

      Click Add to add the attribute after entering the attribute name and attribute type. 

      AttributeAttribute Type
      house_idint
      maxValfloat
      minValfloat
      avgValdouble
      currentTimestring

       

    4. Click Next[Persist Event].

    5. Select the Persist Event Stream check box. As a result, the Persist Attribute check box is selected for all the attributes as shown below.

      persisting the output event stream 

    6. Click    Add Event Stream   . The new event stream is added to the list of all available event streams as shown below.

       
    Creating an event publisher  

    Once the events are processed, events publishers are used to publish results to external systems for taking further actions. Event publishers provide the capability to send event notifications and alerts from WSO2 DAS to external systems. Follow the steps below to create a new event publisher.

    1.  Log in to the DAS Management Console, if you are not already logged in.

    2. Click Main, and then click Publishers in the Event menu.

    3. Click Add Event Publisher, and enter the following details as shown below.
      create a new event publisher

      ParameterValue
      Event Publisher NameDATA_PUBLISHER
      Event SourceusageStream:1.0.0
      Output Event Adapter Typelogger
      Message Formattext

    4. Click Add Event Publisher. You view the new event publisher added to the list of all event publishers as shown below. 

      Info

      Since you created a logger type event publisher, the output is written to the CLI.

      list of all available event publishers

    Anchor
    Publishing events
    Publishing events
    Publishing events

    Event Simulator is a tool which you can use for publishing events to event streams. You need to create event(s) by assigning values to event stream attributes to simulate them. For more information, see Publishing Data Using Event Simulation. Follow the steps below to perform event simulation.

    1. Log in to the DAS Management Console, if you are not already logged in.

    2. Click Tools, and then click Event Simulator.

    3. Download the sample.csv  file which contains a set of events records collected from household ‘smart plug’ sensors.

      Note

      If you have the sample.csv file already uploaded you can skip step 4 to 6 below.

    4. In the Send multiple events option, click Choose File.

    5. Select the sample.csv file which you downloaded, and click Upload.

    6. Click OK in the pop up message which indicates successful uploading of the CSV file, and refresh the page to view the uploaded file which is displayed as shown below.
       

    7. Click Configure, and enter the details as shown below.

      Tip

      Select SMARTHOME_DATA:1.0.0 for Select the target event stream, and type a comma in the provided text field for Field delimeter.


      event mapping configuration

      ParameterValue
      File Namesample.csv
      Select the target event streamSMARTHOME_DATA:1.0.0
      Field delimiter,
      Delay between events in milliseconds1000

       

    8. Click Configure, and then click OK in the message which pops up.

    9. Play to start simulating the events in the uploaded file to publish events as shown below.

      play events to publish them

    ...

    Panel

    Anchor
    Analysing data
    Analysing data
    Analysing data

    You can configure any data event stream received by WSO2 DAS to perform batch, real time, and/or interactive analytics as described in the below sections. The first section demonstrates how to perform batch analytics.

    Panel
    Anchor
    Batch analytics
    Batch analytics
    Batch analytics

    You can perform batch analytics when event streams are configured to be persisted for later batch processing scenarios such as data aggregation, summarization etc. WSO2 DAS batch analytics engine is powered by Apache Spark, which accesses the underlying data storage and executes programs to process the event data.  The DAS provides an SQL-like query language to create the jobs through scripts that need to be executed. For more information, see  Data Analysis. You can perform batch analytics either using Spark analytics scripts or using the Spark Console as described below.

    Tip

    You need to follow instructions in the Collecting data section (i.e. create the event stream, persist it, create the event receiver, and publish events to WSO2 DAS), before performing the following batch analytics operations.

    Anchor
    Creating the analytics script
    Creating the analytics script
    Using the analytics script

    Follow the steps below to create the analytics script.

    1. Log in to the DAS Management Console, if you are not already logged in.
    2. Click Main, and then click Scripts in the Batch Analytics menu.
    3. Click Add New Analytics Script.

    4. Enter BATCH_ANALYTICS_SCRIPT in the Script Name parameter. Enter the following Spark SQL script in the Spark SQL Queries parameter as shown below.  

      Code Block
      languagesql
      CREATE TEMPORARY TABLE homeData USING CarbonAnalytics OPTIONS (tableName "SMARTHOME_DATA", schema "id STRING, value FLOAT, property BOOLEAN, plug_id INT, household_id INT, house_id INT");

create temporary table plugUsage using CarbonAnalytics options (tableName "plug_usage", schema "house_id INT, household_id INT, plug_id INT, usage FLOAT -sp");

insert overwrite table plugUsage select house_id, household_id, plug_id, max(value) - min (value) as usage from homeData where property = false group by house_id, household_id, plug_id ;
 
Select * from plugUsage where usage>300


      add new script

      Info

      The above script does the following:

      • Loads data from the DAS Data Access Layer (DAL), and registers temporary tables in the Spark environment.
      • Performs batch processing by calculating the usage value (through the difference of the max and min values), grouped by house_id, household_id, and plug_id from data of the temporary homeData table.
      • Writes back to a new DAL table named plug_usage.
      • Executes the following query: Select * from plugUsage where usage>30
    5. Click Execute Script, to check the validity of the script as shown below. You view the results as shown below.
      execution results of the Spark script
    6. Click Add. You view the new script added to the list of all available scripts as shown below. 
      list of all available scripts
    Anchor
    Using the Spark Console
    Using the Spark Console
    Using the Spark Console

    You can obtain faster results by executing adhoc queries on the indexed attributes through an interactive Web console, named as the named as the Spark Console. Follow the steps below to perform a batch analytics operation using the Spark Console. 

    1. Log in to the DAS Management Console, if you are not already logged in.
    2. Click Main, and then click Console in the Batch Analytics menu.

    3. Enter the following Spark SQL query in the console, and press Enter key.

      Code Block
      languagesql
      Select * from plugUsage where usage>100

      You view the output as shown below.

     

    You can perform realtime analytics in WSO2 DAS  as described in the next Realtime analytics section below .

    Panel
    Anchor
    Realtime analytics
    Realtime analytics
    Realtime analytics

    The realtime analytics engine uses a set of specified queries or rules through a SQL-like Siddhi Query Language defined in an execution plan, to process multiple event streams in realtime. An execution plan consists of a set of queries and import and export streams. It is the store of the event logic that is bound to an instance of the server runtime, and acts as the editor for defining the event processing logic. For more information see,  Working with Execution Plans . You can perform for realtime analytics using the same event stream which you used before to perform batch analytics by proceeding the event streams inflow through the WSO2 real time analytics engine as described below.

    Tip

    You need to follow instructions in the Collecting data section (i.e. create the event stream, persist it, create the event receiver, create the event publisher, and publish events to WSO2 DAS), before performing the following realtime analytics operations.

    Creating the execution plan

    Follow the steps below to create an execution plan.

    1. Log in to the DAS Management Console, if you are not already logged in.
    2. Click Main, and then click Execution Plans in the Streaming Analytics menu.

    3. Click Add Execution Plan, and enter the following details as shown below.

      Info

      In the below execution plan, the realtime engine (Siddhi) collects 10 incoming data events within a one minute time window from the incoming filtered stream, calculates the average, maximum and minimum values in realtime grouped by house_id, and sends the processed event to another newly defined stream.


      • Select  SMARTHOME_DATA:1.0.0 for Import Stream, enter  inputStream for As, and click Import.

      • Add the following query at the end of the provided space.

        Code Block
        languagesql
        from inputStream[value>0]#window.time(1 min) 
select house_id,max(value) as maxVal,min(value) as minVal, avg(value) as avgVal,time:currentTime() as currentTime
group by house_id
insert current events into usageStream ;


      create a new execution plan

    4. Select  usageStream 1.0.0 for StreamID of Export Stream, and click Export as shown below.
    5. Click Validate Query Expressions, to validate the execution plan.
    6. Click Add Execution Plan. You view the new execution plan added to the list of all available execution plans as shown below.  
      list of all available execution plans

    7. Publish events to WSO2 DAS by simulating events. For instructions, see Publishing events.

    8. View the output logged with the published events in the CLI on which you ran WSO2 DAS as shown below.  

      output logs of DAS CLI

    You can perform interactive analytcis in WSO2 DAS  as described in the next Interactive analytics section below .

    Panel
    Anchor
    Interactive analytics
    Interactive analytics
    Interactive analytics 

    Interactive analytics are used for retrieval of fast results through ad hoc querying of a received and processed data. It is possible in the DAS, when you select to index event stream attributes. You can obtain faster results by executing ad hoc queries on the indexed attributes through the provided Data Explorer.

    Tip

    You need to follow instructions in the Collecting data section (i.e. create the event stream, persist it, create the event receiver, and publish events to WSO2 DAS), before performing the following interactive analytics operations. The house_id field which you indexed in the step 5 of data collection is used to search data that match the specified query.

    Using the Data Explorer

    The Data Explorer is the Web console for searching analytical data. Primary key, data range, facet search options are available for simple analytical record searches. It is also possible to search records by providing Lucene queries for advanced searches. Follow the steps below to perform an interactive analytics operation using the Data Explorer.  

    1. Log in to the DAS Management Console, if you are not already logged in.
    2. Click Main, and then click Data Explorer in the Interactive Analytics menu.
    3. Select SMARTHOME_DATA for the Table Name parameter.
    4. Select By Query option and enter house_id:39 in the search field as shown below.
      message console
    5. Click Search. You view the output as shown below.
      output of the Message Console

    ...

    Panel

    Anchor
    Communicating results
    Communicating results
    Communicating results

    The final step in the event flow is to visualize the data. WSO2 DAS uses several presentation mechanisms to present event notifications and processing results. Thereby, it provides the Analytics Dashboard to visualize the processed data for decision making.

    The Analytics Dashboard is used to create customizable dashboards for analytics data visualization. Dashboard creation is wizard driven, where you can use gadgets/widgets such as Line, Bar, Arc charts to get data from analytical tables and add them on a structured grid layout to provide an overall view on the analyses. For more information, see  Presenting Data .

    Using the Analytics Dashboard

    WSO2 DAS provides an Analytics Dashboard for creating customizable dashboards for visualization of analytical data. Follow the steps below to present data using the Analytics Dashboard. 

    1. Log in to the DAS Management Console, if you are not already logged in.
    2. Click Main, and then click Analytics Dashboard in the Dashboard menu.
    Creating a Dashboard

    You can create a new Dashboard to present the data of the above analytics as shown in the example below. Follow the steps below to create a new Dashboard in the Analytics Dashboard. For more information, see  Adding a Dashboard.

    1. Click the  following CREATE DASHBOARD button in the top navigational bar to create a new dashboard.

    2. Enter a Title and a Description for the new dashboard as shown below, and click Next.

    3. Select a layout to place its components as shown below.

      selecting a layout for the new dashboard

    4. Click Select. You view a layout editor with the chosen layout blocks marked using dashed lines as shown below. Now the dashboard is persisted to the disk.

      dashboard editor

    5. Click the following icon in the top menu. button to view all dashboards

    6. Click Dashboards to view the new dashboard added to the list of all available dashboards as shown below.
      creating a new Dashboard

    Creating a Gadget

    Follow the steps below to create a Bar Chart Gadget to visualize the analyzed data by selecting the event stream created above (i.e.  UsageStream) as the data source, and add it to the Power_Dashboard Dashboard you created above. For more information, see  Adding Gadgets to a Dashboard.

    1. Log in to the Analytics Dashboard, if you are not already logged in.

    2. Click the following CREATE GADGET icon in the top menu bar. 

    3. Select the  UsageStream as the input data source as shown below.

      select the data source

    4. Select  Chart Type  and enter the preferred x, y axis and additional parameters based on the selected chart type as shown below. 

      configuring the chart type

    5. Click  Add to Gadget Store  to generate a gadget with the information you provided. 

    6. Click the corresponding  Design   button of the dashboard to which you want to add a gadget as shown below.

    7. Click the following gadget browser icon in the side menu bar.

      You view the new gadget listed in the gadget browser. If not, search for it using its name.

    8. Click on  the new gadget, drag it out, and place it in the preferred grid of the selected layout in the dashboard editor as shown below.

      viewing the new gadget on dashboard

    ...