Introducing DAS

eWSO2 Data Analytics Server (DAS) listens to a constant stream of events that represents the transactions and activities of an enterprise from different sources, processes them in real time and communicates the results in a variety of interfaces. This allows organisations to respond quickly to their environments, thus gaining an advantage over their competitors. In addition, WSO2 DAS combines real-time analytics with batch (equipped with incremental processing), interactive and predictive (via machine learning) analysis of data into one integrated platform to support the multiple demands of Internet of Things (IoT) solutions, as well as mobile and Web apps as illustrated by the image below.

As a part of WSO2’s analytics platform, WSO2 DAS introduces a single solution with the ability to build systems and applications that collect and analyze both realtime and persisted, data and communicate the results. It is designed to treat millions of events per second, and is therefore capable to handle Big Data volumes and Internet of Things projects.

WSO2 DAS workflow consists of the following three main phases.

Collecting data

WSO2 DAS exposes a single API for external data sources to publish data events to it. Further, it provides configurable options either to process the data event stream inflow (in memory) for realtime analytics, to persist (in data storage) for batch analytics and/or to index for interactive analytics as shown in the following diagram.

Creating event streams

The first step in aggregating data is defining an event stream by creating the event stream definition. A stream definition provides the initial structure and identification required for event processing. It includes a set of data types as properties, name, version and other attributes. When an external data publisher sends data events to WSO2 DAS (the receiver), it needs to specify the name and version of the stream intended. Also, data events should be sent according to the structure defined in the stream definition. For more information on event streams, see Event Streams.

Persisting events

This step is required if data needs to be stored in databases for batch analytics. WSO2 DAS introduces a pluggable architecture which allows you to persist data events into any relational data storage (i.e. Oracle, MSSQL, MySQL etc.), or NoSQL storage (i.e. Apache HBase, Apache Cassandra etc.). It is also possible for multi data event storage. For an example, the events can be stored in a NoSQL storage while the processed data events can be stored in a relational data storage.

Creating event receivers

Event Receivers are the connectors to different data sources in WSO2 DAS. WSO2 DAS support s event retrieval from many transport protocols and different formats. For information on the supported transport protocols and event formats, see Configuring Event Receivers.

Analyzing data

You can configure any data event stream received by WSO2 DAS for real time and/or batch analytics.

Realtime analytics

You can process the event streams inflow through the WSO2 real time analytics engine which is powered by Siddhi. The realtime analytic engine can process multiple event streams in realtime. For this, you need to specify a set of queries or rules using the SQL like Siddhi Query Language in an execution plan. Execution plan is the editor for the event processing logic. An execution plan consists of a set of queries and import and export streams. For more information see the following sections.

Batch and interactive analytics

You can perform batch analytics when you configure and persist event streams for batch processing scenarios such as data aggregation, summarization etc. WSO2 DAS batch analytics engine is powered by Apache Spark, which accesses the underlying data storage and executes programs to process the event data. An SQL-like query language is provided to create the jobs that needs to be executed. For more information, see Batch Analytics Using Spark SQL.

Interactive analytics are used when you need to get fast results through adhoc querying of a data set. Interactive analytics in WSO2 DAS is possible when you select to index event stream attributes.

Spark console

You can obtain faster results by executing adhoc queries on the indexed attributes through an interactive Web console, named as the Batch Analytics Console.

Event publishers

Event publishers provide the capability to send event notifications and alerts from WSO2 DAS to external systems. For more information, see Creating Alerts.

Event flow

You can use the Event Flow feature of WSO2 DAS to visualize how the components of it are connected with each other. Also, you can use it to validate the flow of the events within the DAS.

Event simulation

Event Simulator is a tool which you can use for monitoring and debugging event streams. You need to create event(s) by assigning values to event stream attributes to simulate them. For more information, see Publishing Data Using Event Simulation.

Data Explorer

The Data Explorer is the Web console for searching analytical data. Primary key, data range, facet search are some available options for simple analytical record searches. It is also possible to search records by providing Lucene queries for advanced searches. For more information, see Data Explorer.

Predictive analytics

You can perform predictive analytics for data in WSO2 DAS by integrating it with WSO2 ML. This integration allows WSO2 ML to use an Analytics table in WSO2 DAS as a dataset and make predictions/recommendations using that data by applying Machine Learner algorithms.

Communicating results

WSO2 DAS uses several presentation mechanisms to present event notifications and processing results. For more information, see Communicating Results.

Analytics Dashboard

WSO2 DAS provides an Analytics Dashboard for creating customizable dashboards for visualization of analytical data. Dashboard creation is wizard driven, where you can use widgets/gadgets such as line, bar, and arc charts to get data from analytical tables and add them on a structured grid layout to provide an overall view. For more information, see Analytics Dashboard.