Extending Mediation Statistics

The latest WSO2 ESB releases introduce the concept of custom statistics consumers (CSC). A CSC is a user developed extension of the mediation statistics component. It can be deployed on the ESB to get statistics from the in-memory data store and further process them. One of the most common use cases of CSC is to get the mediation statistics collected by the bus and write them to an external database. Another usage is to make the statistics available to an external monitoring application.

Follow the instructions below to develop and deploy a CSC on WSO2 ESB.

Step 1. Implement the MediationStatisticsObserver Interface

All custom statistics consumers must implement the MediationStatisticsObserver interface. This interface can be found in the org.wso2.carbon.mediation.statistics component. In WSO2 ESB version 3.0 this JAR file can be found in the repository/components/plugins directory. The interface consists of two methods. A simple implementation of the interface is given below as an example. It prints out some information using the statistics records and demonstrates the power of the API.

package org.wso2.esb.demo;

import org.wso2.carbon.mediation.statistics.MediationStatisticsObserver;
import org.wso2.carbon.mediation.statistics.MediationStatisticsSnapshot;
import org.wso2.carbon.mediation.statistics.StatisticsRecord;

import java.util.Date;

public class MyStatisticsConsumer implements MediationStatisticsObserver {

    public void destroy() {
        System.out.println("Destroying the custom statistics consumer");
    }

    public void updateStatistics(MediationStatisticsSnapshot snapshot) {
        System.out.println("Received statistics update at : " + new Date());

        StatisticsRecord latestRecord;
        if (snapshot.getEntitySnapshot() == null) {
            // Entity snapshot is null for the very first update
            latestRecord = snapshot.getUpdate();
        } else {
            // If the entity snapshot is not null combine it with the current update
            // to obtain the latest cumulative record
            latestRecord = new StatisticsRecord(snapshot.getEntitySnapshot());
            latestRecord.updateRecord(snapshot.getUpdate());
        }

        String direction = latestRecord.isInStatistic() ? "In" : "Out";

        System.out.println("Latest statistics for " + latestRecord.getType() + " " +
                latestRecord.getResourceId() + "(" + direction + ")");

        System.out.println("\tTotal count: " + latestRecord.getTotalCount());
        System.out.println("\tFault count:" + latestRecord.getFaultCount());
        System.out.println("\tMax time:" + latestRecord.getMaxTime());
        System.out.println("\tMin time:" + latestRecord.getMinTime());
        System.out.println("\tAverage time:" + latestRecord.getAvgTime() + "\n");
    }
}

The updateStatistics method is invoked by the mediation statistics component whenever a record in the in-memory data store is updated. This method takes a MediationStatisticsSnapshot object as an argument. The following methods can be invoked on this object to retrieve the latest statistical information:

For the first update received from the mediation engine, entity snapshot and category snapshot will be null. The name of the Proxy, sequence or endpoint being updated can be retrieved as follows:

snapshot.getUpdate().getResourceID();

Or

snapshot.getEntitySnapshot().getResourceID();

The type of the resource can be determined by one of the following methods:

snapshot.getUpdate().getType();
snapshot.getEntitySnapshot().getType();
snapshot.getCategorySnapshot().getType();

The returned value is of type org.apache.synapse.aspects.ComponentType which is a Java enum. The following elements are defined in this enum:

• PROXYSERVICE
• ENDPOINT
• SEQUENCE
• ANY (not relevant to CSC)

The destroy() method of the interface is called at system shutdown. This method should be used to cleanup and release any resources used by the CSC.

Step 2. Deploy the Custom Statistics Consumer

Compile the Java code and package it into a JAR file. To compile the above sample code you have to add the Synapse core component to the classpath. In WSO2 ESB version 3.0 this component is named org.apache.synapse.synapse-core and it can be found in the repository/components/plugins directory.

Having built the JAR file containing the CSC implementation, place it in the repository/components/lib directory of the ESB. Now we should instruct the ESB to load the CSC at runtime. To do this, open up the carbon.xml file in the repository/conf directory and add the following XML snippet into it:

<MediationStat>
        <Observers>org.wso2.demo.CustomStatConsumer</Observers>
</MediationStat>

The text under the Observers element should represent a class name or a comma separated list of class names.

Step 3. Start the ESB and Test the Consumer

Now start the ESB. Deploy a sequence or a Proxy Service, and enable statistics on it using the user inteface. Send a few requests to the ESB so that some data will be collected by the data store. You will be able to see something similar to the following being printed on the console, by the custom consumer:

Received statistics update at : Sun Jun 06 15:41:57 IST 2010
Latest statistics for PROXYSERVICE FooProxy(In)
	Total count: 1
	Fault count:0
	Max time:269
	Min time:269
	Average time:269.0

Received statistics update at : Sun Jun 06 15:41:57 IST 2010
Latest statistics for PROXYSERVICE FooProxy(Out)
	Total count: 1
	Fault count:0
	Max time:1
	Min time:1
	Average time:1.0

Received statistics update at : Sun Jun 06 15:42:17 IST 2010
Latest statistics for PROXYSERVICE FooProxy(In)
	Total count: 2
	Fault count:0
	Max time:269
	Min time:10
	Average time:139.5

Received statistics update at : Sun Jun 06 15:42:17 IST 2010
Latest statistics for PROXYSERVICE FooProxy(Out)
	Total count: 2
	Fault count:0
	Max time:1
	Min time:0
	Average time:0.5

Best Practices for Writing Custom Statistics Consumers

• Implementations of the MediationStatisticsObserver interface should not attempt to modify the values of given StatisticsRecord instances, within the updateStatistics method. Doing so will modify the values stored in the in-memory data store. If it is required to modify the values, a copy of the relevant StatisticsRecord should be obtained by invoking the copy constructor:

  StatisticsRecord copy = new StatisticsRecord(original);

• The updateStatistics method should not take a long time to execute. Otherwise subsequent updates will be delayed. Time consuming tasks such as database lookups should not be performed within the updateStatistics method. Such operations should be carried out by separate threads. PersistingStatisticsObserver, one of the built-in statistics consumers, takes this approach. This consumer writes statistics to the registry via the WSO2 Registry API. Registry operations could be time consuming due to the database access overhead and the network latency. Therefore PersistingStatisticsObserver queues up all updates in an in-memory queue. The registry operations are then carried out by a separate thread.
• Always use a package name when developing a CSC. This is important to avoid any OSGi class loading issues.
• Integrate logging into the CSC implementation. This will help greatly while debugging and testing. One could easily use the Apache Commons Logging framework which is the default used in WSO2 ESB and WSO2 Carbon.
• If it is required to merge two records into one, use the updateRecord method of the StatisticsRecord class. For an example to merge the record "a" with "b" you can invoke the following statement:

  a.updateRecord(b);

Registering Custom Statistics Consumers

Previously the carbon.xml file was used to register the custom consumer with the mediation statistics component (step 2). It is also possible to do this programmatically as shown below:

MediationStatisticsStore.getInstance().registerObserver(customConsumer);

If you are developing a Carbon component, the MediationStatisticsStore can be accessed via OSGi declarative services. Use the Maven SCR plug-in to generate the service descriptors and add the following annotation to the service component class.

/**
* @scr.reference name="mediation.statistics"
* interface="org.wso2.carbon.mediation.statistics.services.MediationStatisticsService"
* cardinality="1..1" policy="dynamic"
* bind="setMediationStatisticsService" unbind="unsetMediationStatisticsService"
*/

Now in the Java code, one could do the following:

mediationStatisticsService.getStatisticsStore().registerObserver(customObserver);

This is the best way of registering custom statistics consumers programmatically since this method additionally ensures consistent ordering in class loading.