Publishing API Runtime Statistics Using WSO2 BAM

In this section, we explain how to set up WSO2 Business Activity Monitor (version 2.5.0 is used here) to collect and analyze runtime statistics from the API Manager. To publish data from the API Manager to BAM, the Thrift protocol is used. Information processed in BAM is stored in a database from which the API Publisher retrieves information before displaying it in the corresponding UI screens.

By default, org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataPublisher is configured to push data events to WSO2 BAM. If you use a product other than WSO2 BAM to collect and analyze runtime statistics, you write a new data publishing agent by extending APIMgtUsageDataPublisher. Find the API templates inside <APIM_HOME>/repository/resources/api_templates. When writing a new data publishing agent, make sure the data publishing logic has a minimal impact to API invocation.

• JDK 1.6.* or 1.7

• Cygwin (http://www.cygwin.com): Required only if you are using Windows. WSO2 BAM analytics framework depends on Apache Hadoop, which requires Cygwin in order to run on Windows. Install at least the basic net (OpenSSH,tcp_wrapper packages) and security related Cygwin packages. After Cygwin installation, update the PATH variable with C:/cygwin/bin and restart BAM.

Next prepare BAM to collect and analyze statistics from the API Manager.

1. Download WSO2 BAM 2.5.0 from location: http://wso2.com/products/business-activity-monitor.

2. Apply an offset of 3 to the default BAM port by editing the <BAM_HOME>/repository/conf/carbon.xml file.

   <Offset>3</Offset>

   This increments all ports used by the server by 3, which means the BAM server will run on port 9446. Port offset is used to increment the default port by a given value. It avoids possible port conflicts when multiple WSO2 products run on same host. 

3. Install MySQL server and a suitable client like the MySQL Workbench. You can get the instructions manual from http://dev.mysql.com/doc/.

4. Go to the command-line and issue the following commands to connect to the MySQL server and create a database (e.g., TestStatsDB). This database is used to save the statistical data collected by the BAM. You do not need to create any tables in it.
   
   

   mysql -u <username> -p <password> -h <host_name or IP>; CREATE DATABASE TestStatsDB;

5. Save the MySQL connector JAR inside both <APIM_HOME>/repository/components/lib and <BAM_HOME>/repository/components/lib folders.

6. Give the datasource definition under the <datasource> element in the <BAM_HOME>/repository/conf/datasources/master-datasources.xml file. The tables are created automatically when the Hive script runs. You just need to create the schema. WSO2AM_STATS_DB is used to fetch analytical data from the database. The example below connects to a MySQL instance:

    <datasource> <name>WSO2AM_STATS_DB</name> <description>The datasource used for getting statistics to API Manager</description> <jndiConfig> <name>jdbc/WSO2AM_STATS_DB</name> </jndiConfig> <definition type="RDBMS"> <configuration> <url>jdbc:mysql://localhost:3306/TestStatsDB</url> <username>db_username</username> <password>db_password</password> <driverClassName>com.mysql.jdbc.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource>

7. Restart BAM server by running <BAM_HOME>/bin/wso2server.[sh/bat].

1. Start the API Manager and log in to its Admin Dashboard Web application (https://<Server Host>:9443/admin-dashboard).

2. Click the Configure Analytics menu.   
   

3. Select the Enable check-box to enable statistical data publishing and do the rest of the configurations using the information given below:

   Open

   

4. Click Save when you are done. It deploys the Analytics toolbox, which describes the information collected, how to analyze the data, and the location of the database where the analyzed data is stored.

   You change the stream names, versions, publisher class by editing the <APIM_HOME>/repository/conf/api-manager.xml file as given in the example below:

   <APIUsageTracking> <!-- Below property is used to skip trying to connect to event receiver nodes when publishing events even if the stats enabled flag is set to true. --> <SkipEventReceiverConnection>false</SkipEventReceiverConnection>   <!-- API Usage Data Publisher. --> <PublisherClass>org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher</PublisherClass> <!-- If below property set to true,then the response message size will be calculated and publish with each successful API invocation event. --> <PublishResponseMessageSize>false</PublishResponseMessageSize> <!-- Data publishing stream names and versions of API requests, responses and faults. If the default values are changed, the toolbox also needs to be changed accordingly. --> <Streams> <Request> <Name>org.wso2.apimgt.statistics.request</Name> <Version>1.0.0</Version> </Request> <Response> <Name>org.wso2.apimgt.statistics.response</Name> <Version>1.0.0</Version> </Response> <Fault> <Name>org.wso2.apimgt.statistics.fault</Name> <Version>1.0.0</Version> </Fault> <Destination> <Name>org_wso2_apimgt_statistics_destination</Name> <Version>1.0.0</Version> <BAMProfileName>bam-profile</BAMProfileName> </Destination> <Throttle> <Name>org.wso2.apimgt.statistics.throttle</Name> <Version>1.0.0</Version> </Throttle> <Workflow> <Name>org.wso2.apimgt.statistics.workflow</Name> <Version>1.0.0</Version> </Workflow> </Streams> </APIUsageTracking>

After configuring WSO2 BAM to collect and analyze statistics of APIs hosted and managed by the API Manager, you can view them through various statistical dashboards in the API Publisher and Store, depending on your permission levels. For information, see Viewing API Statistics.

To use a different database than the default H2 for statistical publishing, you change the properties of the datasource element, and additionally delete some metadata tables created by previous executions of the Hive script, if there are any.

To delete the metadata tables,

1. Log in to BAM management console and select Add in Analytics menu.

2. Go to the Script Editor in the window that opens.

3. Execute the following script.

   drop TABLE APIRequestData; drop TABLE APIRequestSummaryData; drop TABLE APIVersionUsageSummaryData; drop TABLE APIResourcePathUsageSummaryData; drop TABLE APIResponseData; drop TABLE APIResponseSummaryData; drop TABLE APIFaultData; drop TABLE APIFaultSummaryData; drop TABLE APIDestinationData; drop TABLE APIDestinationDataSummaryData; drop TABLE APIRequestDataMinimal; drop TABLE APIRequestSummaryDataMinimal; drop TABLE APIThrottleData; drop TABLE APIThrottleSummaryData;

4. If there are previous executions of the Hive scripts, manually execute them again by going to Main > Analytics > List in the management console of BAM. Alternatively, you can wait until the periodical execution time occurs.

The following are the summarized tables that exist in API Manager.

This table contains the summarized data of the API destinations and is derived from the destination event stream. 

This table contains the summarized data of faulty API invocations and is derived from the fault event stream.

This table contains the summary data of the request event stream.

This table contains the summarized data for API Manager usage by resources and is derived from the request event table.

This table contains the summarized data from API responses. It is derived from the response event table.

This table contains the summary data for API Manager usage. It is also derived from the request event table.

This table contains the summary of the throttle out API invocation data. It is derived from the throttle out event table and request table.