Introducing the WSO2 API Manager Statistics Model

Introduction

This section describes and illustrates the API Manager statistic publishing and generating model. It describes the internal components of API Manager, external analyzer information and other data retrieval components. API Manager generates events based on the API Manager invocation pattern and publishes them to all the listening event analyzers. The analyzer is responsible for the accumulation of all events and generates summary data based on the defined summarisation logic (C-App). After the summarized data is generated, the API Manager Dashboard can retrieve statistics from the analyzer data-source to the UI via the API Manager analytics client.

API Manager usage publisher

The internal API Manager component listens to the API Manager invocations and its behavior. Based on the request and responses, the event is generated and published to all the event receivers. This publisher publishes the following event streams,

• org.wso2.apimgt.statistics.request

• org.wso2.apimgt.statistics.response

• org.wso2.apimgt.statistics.fault

• org.wso2.apimgt.statistics.throttle

BAM mediator

The BAM mediator publishes the org_wso2_apimgt_statistics_destination event stream.

Workflow executor

This component publishes the org.wso2.apimgt.statistics.workflow event stream containing work-flow related event data.

API Manager usage client

The usage client is responsible for retrieving statistics data from the available data source. There can be more than one usage client in API Manager. By default, there are two usage clients and either one can be in use at a given time. From API-M 1.10.x onwards, two usage client exist; REST and RDBMS. You can configure and use either client. By default, the REST client is configured and uses the WSO2 DAS REST API to fetch summarized data.

In the diagram above, the paths indicated in black demonstrate the path when API Manager is configured with the REST client. The path in red demonstrates the RDBMS client data flow path. In this case, the same data store is stored in the DAS internal table into the external RDBMS. These two clients use different summary data generating plans defined in the API_Manager_Analytics_RDBMS.car and API_Manager_Analytics_REST.car files.

Data analyzer

The data analyzer is a summary data generator based on the received event. WSO2 API-M 1.10.x uses WSO2 DAS 3.0.x to perform analytics tasks. DAS uses Apache Spark as a processing language. API Manager has a pluggable component, DAS C-App, which can be deployed into DAS. It contains details of all summarized data and their destination. API Manager has two in-built C-App files; API_Manager_Analytics_RDBMS.car and API_Manager_Analytics_REST.car. Both contain the same summarization logic except for the REST client final destination, which is the DAS internal database and the RDBMS client final destination, which is the external RDBMS.

API Manager event streams

API-M 1.10.x provides six types of event streams as listed below.

1. org.wso2.apimgt.statistics.request 

   This stream tracks information for the API request.

   org.wso2.apimgt.statistics.request

   consumerKey     :       STRING			: 			Consumer key of API invoked client application	
   context         :       STRING			: 			API context depending on the user's request
   api_version     :       STRING			: 			API synapse artifact contained name [API Provider +"--"+API Name]
   api             :       STRING			: 			API Name
   resourcePath    :       STRING			: 			API resource URL pattern of API request
   method          :       STRING			: 			HTTP Verb of API request [e.g.:GET/POST]
   version         :       STRING			: 			API version
   request         :       INT				: 			Request count (e.g. 1)
   requestTime     :       LONG			: 			API request hit time in APIM
   userId          :       STRING			: 			API invoked end user name
   tenantDomain    :       STRING			: 			Tenant domain of API provider
   host            :       STRING			: 			API Manager server host
   apiPublisher    :       STRING			: 			API provider
   application     :       STRING			: 			Name of the client application
   applicationId   :       STRING			: 			ID of the client application
   userAgent       :       STRING			: 			User agent of the user
   tier            :       STRING			: 			Name of the throttling policy assigned to the request	
   throttledOut    :       BOOL			: 			Describes whether this is a throttled request or not

2. org.wso2.apimgt.statistics.response

   This stream tracks information for the API response. It includes the time taken for the response to get back, the received time etc.
    
   org.wso2.apimgt.statistics.response

   consumerKey     :       STRING			: 			Consumer key of the client application invoking the API
   context         :       STRING			: 			API context depending on the user's request
   api_version     :       STRING			: 			API synapse artifact contained name [API Provider +"--"+API Name]	
   api             :       STRING			: 			API Name
   resourcePath    :       STRING			: 			API resource URL pattern of API request
   method          :       STRING			: 			HTTP Verb of API request [e.g.:GET/POST]
   version         :       STRING			: 			API version
   response        :       INT				: 			Response count (e.g. 1)
   responseTime    :       LONG			: 			Total time taken for request/response flow[serviceTime+backendTime]
   serviceTime     :       LONG			: 			Time taken to serve the API request in APIM side
   backendTime     :       LONG			: 			Time taken process the request at the backend	
   user            :       STRING			: 			API invoked end user	
   eventTime       :       LONG			: 			Timestamp of response event published
   tenantDomain    :       STRING			: 			Tenant domain of API provider
   host            :       STRING			: 			API Manager server hostname
   apiPublisher    :       STRING			: 			API provider
   application     :       STRING			: 			Name of the client application
   applicationId   :       STRING			: 			ID of the client application
   cacheHit        :       BOOL			: 			Describes if response caching is enabled or not
   responseSize    :       LONG			: 			Response message size in bytes	
   protocol        :       STRING 			: 			Protocol used to send the response (HTTP/HTTPS) and the port

3. org.wso2.apimgt.statistics.fault

   This stream contains the fault API invocations. It includes the API with back end errors, timeout etc
    
   org.wso2.apimgt.statistics.fault

   consumerKey     :       STRING			: 			Consumer key of the client application invoking the API	
   context         :       STRING			: 			API context depending on the user's request	
   api_version     :       STRING			: 			API version
   api             :       STRING			: 			API Name
   resourcePath    :       STRING			: 			API resource url pattern of API request
   method          :       STRING			: 			HTTP Verb of API request [e.g.:GET/POST]
   version         :       STRING			: 			API version
   errorCode       :       STRING			: 			HTTP error code
   errorMessage    :       STRING			: 			Description of error message
   requestTime     :       LONG			: 			API request time in millisecond
   userId          :       STRING			: 			API invoked end user name
   tenantDomain    :       STRING			: 			Tenant domain of API provider
   host            :       STRING			: 			API Manager server host
   apiPublisher    :       STRING     		: 			API provider
   application     :       STRING			: 			Name of the client application
   applicationId   :       STRING			: 			ID of the client application 
   protocol        :       STRING			: 			Protocol used to send the response (HTTP/HTTPS) and the port

4. org_wso2_apimgt_statistics_destination

   This stream tacks related information based on the endpoint URL.

   org_wso2_apimgt_statistics_destination

   tenant_id           :       INT				: 			ID of the tenant user
   http_method         :       STRING			: 			HTTP Verb of API request [e.g.:GET/POST]
   character_set_encoding:     STRING			: 			Character set encoding e.g. :UTF-8
   remote_address      :       STRING			: 			IP address of API invoked client
   transport_in_url    :       STRING			: 			API Context + resource path
   message_            :       STRING			: 			Content type of API request
   remote_host         :       STRING			: 			Host address of API request e.g.: application/xml
   service_prefix      :       STRING			: 			Host + port of the API request  
   host                :       STRING			: 			API Manager server host
   correlation_activity_id: 	STRING			:			Activity ID which comes with message tracing feature
   message_direction   :       STRING			: 			Direction of the message e.g.: IN/OUT
   operation_          :       STRING			: 			Operation name
   message_id          :       STRING			: 			ID of the message e.g. : uuid:41c72f3a-fe7e-4486-a704-133414cc63e9	
   timestamp           :       LONG			: 			Record insert time for table in DAS side
   api                 :       STRING			: 			API Name
   version             :       STRING			: 			API version
   request             :       INT				: 			Number of API requests(request count)
   apiPublisher        :       STRING			: 			API provider
   context             :       STRING			: 			API context
   destination         :       STRING			: 			API endpoint hostname
   requestTime         :       LONG 			: 			API request time in millisecond
   host                :       STRING			: 			APIM server host

5. org.wso2.apimgt.statistics.throttle

   This stream contains the API invocation with throttle information. Throttling can happen due to any of the following reasons:

   The application limit has exceeded.
   The resource limit has exceeded.
   The API limit has exceeded.
   The hard level limit has exceeded.
    

   org.wso2.apimgt.statistics.throttle

   accessToken         :       STRING			: 			Access token of the request
   userId              :       STRING			: 			API invoked end user name
   tenantDomain        :       STRING			: 			Tenant domain of API provider
   api                 :       STRING			: 			API Name
   api_version         :       STRING			: 			API synapse artifact contained name [API Provider +"--"+API Name]
   context             :       STRING			: 			API context depending on the user's request
   apiPublisher        :       STRING			: 			API provider
   throttledTime       :       LONG			: 			The timestamp which throttle out event triggers
   application         :       STRING			: 			Name of the client application
   applicationId       :       STRING			: 			ID of the client application
   throttledOutReason  :       STRING 			: 			The reason describing why the request has been throttled out

6. org.wso2.apimgt.statistics.workflow

   This event stream creates events based on the API Manager workflow and publishes them to the analyzer.
    
   org.wso2.apimgt.statistics.workflow

   workflowReference   :       STRING			: 			Holds the workflow reference ID	
   workflowStatus      :       STRING			: 			Status of the workflow e.g.: CREATED, APPROVED, REJECTED, REGISTERED
   tenantDomain        :       STRING			: 			Tenant domain of subscriber who triggers the workflow in APIStore
   workflow            :       STRING			: 			Type of the workflow e.g.:AM_APPLICATION_CREATION,
   createdTime         :       LONG			: 			The workflow was creation time in milliseconds
   updatedTime         :       LONG			: 			The last updated time of the workflow in milliseconds

API Manager summarized tables (DAS internal storage)

Note that these summarized tables are stored in the DAS internal storage. Both the C-Apps that API Manager deploy on DAS first stores the summary data into these tables. There are additional columns in some of these tables containing the composition column of other columns. Those columns types are of facet type, which is used to support the aggregation function on the DAS REST API. All the columns are indexed in order to search using Apache Lucene and supports the DAS REST API. When API Manager is configured with the RDBMS client, all these tables are replicated in the external RDBMS except for the facet columns.

1. • API_REQUEST_SUMMARY 

     This table contains the summary data of the request event stream.
     API_REQUEST_SUMMARY table schema

     CREATE TEMPORARY TABLE API_REQUEST_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_REQUEST_SUMMARY",
       schema "api string -i,
       api_version string -i,
       version string -i,
       apiPublisher string -i,
       consumerKey string -i,
       userId string -i,
       context string -i,
       max_request_time long -i,
       total_request_count int -i,
       hostName string -i,
       year int -i,
       month int -i,
       day int -i,
       time string -i,
       key_api_facet facet -i,
       key_userId_facet facet -i,
       api_version_userId_facet facet -i,
       api_version_userId_apiPublisher_facet facet -i,
       api_version_userId_context_facet facet -i",
       primaryKeys "api,api_version,version,apiPublisher,consumerKey,userId,context,hostName,year,month,day"
       );

      Expand to find the table of descriptions for each column

     api				: API Name
     api_version		: API synapse artifact contained name [API Provider +"--"+API Name]
     version			: API version
     apiPublisher	: API provider
     context			: API context depending on the user's request	
     consumerKey		: Consumer key of the client application invoking the API
     userId			: End user name invoked by the API
     max_request_time: Time of the latest API request occurrence
     total_request_count: Total request count for the requests coming for same API
     hostname		: APIM server hostname
     year			: The year of initial API request occurred of the batch of API requests
     month			: The month of initial API request occurred of the batch of API requests
     day				: The date of API initial request occurred of the batch of API requests
     time			: The time of API initial request occurred of the batch of API requests

   • API_VERSION_USAGE_SUMMARY

     This table contains the summary data for API Manager usage. It is also derived from the request event table.

     VERSION_USAGE_SUMMARY table schema

     CREATE TEMPORARY TABLE API_VERSION_USAGE_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_VERSION_USAGE_SUMMARY",
       schema "api string -i,
       version string -i,
       apiPublisher string -i,
       context string -i,
       total_request_count int -i,
       hostName string -i,
       year int -i,
       month int -i,
       day int -i,
       time string -i,
       max_request_time long -i,
       api_version_context_facet facet -i",
       primaryKeys "api,version,apiPublisher,context,hostName,year,month,day"
       );

      Expand to find the table of descriptions for each column

     api				: API Name
     version			: API version
     apiPublisher	: API provider
     context			: API context depending on the user's request	
     total_request_count: Total request count of an API version
     hostname		: APIM server hostname
     year			: The year of initial API request occurred of the batch of API requests
     month			: The month of initial API request occurred of the batch of API requests
     day				: The date of API initial request occurred of the batch of API requests
     time			: The time of API initial request occurred of the batch of API requests

   • API_Resource_USAGE_SUMMARY

     This table contains the summarized data for API Manager usage by resources and it is also derived from request event table.

     API_Resource_USAGE_SUMMARY table schema

       CREATE TEMPORARY TABLE API_Resource_USAGE_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_Resource_USAGE_SUMMARY",
       schema "api string -i,
       version string -i,
       apiPublisher string -i,
       consumerKey string -i,
       resourcePath string -i,
       context string -i,
       method string -i,
       total_request_count int -i,
       hostName string -i,
       year int -i,
       month int -i,
       day int -i,
       time string -i,
       max_request_time long -i,
       key_api_method_path_facet facet -i,
       api_version_context_method_facet facet -i",
       primaryKeys "api,version,apiPublisher,consumerKey,context,resourcePath,method,hostName,year,month,day"
       );

      Expand to find the table of descriptions for each column

     api				: API Name
     version			: API version
     apiPublisher	: API provider
     context			: API context depending on the user's request
     consumerKey		: Consumer key of the client application invoking the API	
     resourcePath	: API resource url pattern of API request
     method			: HTTP Verb of API request [eg:GET/POST]
     total_request_count :Total request count for a particular API resource pattern
     hostname		: API Manager server hostname
     year			: The year of initial API request occurred of the batch of API requests
     month			: The month of initial API request occurred of the batch of API requests
     day				: The date of API initial request occurred of the batch of API requests
     time			: The time of API initial request occurred of the batch of API requests

   • API_RESPONSE_SUMMARY

     This table contains the summarized data from API responses. It is derived from the response event table.

     API_RESPONSE_SUMMARY table schema

       CREATE TEMPORARY TABLE API_RESPONSE_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_RESPONSE_SUMMARY",
       schema "api_version string -i,
       apiPublisher string -i,
       context string -i,
       serviceTime int -i,
       total_response_count int -i,
       hostName string -i,
       year int -i,
       month int -i,
       day int -i,
       time string -i,
       max_request_time long -i,
       api_version_context_facet facet -i",
       primaryKeys "api,version,apiPublisher,context,hostName,year,month,day"
       );

      Expand to find the table of descriptions for each column

     api_version			: API synapse artifact contained name [API Provider +"--"+API Name]
     apiPublisher		: API provider
     context				: API context depending on the user's request
     consumerKey			: Consumer key of the client application invoking the API	
     serviceTime			: Total time taken to serve the batch of API requests in APIM side
     total_response_count: Total response count for the API requests for a specific API 
     hostname			: API Manager server hostname
     year				: The year of initial API request occurred of the batch of API requests
     month				: The month of initial API request occurred of the batch of API requests
     day					: The date of API initial request occurred of the batch of API requests
     time				: The time of API initial request occurred of the batch of API requests

   • API_FAULT_SUMMARY

     This table contains the summarized data of faulty API invocations and is derived from the fault event stream.

     API_FAULT_SUMMARY table schema

       CREATE TEMPORARY TABLE API_FAULT_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_FAULT_SUMMARY",
       schema "api string -i,
       version string -i,
       apiPublisher string -i,
       consumerKey string -i,
       context string -i,
       total_fault_count int -i,
       hostName string -i,
       year int -i,
       month int -i,
       day int -i,
       time string -i,
       max_request_time long -i,
       consumerKey_api_facet facet -i,
       api_version_apiPublisher_context_facet facet -i",
       primaryKeys "api,version,apiPublisher,consumerKey,context,hostName,year,month,day"
       );

      Expand to find the table of descriptions for each column

     api				: API Name
     version			: API version
     apiPublisher	: API provider
     consumerKey		: Consumer key of the client application invoking the API	
     context			: API context depending on the user's request
     total_fault_count: Total faulty API request count for a specific API 
     hostname		: APIM server hostname
     year			: The year of initial API request occurred of the batch of API requests
     month			: The month of initial API request occurred of the batch of API requests
     day				: The date of API initial request occurred of the batch of API requests
     time			: The time of API initial request occurred of the batch of API requests

   • API_DESTINATION_SUMMARY

     This table contains the summarized data of the API destinations and is derived from the destination event stream.

     API_DESTINATION_SUMMARY table schema

       CREATE TEMPORARY TABLE API_DESTINATION_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_DESTINATION_SUMMARY",
       schema "api string -i,
       version string -i,
       apiPublisher string -i,
       context string -i,
       destination string -i,
       total_request_count int -i,
       hostName string -i,
       year int -i,
       month int -i,
       day int -i,
       time string -i,
       max_request_time long -i,
       api_version_context_dest_facet facet -i",
       primaryKeys "api,version,apiPublisher,context,destination,hostName,year,month,day"
       );

      Expand to find the table of descriptions for each column

     api					: API Name
     version				: API version
     apiPublisher		: API provider
     context				: API context depending on the user's request
     destination			: API endpoint hostname
     total_request_count	: Total request count for the requests coming for same API to same destination
     hostname			: APIM server hostname
     year				: The year of initial API request occurred of the batch of API requests
     month				: The month of initial API request occurred of the batch of API requests
     day					: The date of API initial request occurred of the batch of API requests
     time				: The time of API initial request occurred of the batch of API requests

   • API_LAST_ACCESS_TIME_SUMMARY

     This table contains the summary data of the last access times of the API and is derived from the request event stream.

     API_LAST_ACCESS_TIME_SUMMARY table schema

       CREATE TEMPORARY TABLE API_LAST_ACCESS_TIME_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_LAST_ACCESS_TIME_SUMMARY",
       schema "tenantDomain string -i,
       apiPublisher string -i,
       api string -i,
       version string -i,
       userId string -i,
       context string -i,
       max_request_time long -i",
       primaryKeys "tenantDomain,apiPublisher,api"
       );

      Expand to find the table of descriptions for each column

     api				: API Name
     version			: API version
     apiPublisher	: API provider
     context			: API context depending on the user's request
     userId			: API invoked end user name
     tenantDomain	: Tenant domain of API provider
     max_request_time: Time of the latest API request occurrence

   • API_THROTTLED_OUT_SUMMARY

     This table contains the summary of the throttle out API invocation data. It is derived from the throttle out event table and request table.

     API_THROTTLED_OUT_SUMMARY table schema

     CREATE TEMPORARY TABLE THROTTLED_OUT_FINAL_SUMMARY USING CarbonAnalytics OPTIONS (tableName "API_THROTTLED_OUT_SUMMARY",
       schema "api string -i,
       api_version string -i,
       context string -i,
       apiPublisher string -i,
       applicationName string -i,
       tenantDomain string -i,
       year int -i,
       month int -i,
       day int -i,
       week int -i,
       time string -i,
       success_request_count int -i,
       throttleout_count int -i,
       max_request_time long -i,
       api_year_month_week_day_facet facet -i,
       applicationName_facet facet -i",
       primaryKeys "api,api_version,context,apiPublisher,applicationName,tenantDomain,year,month,day"
     );

      Expand to find the table of descriptions for each column

     api					: API Name
     api_version			: API synapse artifact contained name [API Provider +"--"+API Name]
     apiPublisher		: API provider
     context				: API context depending on the user's request
     applicationName		: Name of the client application
     tenantDomain		: Tenant domain of API provider
     throttleout_count	: Total throttled out API request count for the particular API
     success_request_count: Total successful API request count for the particular API
     year				: The year of initial API request occurred of the batch of API requests
     month				: The month of initial API request occurred of the batch of API requests
     day					: The date of API initial request occurred of the batch of API requests
     time				: The time of API initial request occurred of the batch of API requests

API Manager summarized tables (RDBMS)

You can view the summarized tables that exist in API Manager in the <APIM_HOME>/dbscripts/stat/sql/mysql.sql file. 

API Manager statistics

API statistics are provided in both the API Publisher and the API Store. Apart from the number of subscriptions per API, all other statistical dashboards require an instance of WSO2 Data Analytics Server installed. For information on the available statistics and how to view them, see Viewing API Statistics.