Enabling Metrics and Storage Types

Given below are the configurations that should be in place for your API Manager server in order to use the metrics feature. You need to first enable metrics for your server and then enable the required storage types (reporters) that are used for storing the metrics data. See the following topics for instructions:

Enabling metrics

To enable metrics for your product, set the Enabled parameter under the Metrics element to true in the <APIM_HOME>/repository/conf/metrics.xml file. Alternatively, you can enable metrics at the time of starting the API Manager server by using the following command:

-Dmetrics.enabled=true

Configuring the storage of metrics

WSO2 API Manager is configured by default to store the information from metrics in the following reporters: JMX, CSV and JDBC. These reporters are configured in the metrics.xml file (stored in the <APIM_HOME>/repository/conf directory). You can disable metrics for individual reporters by setting the Enabled parameter to false.

If you set the the Enabled parameter under the metrics element to false in the metrics.xml file, metrics is disabled for all the reporters and it is not possible to enable metrics for individual reporters.

See the following topics for information on configuring each of the available storage types.

JMX

The following parameters in the metrics.xml file can be used to configure a JMX storage for metrics data.

Element Name	Description	Type	Default Value	Mandatory/Optional
Enabled	This parameter specifies whether metics monitoring is enabled for JMX or not.	Boolean	true	Mandatory

CSV

The following parameters in the metrics.xml file can be used to configure a CSV storage for metrics data.

Element Name	Description	Type	Default Value	Mandatory/Optional
Enabled	This parameter specifies whether metrics monitoring is enabled for CSV or not.	Boolean	false	Mandatory
Location	The location where the CSV files are stored.	String	${carbon.home}/repository/logs/metrics/
PollingPeriod	The time interval between polling activities that are carried out to update the metrics dashboard based on latest information. For example, if the polling period is 60, polling would be carried out every 60 milliseconds.	Integer	60

JDBC

The following parameters in the metrics.xml file can be used to configure a JDBC storage for metrics data.

Element Name	Description	Type	Default Value	Mandatory/Optional	Notes
Enabled	This parameter specifies whether metrics monitoring is enabled for JDBC or not. If you need to use metrics in WSO2 API-M, change this value to `true`. For more information, see Creating the datasource connection for the Metrics database.	Boolean	`false`	Mandatory
DataSourceName	The name of the datasource used.	String	`jdbc/WSO2MetricsDB`
PollingPeriod	The time interval between polling activities that are carried out to update the metrics dashboard based on latest information. For example, if the polling period is 60 , polling would be carried out every 60 seconds.	Integer	60		This value is specified in seconds.
ScheduledCleanup	This element contains parameters relating to scheduled cleanup. The possible values are `Enabled`, `ScheduledCleanupPeriod` and `DaysToKeep`. Scheduled cleanup involves scheduling a task to clear metric data in the database after a specified time interval. This is done to avoid excessive memory usage.
ScheduledCleanup/Enabled	This parameter specifies whether scheduled cleanup is enabled or not.	Boolean	true
ScheduledCleanup/ScheduledCleanupPeriod	The number of seconds that should elapse after a cleanup task before the next clean-up task is carried out.	Integer	86400
ScheduledCleanup/DaysToKeep	The number of days during which the scheduled clean-up task should be carried out.	Integer	7

If you have enabled JDBC, then you also need to specify a datasource configuration to be used to create the connection between WSO2 API Manager and the JDBC data storage system. The metrics-datasources.xml file is used for configuring this datasource for metrics.

Parameters that can be configured for a datasource are as follows:

XML element	Attribute	Description	Data type	Default value	Mandatory/Optional
`<datasources-configuration>`	`xmlns`	The root element. The namespace is specified as: `xmlns:svns="http://org.wso2.securevault/configuration"`			Mandatory
`<providers>`		The container element for the datasource providers.			Mandatory
`<provider>`		The datasource provider, which should implement `org.wso2.carbon.ndatasource.common .spi.DataSourceReader`. The datasources follow a pluggable model in providing datasource type implementations using this approach.	Fully qualified Java class		Optional
`<datasources>`		The container element for the datasources.			Mandatory
`<datasource>`		The root element of a datasource.			Mandatory
`<name>`		Name of the datasource.	String		Mandatory
`<description>`		Description of the datasource.	String		Optional
`<jndiConfig>`		The container element that allows you to expose this datasource as a JNDI datasource.			Optional
`<name>`		The JNDI resource name to which this datasource should be bound.	String		Mandatory if specifying JNDI configuration
`<environment>`		The container element in which you specify the following JNDI properties: `java.naming.factory.initial`: Selects the registry service provider as the initial context. `java.naming.provider.url`: Specifies the location of the registry when the registry is being used as the initial context.	Fully qualified Java class		Optional
`<definition>`	`type`	The container element for the data source definition. Set the type attribute to RDBMS, or to custom if you're creating a custom type. The "RDBMS" data source reader expects a "configuration" element with the sub-elements listed below.	String		Mandatory
`<configuration>`		The container element for the RDBMS properties.			Mandatory if definition type is RDBMS
`<url>`		The connection URL to pass to the JDBC driver to establish the connection.	URL		Mandatory
`<username>`		The connection user name to pass to the JDBC driver to establish the connection.	String		Optional
`<password>`		The connection password to pass to the JDBC driver to establish the connection.	String		Optional
`<driverClassName>`		The class name of the JDBC driver to use.	Fully qualified Java class		Mandatory
`<maxActive>`		The maximum number of active connections that can be allocated from this pool at the same time.	Integer	100	Optional
`<maxWait>`		Maximum number of milliseconds that the pool waits (when there are no available connections) for a connection to be returned before throwing an exception.	Integer	30000 (30 seconds)	Optional
`<testOnBorrow>`		Specifies whether objects are validated before being borrowed from the pool. If the object fails to validate, it is dropped from the pool, and we will attempt to borrow another. When set to true, the `validationQuery` parameter must be set to a non-null string.	Boolean	false	Optional
`<validationQuery>`		The SQL query used to validate connections from this pool before returning them to the caller. If specified, this query does not have to return any data, it just can't throw a SQLException. The default value is null. Example values are SELECT 1(mysql), select 1 from dual(oracle), SELECT 1(MS Sql Server).	String	null	Mandatory when `testOnBorrow` is set to true
`<validationInterval>`		To avoid excess validation, only run validation at most at this frequency (interval time in milliseconds). If a connection is due for validation, but has been validated previously within this interval, it will not be validated again. The default value is 30000 (30 seconds).	Long	30000 (30 seconds)	Optional

Sample configuration

Shown below is a sample metrics.xml file with the default configurations specifying the types of storages enabled for metrics data. See the above topics for instructions.

The default configurations in the metrics.xml file

-->

<!--
    This is the main configuration file for metrics
-->

<Metrics xmlns="http://wso2.org/projects/carbon/metrics.xml">

    <!--
        Enable Metrics
    -->
    <Enabled>false</Enabled>
    <!--
        Metrics reporting configurations
    -->

    <Reporting>
        <JMX>
            <Enabled>true</Enabled>
        </JMX>
        <CSV>
            <Enabled>false</Enabled>
            <Location>${carbon.home}/repository/logs/metrics/</Location>
            <!-- Polling Period in seconds -->
            <PollingPeriod>60</PollingPeriod>
        </CSV>
        <JDBC>
            <Enabled>true</Enabled>
            <!-- Source of Metrics, which will be used to
                identify each metric in database -->
            <!-- Commented to use the hostname
                <Source>Carbon</Source>
            -->
            <!--
                JNDI name of the data source to be used by the JDBC Reporter.
                This data source should be defined in a *-datasources.xml
                file in conf/datasources directory.
            -->
            <DataSourceName>jdbc/WSO2MetricsDB</DataSourceName>
            <!-- Polling Period in seconds -->
            <PollingPeriod>60</PollingPeriod>
            <ScheduledCleanup>
                <!--
                    Schedule regular deletion of metrics data older than a set number of days.
                    It is strongly recommended that you enable this job to ensure your metrics tables do not get extremely
                    large. Deleting data older than seven days should be sufficient.
                -->
                <Enabled>true</Enabled>
                <!-- This is the period for each cleanup operation in seconds -->
                <ScheduledCleanupPeriod>86400</ScheduledCleanupPeriod>
                <!-- The scheduled job will cleanup all data older than the specified days -->
                <DaysToKeep>7</DaysToKeep>
            </ScheduledCleanup>
        </JDBC>
    </Reporting>
</Metrics>

Changing the default metrics database

Follow the instructions below to change the deault metrics database.

Step 1 - Set up the database
Step 2 - Create the datasource connection for the Metrics database
Step 3 - Create database tables in the Metrics database

Step 1 - Set up the database

You can set up the following database types for the API-M-specific databases:

Note that we recommend to use Fail Over configuration over Load Balanced configuration with the MySQL clusters.

Step 2 - Create the datasource connection for the Metrics database

A datasource is used to establish the connection to a database. By default, the datasource connection for the Metrics database is configured in the metrics-datasources.xml file. This datasource configuration points to the default H2 databases, which is shipped with the product. After setting up new databases to replace the default H2 databases, you can either change the default configurations in the above-mentioned files or configure new datasources.

Follow the steps below to create the datasource connection for the Metrics database:

Open the <API-M _HOME>/repository/conf/datasources/metrics -datasources.xml file and locate the <datasource> configuration element.

Update the URL pointing to you database, the username and password required to access the database, and the driver details as shown below.

Optionally, you can update the other elements for your database connection.

Element	Description
`maxActive`	The maximum number of active connections that can be allocated at the same time from this pool. Enter any negative value to denote an unlimited number of active connections.
`maxWait`	The maximum number of milliseconds that the pool waits (when there are no available connections) for a connection to be returned before throwing an exception. You can enter zero or a negative value to wait indefinitely.
`minIdle`	The minimum number of active connections that can remain idle in the pool without extra ones being created. You can enter zero to create none.
`testOnBorrow`	The indication of whether objects are validated before being borrowed from the pool. If the object fails to validate, it is dropped from the pool, and another attempt is made to borrow another.
`validationQuery`	The SQL query that is used to validate connections from this pool before returning them to the caller.
`validationInterval`	The indication to avoid excess validation, and only run validation at the most, at this frequency (time in milliseconds). If a connection is due for validation but has been validated previously within this interval, it is not validated again.
`defaultAutoCommit`	This property is only applicable to the MB Store database of WSO2 APIM, where this property should be explicitly set to `false`. In all other database connections explained above, auto committing is enabled or disabled at the code level as required for that database, i.e., the default auto commit configuration specified for the RDBMS driver will be effective instead of this property element. Note that auto committing is typically enabled for an RDBMS by default. When auto committing is enabled, each SQL statement will be committed to the database as an individual transaction, as opposed to committing multiple statements as a single transaction.

For more information on other parameters that can be defined in the <API-M_HOME>/conf/datasources/ master-datasources.xml file, see Tomcat JDBC Connection Pool.

Step 3 - Create database tables in the Metrics database

To create the database tables, connect to the databases that you created earlier and run the scripts provided in the product pack. The DB scripts corresponding to the database type are provided in the <API-M_HOME>/dbscripts/metrics directory.

To create the necessary database tables:

Connect to the database and run the relevant script.
For example, run the following command to create the MB tables in a MySQL database.
```
mysql -u root -p -DWSO2_METRICS_DB < '<API-M_HOME>/dbscripts/metrics/mysql.sql';
```
<API-M_HOME>/dbscripts/metrics/mysql.sql is the script that should be used for MySQL 5.6 and prior versions. If you database is MySQL 5.7 or later version, use <API-M_HOME>/dbscripts/metrics/mysql5.7.sql script file.
Restart the WSO2 API-M server.