Enabling Analytics for Australia

This document describes how you can enable analytics using WSO2 Open Banking Business Intelligence.

The WSO2 Open Banking Business Intelligence(WSO2 OB BI) component enables monitoring and recording of API-level usage activity to ensure that the API owners have full awareness of the APIs, applications, and subscriptions. It generates alerts by analyzing the risk involved in transactions and fraudulent activities. It also generates statistics using the data collected from the WSO2 Open Banking Identity and Access Management and API Management modules. WSO2 Open Banking Business Intelligence contains the following features: 

API Analytics

Provides reports, statistics and graphs on the APIs deployed in the WSO2 Open Banking API Management modules. Helps the CDR Register to monitor how the banking APIs perform.

Data Reporting

The WSO2 Open Banking Business Intelligence captures data published in the WSO2 Open Banking Identity and Access Management and WSO2 Open Banking API Management modules. Data Reporting is the process of processing and summarizing these data. 

The placement of the WSO2 OB BI component is as follows:

Configuring databases

1. If you are using MSSQL or Oracle, create the following databases:

   If you are using a mysql database environment, the databases are created automatically when you execute the following scripts:

   • <WSO2_OB_IAM_HOME>/repository/resources/finance/scripts/configure-iam.sh
   • <WSO2_OB_APIM_HOME>/repository/resources/finance/scripts/configure-am.sh

   • openbank_apimgt_statsdb 

   • openbank_openbankingdb 

   • openbank_geolocationdb

   • openbank_ob_reporting_statsdb


   • openbank_ob_reporting_summarizeddb


2. Update the  <WSO2_OB_BI_HOME>/conf/worker/deployment.yaml  file. Modify the datasources below by pointing them to the databases you created in the above step. 

   • APIM_ANALYTICS_DB

   • OPEN_BANKING_DB


   • GEO_LOCATION_DATA

   • OB_REPORTING_DB  

   • OB_REPORTING_SUMMARIZED_DB

   Update the datasource entry in the deployment.yaml with your database configurations.

   Property	Description
   jdbcUrl	This is the port to connect to on database server The jdbcUrl property expects the JDBC driver connection string. A sample jdbcUrl: 'jdbc:mysql://localhost:3306/openbank_ob_reporting_summarizeddb?autoReconnect=true&useSSL=false'
   username	The database user
   password	Password set for the database connection
   driverClassName	The JDBC driver name Mysql JDBC Driver : com.mysql.jdbc.Driver MSSQL JDBC Driver : com.microsoft.sqlserver.jdbc.SQLServerDriver Oracle JDBC Driver : oracle.jdbc.driver.OracleDriver

   If you are using Oracle, update the connectionTestQuery in each datasource with the following value.

   connectionTestQuery: SELECT 1 FROM DUAL

       - name: GEO_LOCATION_DATA
         description: "The data source used for geo location database"
         jndiConfig:
           name: jdbc/GEO_LOCATION_DATA
         definition:
           type: RDBMS
           configuration:
             jdbcUrl: 'jdbc:mysql://localhost:3306/openbank_geolocationdb?autoReconnect=true&useSSL=false'
             username: <USERNAME>
             password: <PASSWORD>
             driverClassName: com.mysql.jdbc.Driver
             maxPoolSize: 50
             idleTimeout: 60000
             connectionTestQuery: SELECT 1
             # Use below for oracle database
             # connectionTestQuery: SELECT 1 FROM DUAL
             validationTimeout: 30000
             isAutoCommit: false
   
       - name: APIM_ANALYTICS_DB
         description: "The datasource used for APIM statistics aggregated data."
         jndiConfig:
           name: jdbc/APIM_ANALYTICS_DB
         definition:
           type: RDBMS
           configuration:
             jdbcUrl: 'jdbc:mysql://localhost:3306/openbank_apimgt_statsdb?autoReconnect=true&useSSL=false'
             username: <USERNAME>
             password: <PASSWORD>
             driverClassName: com.mysql.jdbc.Driver
             maxPoolSize: 50
             idleTimeout: 60000
             connectionTestQuery: SELECT 1
             validationTimeout: 30000
             isAutoCommit: false
   
       - name: OPEN_BANKING_DB
         description: "The datasource used to store payment consents"
         jndiConfig:
           name: jdbc/OPEN_BANKING_DB
         definition:
           type: RDBMS
           configuration:
             jdbcUrl: 'jdbc:mysql://localhost:3306/openbank_openbankingdb?autoReconnect=true&useSSL=false'
             username: <USERNAME>
             password: <PASSWORD>
             driverClassName: com.mysql.jdbc.Driver 
             maxPoolSize: 50
             idleTimeout: 60000
             connectionTestQuery: SELECT 1
             validationTimeout: 30000
             isAutoCommit: false
   
       - name: OB_REPORTING_DB
         description: The datasource used to store statistics for OB Reporting module
         jndiConfig:
           name: jdbc/OB_REPORTING_DB
         definition:
           type: RDBMS
           configuration:
             jdbcUrl: 'jdbc:mysql://localhost:3306/openbank_ob_reporting_statsdb?autoReconnect=true&useSSL=false'
             username: <USERNAME>
             password: <PASSWORD>
             driverClassName: com.mysql.jdbc.Driver 
             maxPoolSize: 20
             idleTimeout: 60000
             connectionTestQuery: SELECT 1
             validationTimeout: 30000
             isAutoCommit: false
   
       - name: OB_REPORTING_SUMMARIZED_DB
         description: The datasource used to store statistics for OB Reporting module
         jndiConfig:
           name: jdbc/OB_REPORTING_SUMMARIZED_DB
         definition:
           type: RDBMS
           configuration:
             jdbcUrl: 'jdbc:mysql://localhost:3306/openbank_ob_reporting_summarizeddb?autoReconnect=true&useSSL=false'
             username: <USERNAME>
             password: <PASSWORD>
             driverClassName: com.mysql.jdbc.Driver 
             maxPoolSize: 20
             idleTimeout: 60000
             connectionTestQuery: SELECT 1
             validationTimeout: 30000
             isAutoCommit: false       

3. Modify the datasources of OB_REPORTING_DB  in the <WSO2_OB_BI_HOME>/conf/dashboard/deployment.yaml file.
   Use your database configurations and update the datasource properties as follows: 

   #Main datasource used in API Manager
     - name: AM_DB
       description: Main datasource used by API Manager
       jndiConfig:
         name: jdbc/AM_DB
       definition:
         type: RDBMS
         configuration:
           jdbcUrl: "jdbc:mysql://localhost:3306/openbank_apimgtdb?autoReconnect=true&useSSL=false"
           username: <USERNAME>
           password: <PASSWORD>
           driverClassName: 'com.mysql.jdbc.Driver'
           maxPoolSize: 10
           idleTimeout: 60000
           connectionTestQuery: SELECT 1
           validationTimeout: 30000
           isAutoCommit: false    
   
     - name: OB_REPORTING_DB
         description: The datasource used to store statistics for OB Reporting module
         jndiConfig:
           name: jdbc/OB_REPORTING_DB
         definition:
           type: RDBMS
           configuration:
             jdbcUrl: 'jdbc:mysql://localhost:3306/openbank_ob_reporting_statsdb?autoReconnect=true&useSSL=false'
             username: <USERNAME>
             password: <PASSWORD>
             driverClassName: 'com.mysql.jdbc.Driver'
             maxPoolSize: 20
             idleTimeout: 60000
             connectionTestQuery: SELECT 1
             # Use below for oracle database
             # connectionTestQuery: SELECT 1 FROM DUAL
             validationTimeout: 30000
             isAutoCommit: false

   If you are using Oracle for databases,

   Add the following configuration to the APIM_ANALYTICS_DB and AM_DB datasources:

   • APIM_ANALYTICS_DB datasource in the <WSO2_OB_BI_HOME>/conf/worker/deployment.yaml and <WSO2_OB_BI_HOME>/conf/dashboard/deployment.yaml files  
   • AM_DB datasource in the < WSO2_OB_BI_HOME>/conf/dashboard/deployment.yaml files

   connectionInitSql: alter session set NLS_DATE_FORMAT='YYYY-MM-DD HH24:MI:SS'

   See the sample datasource configuration is given below:

   #Main datasource used in API Manager
     - name: AM_DB
       description: Main datasource used by API Manager
       jndiConfig:
         name: jdbc/AM_DB
       definition:
         type: RDBMS
         configuration:
           jdbcUrl: "jdbc:oracle:thin:@192.168.108.7:1521/ora12c"
           username: '<USERNAME>'
           password: '<PASSWORD>'
           driverClassName: 'oracle.jdbc.driver.OracleDriver'
           maxPoolSize: 10
           idleTimeout: 60000
           connectionTestQuery: SELECT 1 FROM DUAL
           connectionInitSql: alter session set NLS_DATE_FORMAT='YYYY-MM-DD HH24:MI:SS'
           validationTimeout: 30000
           isAutoCommit: false

   You need to follow the steps below when you are using Oracle or MS SQL database servers.

   You need to include the database driver corresponding to the database for the WSO2 Open Banking Business Intelligence server to communicate with the database. The WSO2 Open Banking Business Intelligence is an OSGi-based product. Therefore, when you integrate third party products such as Oracle and MS SQL with WSO2 OB BI, you need to check whether the libraries you need to add are OSGi based. If they are not, you need to convert them to OSGi bundles before adding them to the <WSO2_OB_BI_HOME>/lib directory.

   To convert the jar files to OSGi bundles, follow the steps below.

   1. Download the non-OSGi jar for the required third party product, and save it in a preferred directory in your machine.

   2. Go to the <WSO2_OB_BI_HOME>/bin directory. Run the command given below to generate the converted file in the <WSO2_OB_BI_HOME>/lib directory.

   ./jartobundle.sh <PATH_TO_NON-OSGi_JAR> ../lib

Configuring WSO2 OB BI worker 

1. Update listenerConfigurations in the <WSO2_OB_BI_HOME>/wso2/worker/conf/transports/netty-transports.yml as follows:

   Note that keyStorePass: wso2carbon needs to be changed as keyStorePassword: wso2carbon.

   listenerConfigurations:
    -
     id: "default"
     host: "0.0.0.0"
     port: 9090
     messageProcessorId: "MSF4J-CM-PROCESSOR"
   
    -
     id: "https"
     host: "0.0.0.0"
     port: 9443
     scheme: https
     keyStoreFile: ${carbon.home}/resources/security/wso2carbon.jks
     keyStorePassword: wso2carbon
     certPass: wso2carbon
     messageProcessorId: "MSF4J-CM-PROCESSOR"

2. Remove OB_REPORT_GENERIC_SUMMARIZATION.siddhi app from <WSO2_OB_BI_HOME>/deoployment/siddhi-fles.

Configuring keystores

In the SSL handshake between the Open Banking API Manager and Open Banking Business Intelligence servers, the client (i.e. Open Banking API Manager) needs to verify the certificate presented by the server (i.e. Open Banking Business Intelligence). For this purpose, the client stores the trusted certificate of the server in the client-truststore.jks keystore.

If you use a custom keystore in Open Banking API Manager and/or Open Banking Business Intelligence, import the public key certificate of Open Banking Business Intelligence into the client-truststore.jks file of the Open Banking API Manager. To export the public key from the server and import it into the client's trust store, follow the steps given in Adding CA-signed certificates to keystores in the Administration Guide.

Enabling data publishing in API Manager

Follow the instructions below to configure the WSO2 Open Banking API Manager to publish statistics. 

1. Open the <WSO2_OB_APIM_HOME>/repository/conf/deplpoyment.toml file

2. Enable the[apim.analytics] sub-element by configuring it as true.

   [apim.analytics]
   enable = false

3. Change the values for the following properties accordingly:

   [apim.analytics]
   eceiver_urls = "{tcp://<WSO2_OB_BI_HOST>:7612}"
   receiver_username = "$ref{super_admin.username}@carbon.super"
   receiver_password = "$ref{super_admin.password}"
   store_api_username = "$ref{super_admin.username}@carbon.super"

   Enable alerts:

   If you are using Open Banking Business Intelligence for Analytics, follow the steps below to enable alerts:

   • Open the <WSO2_OB_BI_HOME>/conf/worker/deployment.yaml file.

   • Update the file as follows: 

     analytics.solutions:
       APIM-alerts.enabled: true 
       APIM-analytics.enabled: true

   • See Managing Alert for more information.

   Unattended session count throttling configuration

   This is available only as a WSO2 Update and is effective from September 14, 2021 (09-14-2021). For more information on updating WSO2 Open Banking, see Updating WSO2 Products.

   1. To enable Unattended session count throttling:
      1. Open the <WSO2_OB_APIM_HOME>/repository/conf/deplpoyment.toml file

      2. To enable Unattended session count throttling, add the following configurations:

         [open_banking.au.unattended_session_count_throttling]
         enable = true

      3. Configure the openbank_ob_reporting_statsdb database, using the following tags. Update the values with your database configurations. 

         [open_banking_reporting_database]
         config.url = "jdbc:mysql://localhost:3306/openbank_ob_reporting_statsdb?autoReconnect=true&useSSL=false"
         config.username = "DB_USER"
         config.password = "DB_PASS"
         config.driver = "com.mysql.jdbc.Driver"
         
         [open_banking_reporting_database.config.pool_options]
         maxActive = "150"
         maxWait = "60000"
         minIdle ="5"
         testOnBorrow = true
         validationQuery="SELECT 1"
         #Use below for oracle
         #validationQuery="SELECT 1 FROM DUAL"
         validationInterval="30000"
         defaultAutoCommit=false
         

   2. To enable token encryption, add the following configurations to the deployment.toml files and restart the servers.

      • <WSO2_OB_APIM_HOME>/repository/conf/deplpoyment.toml file

      • <WSO2_OB_IAM_HOME>/repository/conf/deplpoyment.toml file

        [open_banking.bi_server.data_publishing.token_encryption]
        enable=true
        secretKey="wso2"

4. Save the changes and restart the WSO2 Open Banking API Manager server.

Enabling Open Banking Data Publishing

Follow the steps below to enable Data Reporting. 

1. Open  <WSO2_OB_APIM_HOME>/repository/conf/finance/deployment.toml and  <WSO2_OB_IAM_HOME>/repository/conf/finance/deployment.toml  files.
2. Under the [open_banking.bi_server.data_publishing] sub-element set the enable property to true to enable the feature. 

3. Replace the <WSO2_OB_BI_HOST> placeholder with the hostname of your WSO2 OB BI server and configure the other parameters accordingly.  

   [open_banking.bi_server.data_publishing]
   enable = false
   server_url = "{tcp://<WSO2_OB_BI_HOST>:7612}"

   1. Go to the <WSO2_OB_BI_HOME>/resources/finance/cds-siddhi-files directory.
   2. Copy and paste the following to the <WSO2_OB_BI_HOME>/deployment/siddhi-files directory.
      1. CDSInvocationMetricsApp.siddhi
      2. CDSSessionMetricsApp.siddhi

Invoking Data Reporting API

Using the Data Reporting API, the Data Holders can retrieve statistics on API invocation data for a requested period which are required for regulatory reporting. 

1. Deploying the GetInvocationDataApp Siddhi App, which is required to publish data related to data reporting:
   1. Copy the <WSO2_OB_BI_HOME>/resources/finance/cds-siddhi-files/GetInvocationDataApp.siddhi file to the <WSO2_OB_BI_HOME>/deployment/siddhi-files directory.
   2. Restart the WSO2 Open Banking Business Intelligence, Identity and Access Management and API Management servers in the mentioned order, for the changes to take place.
2. Invoking the API:
   
   • Set the date range by changing the values in the fromDate and toDate parameters in the request. Given below is a sample request and response.

Data Reporting - Phase 2

As a part of CDR Reporting - Phase 2, WSO2 Open Banking captures CDR Rule violations that happen during API invocations and combines them into the response of the above Data Reporting API.

Given below is a sample request to retrieve bi-annual reporting data :

You need to capture the data required for the following rules from the banking backend. The scenarios are as follows:

• UNAVAILABLE_BANKING_ACCOUNT:  The consent is valid for the given account but certain data sharing is restricted due to other business validations in the bank backend.

• INVALID_RESOURCE: Trying to access an invalid resource excluding an account. For example, TransactionID. 

  • To capture data for the above scenarios, banks should reply with the correct error code otherwise those numbers will not be captured in the response.
  • For the rest of the rules, the WSO2 Open Banking solution captures data.

For more details on reporting requirements, see CDR Reporting.

CDS Administration API

See Deploying Consumer Data Standards Administration API v1.2.0 and deploy the API.

GET Metrics - GET /admin/metrics

GET /admin/metrics is an endpoint determined by the non-functional requirements for the CDR regime to obtain statistics. 

This is the only endpoint available in the API. A sample request and response are as follows: