Clustering API Manager

WSO2 API Manager is a complete API management solution, used for creating and publishing APIs, creating and managing a developer community, and scalably routing API traffic. The API Manager solution includes a Publisher, Store, Gateway, and Key Manager component.

Ideally in a production deployment, these components will be deployed in a distributed manner. Before you set up a cluster for WSO2 API Manager, you can create a distributed deployment of its four main components: Publisher, Store, Gateway, and Key Manager. This page describes how to set up the distributed deployment in the following sections.

Note that your configurations may vary depending on the API Manager clustering deployment pattern you choose. Additionally these instructions are based on API Manager 1.5.0, 1.6.0, 1.7.0, 1.8.0 and 1.9.0 which are based on Carbon 4.2.0. If you are using multi-tenancy, all nodes should use the same user store, as all servers are servicing the same set of tenants, and it has to share the same Governance Registry space across all nodes.

Understanding the API Manager architecture

API Manager uses the following four main components:

Publisher	Enables API providers to easily publish their APIs, share documentation, provision API keys, and gather feedback on API features, quality, and usage.
Store	Enables consumers to self-register, discover API functionality, subscribe to APIs, evaluate them, and interact with API publishers.
Key Manager	Responsible for all security and key-related operations.
Gateway	Responsible for securing, protecting, managing, and scaling API calls.

For more information on the above, see the main components of a distributed system.

Additionally, API Manager uses three databases. The following databases are shared among the server nodes.

User Manager Database - Stores information related to users and user roles. This information is shared among the Key Manager Server, Store, and Publisher. Users can access the Publisher for API creation and the Store for consuming the APIs.
API Manager Database - Stores information related to the APIs along with the API subscription details. The Key Manager Server uses this database to store user access tokens that are used for verification of API calls.
Registry Database - Shares information between the Publisher and Store. When an API is published through the Publisher, it is made available in the Store via the sharing registry database. Although you would normally share information between the Publisher and Store components only, if you are planning to create this setup for a multi-tenanted environment (create and work with tenants), it is required to share the information in this database between the Gateway and Key Manager components as well.

The API Manager components use the databases as follows:

	(API Manager Database) apimgtdb	(User Manager Database) userdb	(Registry Database) regdb
Publisher	Used	Used	Used
Store	Used	Used	Used
Key Manager	Used	Used	Used (in multi-tenancy mode)
Gateway	Not used	Used (in multi-tenancy mode)	Used (in multi-tenancy mode/in multiple gateway mode)

When we consider distributed deployment of WSO2 API Manager, we have the option of separating its four components and clustering each component as needed. Let's look more closely at how the API Manager components are deployed separately.

Creating the distributed deployment

In the following diagram, the four components are set up in a distributed deployment, and the three databases are connected to the relevant components respectively. The entire setup is also fronted by an instance of WSO2 Elastic Load Balancer (ELB). Notice the port offset and the HTTPS port of each WSO2 server component.

Note that the default communication protocol of Key Manager is Thrift. If your setup has multiple Key Manager nodes that are fronted by an ELB instance for load balancing, change the key management protocol from Thrift to WSClient using the <KeyValidatorClientType> element in <APIM_HOME>/repository/conf/api-manager.xml file. Thrift uses TCP load balancing and the ELB does not support it.

To achieve the above deployment, we will complete the following tasks:

Installing and configuring API Manager

The following steps describe how to download, install, and configure API Manager. We will create four instances of API Manager for this.

Download the latest versions of WSO2 API Manager in each of the servers in the cluster for distributed deployment. This topic provides instructions on how to do this for API Manager version 1.5.0, 1.6.0, 1.7.0, 1.8.0 or 1.9.0.
Unzip the API Manager zipped archive, and rename each of those directories respectively as Key Manager, Gateway, Publisher, and Store. These four directories are located in a server of their own and will be used for each component of the API Manager. Each of these unzipped directories will be referred to as <APIM_HOME> or <PRODUCT_HOME> in this document.
Only do this step if you wish to run WSO2 servers (KeyManager, Gateway, Publisher, and Store) on the same machine. You need to set the Offset attribute to avoid any port usage conflict, as follows:

In each of the directories you just created, open the <PRODUCT_HOME>/repository/conf/carbon.xml file.
Edit the <Offset> attribute in each file as shown in the Port Offset column in the following table. The port value will automatically be increased as shown in the Port Value column, allowing all five WSO2 server instances to run on the same machine.
WSO2 Server instance
Port Offset
Port Value
KeyManager
0
9444
Gateway
1
9445
Publisher
2
9446
Store
3
9447

This step is optional and only required if all server instances are running in the same machine. This is not a recommended approach for production environments. Note that you need to change all ports used in your configurations based on the offset value if you are to do this. See Changing the Default Ports with Offset for more information.

Installing and configuring the databases

The following steps describe how to download and install MySQL Server, create the databases, configure the data sources, and configure the API Manager components to connect to them. Additionally you may need to configure your API Analytics platform (BAM) with API Manager in the distributed setup.

Download and install MySQL Server.
Download the MySQL JDBC driver.
Unzip the downloaded MySQL driver zipped archive, and copy the MySQL JDBC driver JAR (mysql-connector-java-x.x.xx-bin.jar) into the <PRODUCT_HOME>/repository/components/lib directory of all the nodes in the cluster.
Define the host name for configuring permissions for the new database by opening the /etc/hosts file and adding the following line:
```
<MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com
```
You would do this step only if your database is not on your local machine and on a separate server.
Enter the following command in a terminal/command window, where username is the username you want to use to access the databases:
mysql -u username -p
When prompted, specify the password that will be used to access the databases with the username you specified.
Create the three databases using the following commands, where <APIM_HOME> is the path to any of the API Manager instances you installed, and username and password are the same as those you specified in the previous steps.
About using MySQL in different operating systems
For users of Microsoft Windows, when creating the database in MySQL, it is important to specify the character set as latin1. Failure to do this may result in an error (error code: 1709) when starting your cluster. This error occurs in certain versions of MySQL (5.6.x) and is related to the UTF-8 encoding. MySQL originally used the latin1 character set by default, which stored characters in a 2-byte sequence. However, in recent versions, MySQL defaults to UTF-8 to be friendlier to international users. Hence, you must use latin1 as the character set as indicated below in the database creation commands to avoid this problem. Note that this may result in issues with non-latin characters (like Hebrew, Japanese, etc.). The following is how your database creation command should look.
```
mysql> create database <DATABASE_NAME> character set latin1;
```
For users of other operating systems, the standard database creation commands will suffice. For these operating systems, the following is how your database creation command should look.
```
mysql> create database <DATABASE_NAME>;
```
```
mysql> create database apimgtdb;
mysql> use apimgtdb;
mysql> source <APIM_HOME>/dbscripts/apimgt/mysql.sql;
mysql> grant all on apimgtdb.* TO username@localhost identified by "password";


mysql> create database userdb;
mysql> use userdb;
mysql> source <APIM_HOME>/dbscripts/mysql.sql;
mysql> grant all on userdb.* TO username@localhost identified by "password";


mysql> create database regdb;
mysql> use regdb;
mysql> source <APIM_HOME>/dbscripts/mysql.sql;
mysql> grant all on regdb.* TO username@localhost identified by "password"; 
```
Ensure that MySQL is configured so that all nodes can connect to it.

Configure the data sources for the three database as follows:

Open the <APIM_HOME>/repository/conf/datasources/master-datasources.xml file in all four API Manager components.

Enable the components to access the API Manager database by modifying the WSO2AM_DB data source in all four master-datasources.xml files as follows:

<datasource>
 <name>WSO2AM_DB</name>
 <description>The datasource used for the API Manager database</description>
 <jndiConfig>
   <name>jdbc/WSO2AM_DB</name>
 </jndiConfig>
 <definition type="RDBMS">
   <configuration>
     <url>jdbc:mysql://apimgtdb.mysql-wso2.com:3306/apimgtdb?autoReconnect=true</url>
     <username>user</username>
     <password>password</password>
     <defaultAutoCommit>false</defaultAutoCommit>
     <driverClassName>com.mysql.jdbc.Driver</driverClassName>
     <maxActive>50</maxActive>
     <maxWait>60000</maxWait>
     <testOnBorrow>true</testOnBorrow>
     <validationQuery>SELECT 1</validationQuery>
     <validationInterval>30000</validationInterval>
   </configuration>
 </definition>
</datasource>

Enable the Key Manager, Publisher, and Store components to access the users database by configuring the WSO2UM_DB data source in their master-datasources.xml files as follows:

<datasource>
 <name>WSO2UM_DB</name>
 <description>The datasource used by user manager</description>
 <jndiConfig>
   <name>jdbc/WSO2UM_DB</name>
 </jndiConfig>
 <definition type="RDBMS">
   <configuration>
     <url>jdbc:mysql://userdb.mysql-wso2.com:3306/userdb?autoReconnect=true</url>
     <username>user</username>
     <password>password</password>
     <driverClassName>com.mysql.jdbc.Driver</driverClassName>
     <maxActive>50</maxActive>
     <maxWait>60000</maxWait>
     <testOnBorrow>true</testOnBorrow>
     <validationQuery>SELECT 1</validationQuery>
     <validationInterval>30000</validationInterval>
   </configuration>
 </definition>
</datasource>

Enable the Publisher and Store components to access the registry database by configuring the WSO2REG_DB data source in their master-datasources.xml files as follows:

<datasource>
 <name>WSO2REG_DB</name>
 <description>The datasource used by the registry</description>
 <jndiConfig>
   <name>jdbc/WSO2REG_DB</name>
 </jndiConfig>
 <definition type="RDBMS">
   <configuration>
     <url>jdbc:mysql://regdb.mysql-wso2.com:3306/regdb?autoReconnect=true</url>
     <username>user</username>
     <password>password</password>
     <driverClassName>com.mysql.jdbc.Driver</driverClassName>
     <maxActive>50</maxActive>
     <maxWait>60000</maxWait>
     <testOnBorrow>true</testOnBorrow>
     <validationQuery>SELECT 1</validationQuery>
     <validationInterval>30000</validationInterval>
   </configuration>
 </definition>
</datasource>

Modify the /etc/hosts entries to map the IP addresses to the data source URLs:

- 127.0.0.1 apimgtdb.mysql-wso2.com
- 127.0.0.1 userdb.mysql-wso2.com
- 127.0.0.1 regdb.mysql-wso2.com

To give each of the components access to the API Manager database, open the <APIM_HOME>/repository/conf/api-manager.xml file in each of the components and add the following line as the first child node of the root element (if it is not already there):
<DataSourceName>jdbc/WSO2AM_DB</DataSourceName>

To give the Key Manager, Publisher, and Store components access to the users database with shared permissions, open the <APIM _HOME>/repository/conf/user-mgt.xml file in each of these three components and add or modify the dataSource property of the <configuration> element as follows:

<configuration> 
...
	<Property name="dataSource">jdbc/WSO2UM_DB</Property>
</configuration>
 
<UserStoreManager class="org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager">
	<Property name="TenantManager">org.wso2.carbon.user.core.tenant.JDBCTenantManager</Property>
    <Property name="ReadOnly">false</Property>
    <Property name="MaxUserNameListLength">100</Property>
    <Property name="IsEmailUserName">false</Property>
    <Property name="DomainCalculation">default</Property>
    <Property name="PasswordDigest">SHA-256</Property>
    <Property name="StoreSaltedPassword">true</Property>
    <Property name="ReadGroups">true</Property>
    <Property name="WriteGroups">true</Property>
    <Property name="UserNameUniqueAcrossTenants">false</Property>
    <Property name="PasswordJavaRegEx">^[\S]{5,30}$</Property>
    <Property name="PasswordJavaScriptRegEx">^[\S]{5,30}$</Property>
    <Property name="UsernameJavaRegEx">^[^~!#$;%^*+={}\\|\\\\&lt;&gt;,\'\"]{3,30}$</Property>
    <Property name="UsernameJavaScriptRegEx">^[\S]{3,30}$</Property>
    <Property name="RolenameJavaRegEx">^[^~!#$;%^*+={}\\|\\\\&lt;&gt;,\'\"]{3,30}$</Property>
    <Property name="RolenameJavaScriptRegEx">^[\S]{3,30}$</Property>
    <Property name="UserRolesCacheEnabled">true</Property>
    <Property name="MaxRoleNameListLength">100</Property>
    <Property name="MaxUserNameListLength">100</Property>
    <Property name="SharedGroupEnabled">false</Property>
    <Property name="SCIMEnabled">false</Property>
</UserStoreManager>

To give the Publisher and Store components access to the registry database, open the <APIM_HOME>/repository/conf/registry.xml file in each of these two components and configure them as follows:

Although it is mentioned that you need to do this on the Publisher and Store components only, if you are planning to create this setup for a multi-tenanted environment (create and work with tenants), it is required to perform the steps below on the Gateway and Key-Manager components as well.

Note: Do not replace the following configuration when adding in the mounting configurations. The registry mounting configurations mentioned in the below steps must be added in addition to the following.

<dbConfig name="wso2registry">
	<dataSource>jdbc/WSO2CarbonDB</dataSource>
</dbConfig>

In the Publisher component's registry.xml file, add or modify the dataSource attribute of the <dbConfig name="govregistry"> element as follows:

<dbConfig name="govregistry">
  <dataSource>jdbc/WSO2REG_DB</dataSource>
</dbConfig>
<remoteInstance url="https://publisher.apim-wso2.com"> 
   <id>gov</id>
   <cacheId>user@jdbc:mysql://regdb.mysql-wso2.com:3306/regdb</cacheId>
   <dbConfig>govregistry</dbConfig>
   <readOnly>false</readOnly>
   <enableCache>true</enableCache>
   <registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/governance" overwrite="true">
   <instanceId>gov</instanceId>
   <targetPath>/_system/governance</targetPath>
</mount>
<mount path="/_system/config" overwrite="true">
   <instanceId>gov</instanceId>
   <targetPath>/_system/config</targetPath>
</mount>

In the Store component's registry.xml file, add or modify the dataSource attribute of the <dbConfig name="govregistry"> element as follows (note that this configuration is nearly identical to the previous step with the exception of the remoteInstance URL):

<dbConfig name="govregistry">
  <dataSource>jdbc/WSO2REG_DB</dataSource>
</dbConfig>
<remoteInstance url="https://store.apim-wso2.com"> 
   <id>gov</id>
   <cacheId>user@jdbc:mysql://regdb.mysql-wso2.com:3306/regdb</cacheId> 
   <dbConfig>govregistry</dbConfig>
   <readOnly>false</readOnly>
   <enableCache>true</enableCache>
   <registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/governance" overwrite="true">
   <instanceId>gov</instanceId>
   <targetPath>/_system/governance</targetPath>
</mount>
<mount path="/_system/config" overwrite="true">
   <instanceId>gov</instanceId>
   <targetPath>/_system/config</targetPath>
</mount>

Modify the /etc/hosts entries to map the relevant IP addresses to the remoteInstance URLs.

- 127.0.0.1 publisher.apim-wso2.com
- 127.0.0.1 store.apim-wso2.com

Once registry caching is enabled, sync the published APIs between the Publisher and Store nodes by enabling clustering in both Store and Publisher nodes. To do this, open the <APIM_HOME>/repository/conf/axis2/axis2.xml file in each of these two components and configure them as follows:
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

Configuring the connections among the components

You will now configure the inter-component relationships illustrated in the following diagram by modifying their <APIM_HOME>/repository/conf/api-manager.xml files.

Open the <APIM_HOME>/repository/conf/api-manager.xml files in the Gateway, Publisher, Key Manager and Store components.
Modify each of the files as follows:
- Key Manager:
- Gateway: configure the connection to the Key Manager component as follows:
- Publisher: configure connections to the Key Manager and Gateway as follows:
  
  Creating a connection to the Key Manager from the Publisher is required in API Manager 1.3.0 but is no longer required from API Manager 1.4.0 onwards.
- Store
  
  Tip: For more information on how different environment types, see Maintaining Separate Production and Sandbox Gateways. This is useful when configuring environments in different geographical locations.

Configuring the API Publisher and API Store to be on the same cluster domain

As explained in the sections above, the API Publisher and API Store use a shared registry for sharing API artifacts. Therefore whenever an artifact is added/modified on the Publisher, the change needs to be reflected on the API Store appropriately. Due to the fact that the API Store fetches artifacts from the registry cache, modifications to artifacts done on the Publisher may not immediately be reflected on the Store unless the relevant cache entry is explicitly cleared off or the cache expires. For the cache to be explicitly cleared off, the Publisher and Store need to be in the same cluster domain and need to be aware of each other's existence. See Configuring Caching for more information.

A Gateway node must be up and running for you to create APIs through the API Publisher.

How to configure the Publisher and Store to be in the same cluster domain.

Follow the steps below to configure the Publisher and Store to be in a single cluster domain. You need to configure both nodes as instructed below

Open the <AM_HOME>/repository/conf/axis2/axis2.xml file and scroll down to the 'Clustering' section.

Set the 'enable' attribute of the <clustering> element to true.

<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

Change the 'membershipScheme' parameter to 'wka'.

<parameter name="membershipScheme">wka</parameter>

Provide a domain for the cluster.

<parameter name="domain">wso2.pub.store.domain</parameter>

Specify the 'localMemberHost' and 'localMemberPort' parameters. If on the publisher, the 'localMemberHost' should be the publisher's IP address. The port value should be the port on which the Publisher will be listening for incoming cluster messages. Same applies to the Store.
```
<parameter name="localMemberHost">192.168.10.1</parameter>
<parameter name="localMemberPort">4000</parameter>
```
Specify well known member. When specifying the well known member, the Publisher should specify the Store information and the Store should specify the Publisher information. The port that we provide here should be equal to the 'localMemberPort' of the other member.
```
<members>
   <member>
       <hostName>192.168.10.2</hostName>
       <port>4000</port>
   </member>
</members>
```
Save and close the file and restart the servers (if running) for the changes to take effect.

Configuring component features

The following is a list of configurations for some of the API Manager features in distributed or clustered deployment.

Configure key management related communication.
If you need to enable JWT you have to enable it in all key-manager and publisher components. Refer Generating JSON Web Token on how to configure JWT.
In the Gateway, set up the Key Manager endpoint in the following files:
- _TokenAPI_xml found in the <APIM_HOME>/repository/deployment/server/synapse-configs/default/api/_TokenAPI_xml file.
- _AuthorizeAPI_xml found in the <APIM_HOME>/repository/deployment/server/synapse-configs/default/api/_AuthorizeAPI_xml file.
- _RevokeAPI_xml found in the <APIM_HOME>/repository/deployment/server/synapse-configs/default/api/_RevokeAPI_xml file.
This value needs to be the host or IP of the Key Manager.

You have now successfully distributed and deployed the WSO2 API Manager components.

Creating the clustered deployment

The above distributed API Manager setup can be extended to a clustered deployment by adding a cluster of Gateway components or adding a cluster of Key Manager components.

Create a cluster of Gateway components

To cluster API Manager, create a cluster of Gateway components. To cluster the Gateway component, use the steps in Clustering the Gateway.

WSO2 Server instance	Port Offset	Port Value
KeyManager	0	9444
Gateway	1	9445
Publisher	2	9446
Store	3	9447