Configuring an Active-Active Deployment

This page walks you through how to manually configure the API Manager with two active nodes when deploying as an all-in-one instance.

Unzip the WSO2 API Manager pack. Let's call it <API-M_HOME>.

Open the <API-M_HOME>/repository/conf/datasources/master-datasources.xml file. This file contains the different datasources used by WSO2 API Manager. By default, the API Manager connects to the local H2 database and it is recommended to use a separate RDBMS server for a production deployment.

Follow the steps below to integrate the API Manager with an external database (in this case, an external MySQL Server).

Update the existing WSO2AM_DB with the configuration given below.

Add two new entries, WSO2GOV_DB and WSO2UM_DB, as shown below.

<datasource>
            <name>WSO2AM_DB</name>
            <description>The datasource used for API Manager database</description>
            <jndiConfig>
                <name>jdbc/WSO2AM_DB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
					<url>jdbc:mysql://localhost:3306/apimgtdb</url>
                    <username>root</username>
                    <password>root</password>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <minIdle>5</minIdle>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                    <defaultAutoCommit>false</defaultAutoCommit>
                </configuration>
            </definition>
</datasource>

<datasource>
            <name>WSO2GOV_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2GovDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
					<url>jdbc:mysql://localhost:3306/regdb</url>
                    <username>root</username>
                    <password>root</password>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <minIdle>5</minIdle>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                    <defaultAutoCommit>false</defaultAutoCommit>
                </configuration>
            </definition>
        </datasource>


<datasource>
            <name>WSO2UM_DB</name>
            <description>The datasource used for API Manager database</description>
            <jndiConfig>
                <name>jdbc/WSO2UM_DB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
  				 	<url>jdbc:mysql://localhost:3306/userdb</url>
                    <username>root</username>
                    <password>root</password>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <minIdle>5</minIdle>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                    <defaultAutoCommit>false</defaultAutoCommit>
                </configuration>
            </definition>
</datasource>

Create the required databases.

WSO2 API Manager is shipped with an H2 database. This embedded H2 database is suitable for development and testing environments. However, for production environments, it is recommended to use an industry-standard RDBMS such as Oracle, PostgreSQL, MySQL, MS SQL, etc.
The following steps describe how to download and install MySQL Server, create the databases, configure the datasources, and configure the API Manager components to connect to them.
1. Download and install MySQL Server.
2. Download the MySQL JDBC driver and unzip the downloaded MySQL driver zipped archive.
3. Copy the MySQL JDBC driver JAR file (mysql-connector-java-x.x.xx-bin.jar) into the <API-M_HOME>/repository/components/lib directory.
4. To define the host name for configuring permissions for the new database, open the /etc/hosts file and add the following:
  
  Do this step only if your database is not on your local machine and on a separate server.
```
<MYSQL-DB-SERVER-IP> carbondb.mysql-wso2.com
```
5. Enter the following command in a command window, where username is the username you want to use to access the databases,
```
mysql -u username -p
```
6. When prompted, specify the password that will be used to access the databases with the username you specified.
7. Create the three databases using the following commands, where <API-M_HOME> is the path to any of the API Manager instances you installed, and username and password are the same as those you specified in the previous steps.
  
  For Microsoft Windows users: When creating the database in MySQL, it is important to specify the character set as latin1. Failure to do this may result in an error (error code: 1709) when starting your cluster. This error occurs in certain versions of MySQL (5.6.x), and is related to the UTF-8 encoding. MySQL originally used the latin1 character set by default, which stored characters in a 2-byte sequence. However, in recent versions, MySQL defaults to UTF-8 to be friendlier to international users. Therefore, in order to avoid this problem, use latin1 as the character set as indicated below in the database creation commands. Note that this may result in issues with non-latin characters (like Hebrew, Japanese, etc.). The database creation command should be as follows:
  mysql> create database <DATABASE_NAME> character set latin1;
  For users of other operating systems: The standard database creation commands will suffice. For these operating systems, the database creation command should be as follows:.
  mysql> create database <DATABASE_NAME>;
  
  From WSO2 Carbon Kernel 4.4.6 onwards there are two MySQL DB scripts available in the product distribution. Click here to identify as to which version of the MySQL script to use.
```
mysql> create database apimgtdb;
mysql> use apimgtdb;
mysql> source <APIM_HOME>/dbscripts/apimgt/mysql.sql;
mysql> grant all on apimgtdb.* TO username@localhost identified by "password";
 
mysql> create database userdb;
mysql> use userdb;
mysql> source <APIM_HOME>/dbscripts/mysql.sql;
mysql> grant all on userdb.* TO username@localhost identified by "password";
 
mysql> create database regdb;
mysql> use regdb;
mysql> source <APIM_HOME>/dbscripts/mysql.sql;
mysql> grant all on regdb.* TO username@localhost identified by "password";
```
Configure the API Manager to refer to the WSO2UM_DB for user information by updating the following configuration in the <API-M_HOME>/repository/conf/user-mgt.xml file:
```
<Property name="dataSource">jdbc/WSO2UM_DB</Property>
```
If you are using the WSO2UM_DB to store users, remember to change the administrator username and password.
Start the API Manager with the following command,
For Linux
```
sh wso2server.sh -Dsetup
```
For Windows
```
wso2server.bat -Dsetup
```
This creates the required tables. Once the server starts successfully, you can shutdown it down and continue with the rest of the steps.

To add a registry entry to reflect the newly added datasource, add the following configurations to the <API-M_HOME>/repository/conf/registry.xml file as shown below:

<dbConfig name="wso2gov">
        <dataSource>jdbc/WSO2GovDB</dataSource>
</dbConfig>

<remoteInstance url="https://localhost:9453/registry">
        <id>wso2gov</id>
        <dbConfig>wso2gov</dbConfig>
        <readOnly>false</readOnly>
        <registryRoot>/</registryRoot>
</remoteInstance>

<mount path="/_system/governance" overwrite="true">
        <instanceId>wso2gov</instanceId>
        <targetPath>/_system/governance</targetPath>
</mount>
<mount path="/_system/config" overwrite="true">
   <instanceId>wso2gov</instanceId>
   <targetPath>/_system/config</targetPath>
</mount>

Do not replace the following configuration when adding the above mounting configurations. The registry mounting configurations mentioned above must be added in addition to the following.

<dbConfig name="wso2registry">
    <dataSource>jdbc/WSO2CarbonDB</dataSource>
</dbConfig>

WSO2 API Manager is shipped with a default keystore named wso2carbon.jks. It is recommended to change this default keystore in a production deployment. For more information on changing this default keystore, see Creating New Keystores.

A load balancer or reverse proxy is required to map external traffic with ports and URLs used internally by API Manager.

Update the ngnix.conf file with the required Nginx configuration given below. In this case, the hostname is assumed to be localhost. Ensure that you generate a certificate and update the certificate and key path in the configuration below:

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    upstream manager-worker {
        server 127.0.0.1:9443;  
	}
	 upstream allnodes {
        ip_hash;
		server 127.0.0.1:9443;
		server 127.0.0.1:9444; 
	}
	upstream allnodes-traffic-http {
        ip_hash;
		server 127.0.0.1:8280;
		server 127.0.0.1:8281;
	}
	upstream allnodes-traffic-https {
        ip_hash;
		server 127.0.0.1:8243;
		server 127.0.0.1:8244;     
	}
	server {
        server_name  localhost;	
		listen 443;
		ssl on;
		ssl_certificate <<Certifacte.crt>>;
		ssl_certificate_key <<CertficateKey.key>>;
		#Carbon - Manager-worker
		location /carbon {
           index index.html;
           proxy_set_header X-Forwarded-Host $host;
           proxy_set_header X-Forwarded-Server $host;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_pass https://manager-worker/carbon/;
           proxy_redirect  https://manager-worker/carbon/  https://localhost/carbon/;
		   #proxy_cookie_path / /carbon/;
       }
		#Store Registry for images - allnodes
		location ~ ^/store/(.*)registry/(.*)$ {
		   index index.html;
		   proxy_set_header X-Forwarded-Host $host;
		   proxy_set_header X-Forwarded-Server $host;
		   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		   proxy_pass https://allnodes/$1registry/$2;
		   proxy_next_upstream     error timeout invalid_header http_500;
			proxy_connect_timeout   2;
		}
		#Publisher Registry for images - manager-worker
       location ~ ^/publisher/(.*)registry/(.*)$ {
           index index.html;
           proxy_set_header X-Forwarded-Host $host;
           proxy_set_header X-Forwarded-Server $host;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_pass https://manager-worker/$1registry/$2;
       }
		# Publisher
		location /publisher {
			  index index.html;
			   proxy_set_header X-Forwarded-Host $host;
			   proxy_set_header X-Forwarded-Server $host;
			   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			   proxy_pass https://manager-worker/publisher;
			   proxy_redirect  https://manager-worker/publisher  https://localhost/publisher;
			   proxy_cookie_path /publisher /publisher;
		  }
		# Admin Console
		location /admin {
		   proxy_set_header X-Forwarded-Host $host;
		   proxy_set_header X-Forwarded-Server $host;
		   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		   proxy_pass https://manager-worker/admin;
		   proxy_redirect  https://manager-worker/admin  https://localhost/admin;
		   proxy_cookie_path /admin /admin;
		}
		#API traffic - All nodes - HTTPS
		location / {
		proxy_pass https://allnodes-traffic-https/;
		proxy_next_upstream     error timeout invalid_header http_500;
        proxy_connect_timeout   2;
		}
		# All Store - All nodes
		location /store {
		   index index.html;
		   proxy_set_header X-Forwarded-Host $host;
		   proxy_set_header X-Forwarded-Server $host;
		   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		   proxy_pass https://allnodes/store;
		   proxy_redirect https://allnodes/store https://localhost/store;
		   proxy_cookie_path /store /store;
		   proxy_next_upstream     error timeout invalid_header http_500;
		   proxy_connect_timeout   2;
		}
	}
		server {
        server_name  localhost;
		listen 80;
		#API traffic - All nodes - HTTP
		location / {
		proxy_pass http://allnodes-traffic-http/;
		proxy_next_upstream     error timeout invalid_header http_500;
        proxy_connect_timeout   2;
		}
		}
}

The ports and URLs that are used internally by API Manager are given below:

Usage	URL	Port
HTTP Servlet	localhost	9763
HTTPS Servlet (UI Consoles)	localhost	9443
NIO transport (HTTP API Traffic)	localhost	8280
NIO transport (HTTPS API Traffic)	localhost	8243

Ensure that the ports and URLs are mapped correctly in the load balancer.

In a Active-Active setup, It is mandatory to set up Session Affinity in the load balancers that front the Publisher and Store, and it is optional in the load balancer (if any) that fronts Key Manager or Gateway.

However, authentication via session ID fails when Session Affinity is disabled in the load balancers of Publisher and store.

Follow the steps below to update the API Store, API Publisher and Admin Portal to work with the Proxy Server configuration.

API Store - Update the <API-M_Home>\repository\deployment\server\jaggeryapps\store\site\conf\site.json file as shown below:

"reverseProxy" : {
        "enabled" : true,    // values true , false , "auto" - will look for  X-Forwarded-* headers
        "host" : "localhost", // If reverse proxy do not have a domain name use IP
        "context":"/store",
      //"regContext":"" // Use only if different path is used for registry
    }

API Publisher - Update the <API-M_Home>\repository\deployment\server\jaggeryapps\publisher\site\conf\site.json file as shown below:

"reverseProxy" : {
        "enabled" : true,    // values true , false , "auto" - will look for  X-Forwarded-* headers
        "host" : "localhost", // If reverse proxy do not have a domain name use IP
        "context":"/publisher",
      //"regContext":"" // Use only if different path is used for registry
    }

Admin Portal - Update the <API-M_Home>\repository\deployment\server\jaggeryapps\admin\site\conf\site.json file as shown below:

"reverseProxy" : {
        "enabled" : true,    // values true , false , "auto" - will look for  X-Forwarded-* headers
        "host" : "localhost", // If reverse proxy do not have a domain name use IP
        "context":"/admin",
      //"regContext":"" // Use only if different path is used for registry
    },

Make a copy of the active instance configured above. Use this copy as the second active instance.
Follow the steps below to enable clustering to ensure that each node is in sync with the changes that happen to the other node.
1. Open the <API-M_HOME>/repository/conf/axis2/axis2.xml file and set the enable attribute of the <clustering> element to true as shown below,
```
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
```
2. Change the membershipScheme parameter to wka as shown below,
```
<parameter name="membershipScheme">wka</parameter>
```
3. Provide a domain for the cluster as shown below,
```
<parameter name="domain">wso2.pub.store.domain</parameter>
```
4. Specify the localMemberHost and localMemberPort parameters. The localMemberHost should be the server's IP address. The port value should be the port on which the server will be listening for incoming cluster messages. If you are running the API Manager nodes on the same machine, you require two different localMemberPort numbers.
```
<parameter name="localMemberHost">192.168.10.1</parameter>
<parameter name="localMemberPort">4000</parameter>
```
5. Specify a well known member. When specifying the well known member, the primary active node should specify the information of the secondary active node and vice versa. The port you provide here should be the same as the localMemberPort of the other member.
```
<members>
   <member>
     <hostName>192.168.10.2</hostName>
       <port>4000</port>
   </member>
</members>
```
6. Save and close the file and restart the servers (if running) for the changes to take effect.
Configure the API Publisher of both nodes to publish to the Gateway of one of the nodes by pointing the <ServerURL> to the same Gateway node.
```
<APIGateway>
	<ServerURL>https://localhost:${mgt.transport.https.port}${carbon.context}services/</ServerURL>
</APIGateway>
```
You require a content synchronization mechanism like Rsync to synchronize the artifacts in the <API-M_HOME>/repository/deployment/server/synapse-configs directory of one node with those of the other node. To set up an Rsync based deployment synchronization, see Configuring Rsync for Deployment Synchronization.
You need to configure the Traffic Manager of each node to be able to publish events to the Traffic Manager of the other node. Let's create an additional JNDI config file, namely jndi2.properties, in both the nodes as shown below,
```
<API-M_HOME>\repository\conf\jndi2.properties
```
Add the configuration given below to the JNDI properties file that you just created. Assuming that you are running both API Managers nodes on the same server, the 2nd API Manager node would be running with a port offset of 1. Therefore, the port is given as 5673 below.
```
connectionfactory.TopicConnectionFactory = amqp://admin:admin@clientid/carbon?brokerlist='tcp://localhost:5673'
connectionfactory.QueueConnectionFactory = amqp://admin:admin@clientID/test?brokerlist='tcp://localhost:5673'
topic.throttleData = throttleData
```
Let's create a new JMS Event Publisher by creating a file (for example, jmsEventPublisher2.xml) in the <API-M_HOME>\repository\deployment\server\eventpublishers directory of each of the two nodes.

Add the configuration given below to the JMSEventPublisher file found in both of the nodes. Note that you refer to the JNDI properties file that you created above in the configuration shown below.

<?xml version="1.0" encoding="UTF-8"?>
<eventPublisher name="jmsEventPublisher2" statistics="disable"
  trace="disable" xmlns="http://wso2.org/carbon/eventpublisher">
  <from streamName="org.wso2.throttle.globalThrottle.stream" version="1.0.0"/>
  <mapping customMapping="disable" type="map"/>
  <to eventAdapterType="jms">
    <property name="java.naming.factory.initial">org.wso2.andes.jndi.PropertiesFileInitialContextFactory</property>
    <property name="java.naming.provider.url">repository/conf/jndi2.properties</property>
    <property name="transport.jms.DestinationType">topic</property>
    <property name="transport.jms.Destination">throttleData</property>
    <property name="transport.jms.ConcurrentPublishers">allow</property>
    <property name="transport.jms.ConnectionFactoryJNDIName">TopicConnectionFactory</property>
  </to>
</eventPublisher>

Save your changes.