Distributed Deployment of the Gateway

This topic provides instructions on how to configure the multiple Gateways in WSO2 API Manager (WSO2 API-M) in a distributed deployment to facilitate high availability (HA). The instructions in this topic are based on the Distributed Deployment of API Manager and these configurations will only work if the configurations in that topic are done correctly. For instance, all datasource configurations are already done for the Gateway in that topic and hence do not need to be repeated here.

The following sections provide specific instructions on configuring the Gateway cluster.

Gateway deployment pattern

The configurations in this topic will be done based on the following pattern. This pattern is used as a basic Gateway cluster where the worker nodes and manager nodes are separated.

Step 1 - Configure the load balancer

NGINX is used for this scenario, but you can use any load balancer that you prefer here. The following are the configurations that need to be done for the load balancer.

Use the following steps to configure Nginx as the load balancer for WSO2 products.

1. Install Nginx using the following command.
   $sudo apt-get install nginx

2. Configure Nginx Plus to direct the HTTP requests to the two worker nodes via the HTTP 80 port using the http://am.wso2.com/<service>. To do this, create a VHost file (am.http.conf) in the /etc/nginx/conf.d/ directory and add the following configurations into it.

   upstream wso2.am.com {
           sticky cookie JSESSIONID;
           server xxx.xxx.xxx.xx4:9763;
           server xxx.xxx.xxx.xx5:9763;
   }
   
   server {
           listen 80;
           server_name am.wso2.com;
           location / {
                  proxy_set_header X-Forwarded-Host $host;
                  proxy_set_header X-Forwarded-Server $host;
                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                  proxy_set_header Host $http_host;
                  proxy_read_timeout 5m;
                  proxy_send_timeout 5m;
                  proxy_pass http://wso2.am.com;
           }
   }

3. Configure Nginx Plus to direct the HTTPS requests to the two worker nodes via the HTTPS 443 port using https://am.wso2.com/<service>. To do this, create a VHost file (am.https.conf) in the /etc/nginx/conf.d/ directory and add the following configurations into it.

   upstream ssl.wso2.am.com {
   	sticky cookie JSESSIONID;
   	server xxx.xxx.xxx.xx4:9443;
   	server xxx.xxx.xxx.xx5:9443;
   }
   
   server {
   listen 443;
   	server_name am.wso2.com;
   	ssl on;
   	ssl_certificate /etc/nginx/ssl/wrk.crt;
   	ssl_certificate_key /etc/nginx/ssl/wrk.key;
   	location / {
                  proxy_set_header X-Forwarded-Host $host;
                  proxy_set_header X-Forwarded-Server $host;
                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                  proxy_set_header Host $http_host;
                  proxy_read_timeout 5m;
                  proxy_send_timeout 5m;
   	    proxy_pass https://ssl.wso2.am.com;
       	}
   }

4. Configure Nginx Plus to access the Management Console as  https://mgt.am.wso2.com/carbon via HTTPS 443 port. This is to direct requests to the manager node. To do this, create a VHost file (mgt.am.https.conf) in the /etc/nginx/conf.d/ directory and add the following configurations into it.

   server {
   	listen 443;
   	server_name mgt.am.wso2.com;
   	ssl on;
   	ssl_certificate /etc/nginx/ssl/mgt.crt;
   	ssl_certificate_key /etc/nginx/ssl/mgt.key;
   
   	location / {
                  proxy_set_header X-Forwarded-Host $host;
                  proxy_set_header X-Forwarded-Server $host;
                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                  proxy_set_header Host $http_host;
                  proxy_read_timeout 5m;
                  proxy_send_timeout 5m;
   	    proxy_pass https://xxx.xxx.xxx.xx3:9443/;
       	}
   	error_log  /var/log/nginx/mgt-error.log ;
              access_log  /var/log/nginx/mgt-access.log;
   }

5. Restart the Nginx Plus server.
   $sudo service nginx restart


   Tip: You do not need to restart the server if you are simply making a modification to the VHost file. The following command should be sufficient in such cases.
sudo service nginx reload

Step 2 - Create SSL certificates

Create SSL certificates for both the manager and worker nodes using the instructions that follow. 

1. Create the Server Key.

   $sudo openssl genrsa -des3 -out <key_name>.key 1024

2. Certificate Signing Request.

   $sudo openssl req -new -key <key_name>.key -out server.csr

3. Remove the password.

   $sudo cp server.key <key_name>.key.org 
   $sudo openssl rsa -in <key_name>.key.org -out <key_name>.key

4. Sign your SSL Certificate.

   $sudo openssl x509 -req -days 365 -in server.csr -signkey <key_name>.key -out <certificate_name>.crt

5. Copy the key and certificate files generated in above step 4 to /etc/nginx/ssl/ location. 

You have now configured the load balancer to handle requests sent to am.wso2.com and to distribute the load among the worker nodes in the worker sub-domain of the wso2.am.domain cluster.

Step 3 - Configure the Gateway manager

Management nodes specialize in management of the setup. Only management nodes are authorized to add new artifacts into the system or make configuration changes. Management nodes are usually behind an internal firewall and are exposed to clients running within the organization only. This section involves setting up the Gateway node and enabling it to work with the other components in the distributed setup.

<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
                port="9763"
                proxyPort="80"
--------
/>
<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
                port="9443"
                proxyPort="443"
--------
/>

<GATEWAY-WORKER-IP> am.wso2.com

xxx.xxx.xxx.xx4 am.wso2.com

Step 4 - Configure the Gateway worker

Worker nodes specialize in serving requests to deployment artifacts and and reading them. They can be exposed to external clients.

<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
                port="9763"
                proxyPort="80"
--------
/>
<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
                port="9443"
                proxyPort="443"
--------
/>

<GATEWAY-MANAGER-IP> mgt.am.wso2.com 

xxx.xxx.xxx.xx3 mgt.am.wso2.com 

Step 5 - Optionally configure Hazelcast

You can seamlessly deploy WSO2 API Manager using local caching in a clustered setup without Hazelcast clustering. However, there are edge case scenarios where you need to enable Hazelcast clustering. To identify whether you need Hazelcast clustering and to configure Hazelcast clustering if needed, see Working with Hazelcast Clustering.

Step 6 - Start the Gateway Nodes

Step 6.1 - Start the Gateway manager

1. Comment the following configurations in the <API-M_HOME>/repository/conf/api-manager.xml file.

   <JMSEventPublisherParameters>
      <java.naming.factory.initial>org.wso2.andes.jndi.PropertiesFileInitialContextFactory</java.naming.factory.initial>
      <java.naming.provider.url>repository/conf/jndi.properties</java.naming.provider.url>
      <transport.jms.DestinationType>topic</transport.jms.DestinationType>
      <transport.jms.Destination>throttleData</transport.jms.Destination>
      <transport.jms.ConcurrentPublishers>allow</transport.jms.ConcurrentPublishers>
      <transport.jms.ConnectionFactoryJNDIName>TopicConnectionFactory</transport.jms.ConnectionFactoryJNDIName>
   </JMSEventPublisherParameters>

2. Start the Gateway manager by typing the following command in the terminal.

   sh <GATEWAY_MANAGER_HOME>/bin/wso2server.sh

Step 6.2 - Start the Gateway worker

1. Update the <API-M_HOME>/repository/conf/api-manager.xml file by commenting out the following configurations.

   <JMSEventPublisherParameters>
      <java.naming.factory.initial>org.wso2.andes.jndi.PropertiesFileInitialContextFactory</java.naming.factory.initial>
      <java.naming.provider.url>repository/conf/jndi.properties</java.naming.provider.url>
      <transport.jms.DestinationType>topic</transport.jms.DestinationType>
      <transport.jms.Destination>throttleData</transport.jms.Destination>
      <transport.jms.ConcurrentPublishers>allow</transport.jms.ConcurrentPublishers>
      <transport.jms.ConnectionFactoryJNDIName>TopicConnectionFactory</transport.jms.ConnectionFactoryJNDIName>
   </JMSEventPublisherParameters>

2. Start the Gateway worker by typing the following command in the terminal.

   sh <GATEWAY_WORKER_HOME>/bin/wso2server.sh -Dprofile=gateway-worker

   The additional -Dprofile=gateway-worker argument indicates that this is a worker node specific to the Gateway. This parameter basically makes a server read-only. A node with this parameter will not be able to do any changes such as writing or making modifications to the deployment repository, etc. Starting this node as a Gateway worker ensures that Store and Publisher related functionality is disabled. This parameter also ensures that the node starts as the worker profile, where the UI bundles will not be activated and only the back end bundles will be activated once the server starts up. See API Manager product profiles for more information on the various product profiles available in API Manager.