Deploying WSO2 API-M in a Distributed Setup

Follow the instructions below to deploy WSO2 API Manager (WSO2 API-M) in a distributed environment, as depicted in the following deployment diagram:

Step 1 - Install and configure WSO2 API-M

Click here for information on installing and configuring WSO2 API-M.

The following steps describe how to download, install, and configure WSO2 API Manager, with five instances.

Download the WSO2 API Manager in each of the five servers in the cluster for distributed deployment.
Unzip the WSO2 API Manager zipped archive, and rename each of those directories respectively as Key Manager, Gateway, Publisher, Store, and Traffic Manager.
These five directories are located in a server of their own and are used for each component of WSO2 API-M. Each of these unzipped directories are referred to as <API-M_HOME> or <PRODUCT_HOME> in this document.
In each of the five servers, replace the default certificates (where CN=localhost) with new certificates generated with proper CN values.
You need to do this in order to avoid getting an error with regard to the fact that the hostname in the certificate did not match.

Note that you should use the same primary keystore for all the API Manager instances here in order to decrypt the registry resources. For more information, see Configuring Primary Keystores in the Administration Guide. When creating the keystore, always use a longer validity period so that it will avoid the need of migration on the registry data when shifting to a new keystore.

Make sure to keep the following web apps, which are required, and remove the unnecessary web apps from the <API-M_HOME>/repository/deployment/server/webapps directory of each node.

The following are the web apps required for each node:

Profile	Required web apps
Gateway Manager	`am#sample#pizzashack#v1` (Only needed if the Pizzashack sample API is used) `api#am#admin#v0.11` (Only needed if you want to perform administrative tasks through Gateway Manager) `authenticationendpoint`
Gateway Worker	`am#sample#pizzashack#v1` (Only needed if the Pizzashack sample API is used) `api#am#admin#v0.11` (Only needed if you want to perform administrative tasks through Gateway Manager)
Traffic Manager	`shindig` (The Shindig web app is used for the WSO2 CEP Dashboard)
Key Manager	`authenticationendpoint` `client-registration#v0.11` `oauth2` `throttle#data#v1`
API Publisher	`am#sample#pizzashack#v1` (Only needed if the Pizzashack sample API is used) `api#am#publisher#v0.11` `authenticationendpoint`
API Store (Developer Portal)	`api#am#store#v0.11` `authenticationendpoint`

Step 2 - Install and configure the databases

You can create the required databases for the API-M deployment in a separate server and point to the databases from the respective nodes. For information on configuring the databases, see Installing and Configuring the Databases.

Step 3 - Configure your deployment with production hardening

Ensure that you have taken into account the respective security hardening factors (e.g., changing and encrypting the default passwords, configuring JVM security, etc.) before deploying WSO2 API-M. For more information, see the Production Deployment Guidelines in the Administration Guide.

Step 4 - Create and import SSL certificates

Create a SSL certificate for each of the WSO2 API-M nodes (e.g., Publisher, Store, Key Manager, Gateway, and Traffic Manager) and import them to the keyStore and the trustStore. For more information, see Creating SSL Certificates in the Administration Guide.

When maintaining high availability (HA) in the WSO2 API-M distributed set up, you need to create and import a SSL certificate for each of the WSO2 API-M HA nodes.

Step 5 - Configure API-M Analytics

If you wish to view reports, statistics, and graphs related to the APIs deployed in the Store, you need to configure API-M Analytics. Follow the standard setup to configure API-M Analytics in a production setup, and follow the quick setup to configure API-M Analytics in a development setup.

If you want to configure high availability (HA) for API-M Analytics, see Minimum High Availability Deployment for WSO2 APIM Analytics.

Step 6 - Configure the connections among the components and start the servers

You will now configure the inter-component relationships of the distributed setup by modifying their <API-M_HOME>/repository/conf/api-manager.xml files. It is recommended to start the components in the following order: Key Manager, Publisher, Store, Traffic Manager, and Gateway.

In a clustered environment, you use session affinity (sticky sessions) to ensure that requests from the same client always get routed to the same server.

It is mandatory to set up Session Affinity in the load balancers that front the Publisher and Store clusters, and it is optional in the load balancer (if any) that fronts a Key Manager cluster or Gateway Cluster. However, you need to enable Session Affinity if you are working with multiple Gateway Managers in a Gateway High Availability (HA) deployment.

However, authentication via session ID fails when session affinity is disabled in the load balancer.

First time authentication happens via Basic Auth and the Gateway gets a cookie. This cookie is used in every consequent request along with the Basic Auth credentials. The admin service validates the cookie and if the validation fails it re-authenticates it using Basic Auth and issues a new cookie.

Click here for information on configuring the connections among the components and starting the servers.

Step 6.1 - Configure the common configurations

Carry out the following configurations on all the API-M profiles/components with the exception of the Gateway profile.

Open the <API-M_HOME>/repository/conf/axis2/axis2.xml file and comment out the following.

<transportSender name="ws" class="org.wso2.carbon.websocket.transport.WebsocketTransportSender">
	<parameter name="ws.outflow.dispatch.sequence" locked="false">outflowDispatchSeq</parameter>
	<parameter name="ws.outflow.dispatch.fault.sequence" locked="false">outflowFaultSeq</parameter>
</transportSender>

Delete the WebSocketInboundEndpoint.xml file from the <API-M_HOME>/repository/deployment/server/synapse-configs/default/inbound-endpoints directory.
Make sure to keep the required Jaggery apps and remove the unnecessary Jaggery apps from the <API-M_HOME>/repository/deployment/server/jaggeryapps directory of each node.
The following are the Jaggery apps that are required for each node:
Profile
Required Jaggery apps
Gateway Worker None
Key Manager None
Traffic Manager None
Publisher publisher, admin
Store store
If you are working with the WSO2 WUM update that was effective from the 13th February 2018 (2018-02-13), make the following changes.

Note that you can deploy updates in a production environment only if you have a valid subscription with WSO2. Read more about WSO2 Updates. For more information on updating WSO2 API Manager, see Using WSO2 Update Manager.

Open the <API-M_HOME>/repository/conf/axis2/axis2.xml file and comment out the following.

<!--transportSender name="wss" class="org.wso2.carbon.websocket.transport.WebsocketTransportSender">
	<parameter name="ws.outflow.dispatch.sequence" locked="false">outflowDispatchSeq</parameter>
    <parameter name="ws.outflow.dispatch.fault.sequence" locked="false">outflowFaultSeq</parameter>
    <parameter name="ws.trust.store" locked="false">
    	<ws.trust.store.location>repository/resources/security/client-truststore.jks</ws.trust.store.location>
    	<ws.trust.store.Password>wso2carbon</ws.trust.store.Password>
	</parameter>
</transportSender-->

Delete the SecureWebSocketInboundEndpoint.xml file from the <API-M_HOME>/repository/deployment/server/synapse-configs/default/inbound-endpoints directory.

When a node starts, it starts all the components and featured bundled with it. If you are concerned about resource utilization, you can run the product on a specific profile, so that only the components and features that are required for that node and common features start up. Click the tabs below to see the startup commands for each profile.

For more information on using multi-profile support, see Product Profiles.

Step 6.2 - Configure and start the Key Manager

This section involves setting up the Key Manager node and enabling it to work with the other components in a distributed deployment.

Skip this step if you are using WSO2 Identity Server as the Key Manager and follow the instructions mentioned in Configuring WSO2 Identity Server as a Key Manager to configure and start the Key Manager.

Open the <API-M_HOME>/repository/conf/api-manager.xml file in the Key Manager node and change the <ServerURL> element that appears under the <APIGateway> section, so that it points to the API Manager Gateway.
You need to add these configurations so that when a user is deleted or when the role of a user is updated in the Key Manager, it will update the Gateway cache by clearing the cache entries of a particular user.
- If you are working with a single Gateway in distributed set up, you need to replace [GATEWAY_SERVER_HOST] with the host of the Gateway node.
- If you are working with Gateways in a High Availability (HA) setup that uses a shared file system (e.g., NFS), you need to replace [GATEWAY_SERVER_HOST] with the host of the Gateway load balancer node.
- If you are working with Gateways in a High Availability (HA) setup that uses rsync, you need to replace [GATEWAY_SERVER_HOST] with the host of the Gateway Manager node.
- You need to replace [port] with the management transport port. For more information, see Default Product Ports.
```
<ServerURL>https://$[GATEWAY_SERVER_HOST]:[port]/services/</ServerURL>
```
Configure the API key validator in the Key Manager.
The Thrift protocol is normally enabled by default. However, if you have disabled the Thrift protocol, enable it as follows in the <API-M_HOME>/repository/conf/api-manager.xml file.

If you wish to encrypt the Auth Keys (access tokens, client secrets, and authorization codes), see Encrypting OAuth Keys.
Disable the data publisher in the Key Manager.
As the Key Manager does not need to publish to the Traffic Manager, you need to disable the data publisher as follows:
Open the <API-M_HOME>/repository/conf/api-manager.xml file in the Key Manager, and change value of the <Enabled> element that appears under the <DataPublisher> element to false.
```
<ThrottlingConfigurations> 
    ...
    <DataPublisher>
           <Enabled>false</Enabled>
          ...
    </DataPublisher>
    ...
</ThrottlingConfigurations>
```
Optionally, configure High Availability (HA) for the Key Manager.

These steps are ONLY applicable if you need to configure HA for the Key Manager.
1. Make a copy of the active instance configured above and use this copy as the second Key Manager active instance.
2. Configure a load balancer to front the two Key Manager nodes.
  For information on configuring the load balancer, see Configuring the Proxy Server and the Load Balancer.
Start the WSO2 API-M Key Manager node(s) by typing the following command in the command prompt. For more information on starting a WSO2 server, see Starting the server.

It is not recommended to share the Solr directory between the Store and Publisher servers. You need to have separate Solr directories for each of the latter mentioned servers so that they will perform Solr indexing separately.

If you get an error similar to the following in both or one of the nodes, check whether you have shared the Solr directory.

org.apache.solr.common.SolrException: 
SolrCore 'registry-indexing' is not available due to init failure: Index locked for write for core registry-indexing

Step 6.3 - Configure and start the API Publisher

This section involves setting up the API Publisher node and enabling it to work with the other components in the distributed deployment.

Open the <API-M_HOME>/repository/conf/api-manager.xml file in the API Publisher node and make the following changes.
1. Configure the Publisher with the Key Manager.
  You need to update the following configuration ONLY when you do not wish to share the user stores with the WSO2 API-M instance.
  
  This step is not applicable if you are enabling Single Sign-on (SSO).
2. Configure the Publisher with the Traffic Manager.
  This configuration enables the publishing of throttling policies, custom templates, and block conditions to the Gateway node.
3. Configure the Publisher with the Gateway.
  You need to add these configurations, because when creating an API, it calls the Gateway endpoint to create the actual Synapse file.
  - If you are using a single Gateway node, configure the Publisher with the Gateway as follows:
    <APIGateway> <Environments> <Environment type="hybrid" api-console="true"> <Name>Production and Sandbox</Name> <Description>This is a hybrid gateway that handles both production and sandbox token traffic.</Description> <ServerURL>https://[API-Gateway-Host-or-IP]:9443/services/</ServerURL> <Username>${admin.username}</Username> <Password>${admin.password}</Password> <GatewayEndpoint>http://[API-Gateway-Host]:8280,https://[API-Gateway-Host]:8243</GatewayEndpoint> </Environment> </Environments> </APIGateway>
  - If you are using multiple Gateway nodes, configure the Publisher with the Gateway nodes as follows:
  To change the admin password, see Changing the super admin password. Furthermore, make sure to adhere to the note given under step 2 in the latter mentioned section if your password has special characters.
4. Configure the Store URL to appear in the Publisher UI.
  For this purpose you need to set the <DisplayURL> to true and provide the URL of the Store.
Configure the blocked apps and API notifications to go to the Topic.
Open the <API-M_HOME>/repository/conf/jndi.properties file and make the following changes.
- The following configuration is related to the Admin App in WSO2 API-M. In this guide it is assumed that the WSO2 API-M Admin App is configured in the Publisher node.
- If you change the default username (i.e., admin) and password (i.e., admin) in the user-mgt.xml file, that username and password should be changed at the broker connection URL as well.
```
connectionfactory.TopicConnectionFactory = amqp://admin:admin@clientid/carbon?brokerlist='tcp://[Traffic-Manager-host]:5672'
topic.throttleData = throttleData
```
Disable the Thrift Server to optimize performance.
You need to configure this in the Publisher <API-M_HOME>/repository/conf/api-manager.xml file.
```
<APIKeyValidator> 
...  
	<EnableThriftServer>false</EnableThriftServer>
</APIKeyValidator>
```
Optionally, configure High Availability (HA) for the Publisher.

These instructions are ONLY applicable if you need to configure HA for the Publisher.
1. Make a copy of the active Publisher instance configured above and use this copy as the second active Publisher instance.
2. Configure a load balancer to front the two Publisher nodes.
  For information on configuring the load balancer, see Configuring the Proxy Server and the Load Balancer.
Start the WSO2 API-M Publisher node(s) by typing the following command in the command prompt.
For more information on starting a WSO2 server, see Starting the server.

Step 6.4 - Configure and start the API Store

This section involves setting up the API Store node and enabling it to work with the other components in the distributed deployment.

Open the <API-M_HOME>/repository/conf/api-manager.xml file in the API Store node and make the following changes.
1. Configure the API Store with the Key Manager.
2. Make the following throttling related changes that correspond to the Traffic Manager.
```
<ThrottlingConfigurations>
        <EnableAdvanceThrottling>true</EnableAdvanceThrottling>
        <DataPublisher>
            <Enabled>false</Enabled>
	    ……………………
        </DataPublisher> 
	    …………………
        <BlockCondition>
            <Enabled>false</Enabled>
	    ………………………
        </BlockCondition>
        <JMSConnectionDetails>
            <Enabled>false</Enabled>
	     …………………………………
        </JMSConnectionDetails>
     ………………………………
</ThrottlingConfigurations>
```
3. Configure the Store with the Gateway.
  - If you are using a single Gateway node, configure the Store with the Gateway as follows:
    <APIGateway> <Environments> <Environment type="hybrid"> ... <ServerURL>https://[API-Gateway-host-or-IP]:9443/services/</ServerURL> <Username>${admin.username}</Username> <Password>${admin.password}</Password> <GatewayEndpoint>http://[API-Gateway-host]:8280,https://[API-Gateway-host]:8243</GatewayEndpoint> </Environment> </Environments> ... </APIGateway>
  - If you are using multiple Gateway nodes, configure the Store with the Gateway nodes as follows:
4. Configure the Token Revoke endpoint to point to Gateway.
Disable the Thrift Server to optimize performance.
You need to configure this in the Store <API-M_HOME>/repository/conf/api-manager.xml file.
```
<APIKeyValidator> 
...  
	<EnableThriftServer>false</EnableThriftServer>
</APIKeyValidator>
```
Optionally, configure High Availability (HA) for the Store.

This is ONLY applicable if you need to configure HA for the Store.
Make a copy of the active instance configured above and use this copy as the second API Store active instance.
Start the API Store node(s) by typing the following command in the command prompt. For more information on starting a WSO2 server, see Starting the server.

Step 6.5 - Configure and start the Traffic Manager

This section involves setting up the Traffic Manager node(s) and enabling it to work with the other components in a distributed deployment.

Delete the <API-M_HOME>/repository/conf/registry.xml file and rename the <API-M_HOME>/repository/conf/registry_TM.xml file as the registry.xml file.
To disable registry indexing when setting up the Traffic Manager, see Registry indexing configurations.
Delete the <API-M_HOME>/repository/conf/axis2/axis2.xml file and rename the <API-M_HOME>/repository/conf/axis2/axis2_TM.xml file as the axis2.xmlfile.
As mentioned in step 1 (4), make sure to remove all the existing webapps in the <API-M_HOME>/repository/deployment/server/webapps directory with the exception of the shindig web application.
Optionally, mount the <API-M_HOME>/repository/deployment/server directory of all the Traffic Manager nodes to the shared file system.

This step is ONLY applicable if you are configuring the Traffic Manager with HA and shared file system as the content synchronization mechanism.
You need to do this to share all the Throttling policies between traffic management nodes.
Disable the Thrift Server to optimize performance.
You need to configure this in the TrafficManager<API-M_HOME>/repository/conf/api-manager.xml file.
```
<APIKeyValidator> 
...  
	<EnableThriftServer>false</EnableThriftServer>
</APIKeyValidator>
```
Optionally, configure High Availability (HA) for the Traffic Manager.

This is ONLY applicable if you need to configure HA for the Traffic Manager.
Make a copy of the active instance configured above and use this copy as the second active Traffic Manager instance.
Start the WSO2 API-M Traffic Manager node(s) by typing the following command in the command prompt. For more information on starting a WSO2 server, see Starting the server.
Always start the Traffic Manager using the -Dprofile=traffic-manager profile to avoid FATAL errors such as the following.
FATAL - ServiceBusInitializer Couldn't initialize the ESB... org.apache.synapse.SynapseException: The synapse.xml location ././ ./repository/deployment/server/synapse-configs /default doesn't exist

Troubleshooting

If you have a firewall between the Traffic Manager and the Gateway, you need to configure the heartbeat value to keep the JMS connection alive. To configure this, open the <APIM_HOME>/repository/conf/advanced/qpid-config.xml file and set the heartbeat to a non-zero value as shown below.

<heartbeat>    
	<delay>60</delay>
    <timeoutFactor>2.0</timeoutFactor>
</heartbeat>

Step 6.6 - Configure and start the Gateway

This section involves setting up the Gateway node and enabling it to work with the other components in the distributed deployment.

Steps 1 to 5 in the following section are common irrespective of your API-M deployment, such as deploying a single Gateway node or deploying multiple Gateway nodes for High Availability (HA). However, if you are using two Gateway nodes for high availability (HA), first follow the instructions that is available in the Distributed Deployment of the Gateway document, and then carry out the following steps to configure the connections from Gateway(s) to other components.

Open the <API-M_HOME>/repository/conf/api-manager.xml file in the Gateway node.
Modify the api-manager.xml file as follows. This configures the connection to the Key Manager component.
If you need to enable JSON Web Token (JWT), you have to enable it in all Gateway and Key Manager components.
For more information on configuring JWT, see Generating JSON Web Token.
Configure the Gateway to communicate with the Traffic Manager.
You need to do this to enable Throttling for the Traffic Manager node(s).

These configurations vary based on whether you have a single Traffic Manager node or multiple Traffic Manager nodes.

Comment out the following section to prevent warning messages during start up of the Gateway node in the cluster setup.

<JMSEventPublisherParameters>
	<java.naming.factory.initial>org.wso2.andes.jndi.PropertiesFileInitialContextFactory</java.naming.factory.initial>
	<java.naming.provider.url>repository/conf/jndi.properties</java.naming.provider.url>
	<transport.jms.DestinationType>topic</transport.jms.DestinationType>
	<transport.jms.Destination>throttleData</transport.jms.Destination>
	<transport.jms.ConcurrentPublishers>allow</transport.jms.ConcurrentPublishers>
	<transport.jms.ConnectionFactoryJNDIName>TopicConnectionFactory</transport.jms.ConnectionFactoryJNDIName>
</JMSEventPublisherParameters>

By default, WSO2 API Manager is shipped with a keystore (wso2carbon.jks) and a trust store (client-truststore.jks). For more information on how to create a new key store and a trust store with a private key and a self-signed certificate, see Configuring Keystore and Truststore and also see the Administration guide for recommendations on setting up keystores in WSO2 products.

Start the WSO2 API-M Gateway node by typing the following command in the command prompt. For more information on starting a WSO2 server, see Starting the server.

Profile	Required Jaggery apps
Gateway Worker	None
Key Manager	None
Traffic Manager	None
Publisher	publisher, admin
Store	store