Production Deployment Guidelines

The requirements for deploying WSO2 products can change based on the deployment scenario and pattern. The recommendations in this topic are for general production use, assuming moderate load conditions. For situations where a high volume of traffic is expected and if there are large deployments, these guidelines may not be sufficient. See Troubleshooting in Production Environments for information on how to obtain and analyze information to solve production issues. The following are the topics addressed in this section.

Installation prerequisites

Prior to installing any WSO2 Carbon-based product, it is necessary to have the appropriate hardware and software for running the product.

System requirements

Environment compatibility

• By default, WSO2 products are installed with OpenJDK, which allows you to run the product as soon as it is installed.

  To use a different JDK, point the JAVA_HOME environment variable to the new JDK. Make sure your JDK version is compatible with the WSO2 product.

• All WSO2 products are generally compatible with most common DBMSs. The embedded H2 database is suitable for development, testing, and some production environments. For most enterprise production environments, however, we recommend you use an industry-standard RDBMS such as Oracle, PostgreSQL, MySQL, MS SQL, etc. For more information, see Working with Databases in the Administration Guide. Also, we do not recommend the H2 database as a user store.
• It is not recommended to use Apache DS in a production environment due to scalability issues. Instead, use an LDAP like OpenLDAP for user management. 
  
• On a production deployment, it is recommended that WSO2 products are installed on latest releases of RedHat Enterprise Linux or Ubuntu Server LTS.
  
• For environments that WSO2 products are tested with, see Compatibility of WSO2 Products.
• If you have difficulty in setting up any WSO2 product in a specific platform or database, contact us.

Installing the WSO2 product

Given below is how to install a WSO2 product:

Download and install the product

If the installation prerequisites are satisfied, follow the steps below:

1. Go to the product page and download the product installer (click Installer pkg).

   Note that there are several options for installing the product in various environments. Use the available links for more information on each option.

2. Double-click to open the installation wizard, which will guide you through the installation. When you finish, the product will be installed and ready for use. 

Access the HOME directory

Let's call the installation location of your product as the <PRODUCT_HOME> directory. This is located in a place specific to your OS as shown below:

Uninstalling the product

To remove an already installed product, follow the instructions below:

sudo bash /Library/WSO2/<PRODUCT_NAME>/<VERSION>/uninstall.sh

sudo apt-get purge <PRODUCT_DISTRIBUTION_NAME>

sudo yum remove <PRODUCT_DISTRIBUTION_NAME>-x86_64

Setting system properties

If you need to set additional system properties when the server starts, you can take the following approaches:

• Set the properties from a script. Setting your system properties in the startup script (i.e. the <PRODUCT_HOME>/bin/wso2server.sh file), is ideal because it ensures that you set the properties every time you start the server. To avoid having to modify the script each time you upgrade, the best approach is to create your own startup script that wraps the WSO2 startup script and adds the properties you want to set, rather than editing the WSO2 startup script directly.

  Be sure to set the org.wso2.ignoreHostnameVerification system property in the <PRODUCT_HOME>/bin/wso2server.sh file to false as follows:

  org.wso2.ignoreHostnameVerification=false 

  This setting will enable hostname verification of HTTP requests and responses in the Carbon server, and thereby avoid security issues in production environments.

• Set the properties from an external registry. If you want to access properties from an external registry, you could create Java code that reads the properties at runtime from that registry. Be sure to store sensitive data such as username and password to connect to the registry in a property file instead of in the Java code and secure the properties file with the secure vault.

You are now ready to run the product.

Running the product

To run WSO2 products, you start the product server at the command line. You can then run the Management Console application to configure and manage the product. 

Before you begin

• When you move into a production environment, it is recommended to grant restricted access to the management console. See Securing Carbon Applications for instructions.
• The config-validation.xml file in the <PRODUCT_HOME>/repository/conf/etc directory contains a list of recommended system parameters, which are validated against your system when the server starts. See Configuring config-validation.xml for details on modifying these parameters before starting the server.
• The Management Console uses the default HTTP-NIO transport, which is configured in the catalina-server.xml file in the <PRODUCT_HOME>/repository/conf/tomcat directory. This transport must be properly configured in this file for the management console to be accessible.

• As explained in the installation prerequisites, the default product installation uses OpenJDK. Therefore, you don't require a different JDK. However, if you have set up Oracle JDK or IBM JDK, be sure to apply the following settings to your product distribution.

Starting the product profiles

Open a command prompt and execute the name of the product distribution (For example, wso2ei-6.30):

<PRODUCT_DISTRIBUTION_NAME>

Stopping the server

To stop the server, press Ctrl+C in the command window, or click the Shutdown/Restart link in the navigation pane in the Management Console.

Running recommendations for security

The following are security related recommendations to be followed when running the product.

• Running as a different user: For security reasons, it's recommended to run the product as an unprivileged user. After adding a user to the system, apply your organizational security policies to that user.
• Running on a different port: If you want to run on a different port, like port 80, the recommended way is to add a port forwarding rule from your firewall.
• Running as a Unix daemon: You have the option of running each product as a standard Unix service. You can start, stop, and restart the WSO2 product instances as follows.
  # sh bin/wso2server.sh [start | stop | restart]

Tuning parameters

• The latency numbers (~50ms) are based on a two datacenter setup with a high-speed network connection. With the default configuration, you might notice intermittent behavior, so it is important to tune the system. 
• It is not recommended to use Metrics for system monitoring (JVM, CPU, etc.) in a production deployment. You can use external monitoring tools for this purpose.

Hazelcast properties

WSO2 products use Hazelcast as its default clustering engine. The following configuration must be placed in the <PRODUCT_HOME>/repository/conf/hazelcast.properties file. Create this file if it does not exist.

hazelcast.shutdownhook.enabled=false
hazelcast.logging.type=log4j

The above configurations are explained below.

• Hazelcast shutdown hook: This configuration disables the shutdown hook in hazelcast, which ensures that the hazelcast instance shuts down gracefully whenever the product node shuts down. If the hazelcast shutdown hook is enabled (which is the default behavior of a product), you will see errors such as "Hazelcast instance is not active!" at the time of shutting down the product node: This is because the hazelcast instance shuts down too early when the shutdown hook is enabled.
• Hazelcast logging type: This configuration sets the hazelcast logging type to log4j, which allows hazelcast logs to be written to the wso2carbon.log file.

Once you enable log4j for hazelcast as explained above, add log4j.logger.com.hazelcast=INFO to the <PRODUCT_HOME>/repository/conf/log4j.properties file. For more information on logging, see Monitoring Logs.

Additionally, Hazelcast indicates that if all members are not mentioned in the well-known member list, there can be a split-brain (network partition) situation. If the cluster spans across data centers, it is important to add all the members to the well-known members list in the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file.

Common guidelines and checklist

The following table lists out the common guidelines and details pertaining to them. These are common to all products and are followed for making an installed WSO2 product ready for production.

<HostName>your.host.name</HostName>

<parameter name="WSDLEPRPrefix" locked="false">[http]://your.host.name:[port]</parameter>

<Ports>
	<JMX>9999</JMX>
</Ports>

-Dhttp.proxyHost=xxxx 
-Dhttp.proxyPort=xxxx

<parameter name="Proxy">
    <Configuration>
    <proxyhost>you.proxy.host</proxyhost>
    <proxyport>your.proxy.port</proxyport>
    </configuration>
</parameter>

Backup and recovery recommendations

None of the WSO2 products persist data in the file systems or retain or generate artifacts. By default, we only store log files in the file system and data and artifacts in the databases and the repository. 

What you should back up

1. Database backups: 
   
   • Back up of all the databases defined in <PRODUCT_HOME>/repository/conf/datasources/master-datasources.xml. 
   • Back up any other databases configured in any files in the <PRODUCT_HOME>/repository/conf/datasources directory. 
2. Artifact backups:
   This includes hot-deployment artifacts, web applications, synapse files, tenant directories, etc. Back up of the <PRODUCT_HOME>/repository directory periodically. The frequency of the back ups depends on your usage. For example, if you are creating or updating APIs daily, take this backup daily.
3. WSO2 product instance backups:
   A one-time-only backup that you take of the entire server directory. This includes all the configuration files, logs, server extensions, and deployment artifacts for both tenants and super tenants. This back up is ideally done when the server is ready to be deployed in a production environment. 

Backup recommendations

We recommend that you use a proper artifact management system such as Puppet to back up and manage your artifacts before deploying them in the WSO2 Carbon runtime. Also, use the WSO2 Update Manager (WUM) tool, which is a command-line utility that allows you to get the latest updates (bug fixes and security fixes) of a particular product release.

Diagram: managing your artifacts using a configuration management system

Recovery recommendations

Be sure to determine the following depending on your business-continuity requirements:

• Recovery Point Objective (RPO): Up to what points are you to recover. This is determined by the latest, known, good point.
• Recovery Time Objective (RTO): How long does it take to recover to the RPO.
• Backup Frequency: How frequently you should take backups. If your RPO is one day, your backup frequency should be daily.
• Disaster Recovery Site: The place where the latest copy of your backup is. This can be from a different shelf in your data center to a completely different geographical location.

We also recommend the following:

1. Align your artifact deployment and recovery processes.
2. Schedule disaster recovery drills to test the recoverability of the system.
3. Test your artifacts in an environment that is identical to the production environment before deploying them into production.
   

Recovery strategy

The following steps include how to recover your setup using the backups:

1. Recover the hot-deployment artifacts by replacing the <PRODUCT_HOME>/repository directory with the backed up copy.
2. Recover the entire WSO2 product by directly replacing the existing WSO2 server directory in the production setup with the backup server directory. This will ensure that all the files, logs, and configurations made to the product do not need to be redone.
3. To recover the databases, follow the recovery strategy recommended by the databases you are using. For information on supported and tested databases, see Tested Database Management Systems.