This site contains the documentation that is relevant to older WSO2 product versions and offerings.
For the latest WSO2 documentation, visit https://wso2.com/documentation/.

Deployment Guidelines in Production

Follow the guidelines below to deploy Identity Server in production. 

The following changes should be applied on a fresh Identity Server instance. Do not start the Identity Server until the configurations are finalized.

The requirements for deploying WSO2 products can change based on the deployment scenario and pattern. The recommendations in this topic are for general production use, assuming moderate load conditions. For situations where a high volume of traffic is expected and if there are large deployments, these guidelines may not be sufficient. See Troubleshooting in Production Environments for information on how to obtain and analyze information to solve production issues. The following are the topics addressed in this section.



Installation prerequisites

Prior to installing any WSO2 Carbon-based product, it is necessary to have the appropriate hardware and software for running the product.

System requirements

Physical
  • 3 GHz Dual-core Xeon/Opteron (or latest)
  • 4 GB RAM (2 GB for JVM and 2 GB for the operating system)
  • 10 GB free disk space
  • ~ Recommended minimum - 2 Cores. For high concurrencies and better performances - 4 Cores.

Disk space is based on the expected storage requirements that are calculated by considering the file uploads and the backup policies. For example, if three WSO2 product instances are running in a single machine, it requires a 4 GHz CPU, 8 GB RAM (2 GB for the operating system and 6 GB (2 GB for each WSO2 product instance)) and 30 GB of free space.

Virtual Machine (VM)
  • 2 compute units minimum (each unit having 1.0-1.2 GHz Opteron/Xeon processor)
  • 4 GB RAM
  • 10 GB free disk space
  • One CPU unit for the operating system and one for JVM.

Three WSO2 product instances running would require VM of 4 compute units, 8 GB RAM, and 30 GBfree space.
~ 512 MB heap size. This is generally sufficient to process typical SOAP messages but the requirements vary with larger message sizes and the number of messages processed concurrently.

EC2
  • One c5.large instance to run one WSO2 product instance.
Cassandra data nodes
  • 4 core processors
  • 8 GB RAM

For more information, see the Cassandra documentation on hardware recommendations for enterprise implementations.


Environment compatibility

  • By default, WSO2 products are installed with OpenJDK, which allows you to run the product as soon as it is installed.

    To use a different JDK, point the JAVA_HOME environment variable to the new JDK. Make sure your JDK version is compatible with the WSO2 product.

  • All WSO2 products are generally compatible with most common DBMSs. The embedded H2 database is suitable for development, testing, and some production environments. For most enterprise production environments, however, we recommend you use an industry-standard RDBMS such as Oracle, PostgreSQL, MySQL, MS SQL, etc. For more information, see Working with Databases in the Administration Guide. Also, we do not recommend the H2 database as a user store.
  • It is not recommended to use Apache DS in a production environment due to scalability issues. Instead, use an LDAP like OpenLDAP for user management. 
  • On a production deployment, it is recommended that WSO2 products are installed on latest releases of RedHat Enterprise Linux or Ubuntu Server LTS.
  • For environments that WSO2 products are tested with, see Compatibility of WSO2 Products.
  • If you have difficulty in setting up any WSO2 product in a specific platform or database, contact us.

Installing the WSO2 product

Given below is how to install a WSO2 product:

Download and install the product

If the installation prerequisites are satisfied, follow the steps below:

  1. Go to the product page and download the product installer (click Installer pkg).

    Note that there are several options for installing the product in various environments. Use the available links for more information on each option.

  2. Double-click to open the installation wizard, which will guide you through the installation. When you finish, the product will be installed and ready for use. 

Access the HOME directory

Let's call the installation location of your product as the <PRODUCT_HOME> directory. This is located in a place specific to your OS as shown below:

OSHome directory
Mac OS/Library/WSO2/<PRODUCT_NAME>/<VERSION>
WindowsC:\Program Files\WSO2\<PRODUCT_NAME>/<VERSION>\
Ubuntu/usr/lib/wso2/<PRODUCT_NAME>/<VERSION>
CentOS/usr/lib64/<PRODUCT_NAME>/<VERSION>

Uninstalling the product

To remove an already installed product, follow the instructions below:

OSInstructions
Mac OS

Open a terminal and run the following command as the root user:

sudo bash /Library/WSO2/<PRODUCT_NAME>/<VERSION>/uninstall.sh
WindowsGo to the Start Menu -> Programs -> WSO2 -> Uninstall <PRODUCT_NAME_VERSION> or search Uninstall <PRODUCT_NAME_VERSION> and click the shortcut icon. This will uninstall the product from your computer.
Ubuntu

Open a terminal and run the following command:

sudo apt-get purge <PRODUCT_DISTRIBUTION_NAME>
CentOS

Open a terminal and run the following command:

sudo yum remove <PRODUCT_DISTRIBUTION_NAME>-x86_64

Setting system properties

If you need to set additional system properties when the server starts, you can take the following approaches:

  • Set the properties from a script. Setting your system properties in the startup script (i.e. the <PRODUCT_HOME>/bin/wso2server.sh file), is ideal because it ensures that you set the properties every time you start the server. To avoid having to modify the script each time you upgrade, the best approach is to create your own startup script that wraps the WSO2 startup script and adds the properties you want to set, rather than editing the WSO2 startup script directly.

    Be sure to set the org.wso2.ignoreHostnameVerification system property in the <PRODUCT_HOME>/bin/wso2server.sh file to false as follows:

    org.wso2.ignoreHostnameVerification=false 

    This setting will enable hostname verification of HTTP requests and responses in the Carbon server, and thereby avoid security issues in production environments.

  • Set the properties from an external registry. If you want to access properties from an external registry, you could create Java code that reads the properties at runtime from that registry. Be sure to store sensitive data such as username and password to connect to the registry in a property file instead of in the Java code and secure the properties file with the secure vault.

Note: When using SUSE Linux, it ignores /etc/resolv.conf and only looks at the /etc/hosts file. This means that the server will throw an exception on startup if you have not specified anything besides localhost. To avoid this error, add the following line above 127.0.0.1 localhost in the /etc/hosts file: <ip_address>  <machine_name> localhost

You are now ready to run the product.


Running the product

To run WSO2 products, you start the product server at the command line. You can then run the Management Console application to configure and manage the product. 

Before you begin

  • When you move into a production environment, it is recommended to grant restricted access to the management console. See Securing Carbon Applications for instructions.
  • The config-validation.xml file in the <PRODUCT_HOME>/repository/conf/etc directory contains a list of recommended system parameters, which are validated against your system when the server starts. See Configuring config-validation.xml for details on modifying these parameters before starting the server.
  • The Management Console uses the default HTTP-NIO transport, which is configured in the catalina-server.xml file in the <PRODUCT_HOME>/repository/conf/tomcat directory. This transport must be properly configured in this file for the management console to be accessible.
  • As explained in the installation prerequisites, the default product installation uses OpenJDK. Therefore, you don't require a different JDK. However, if you have set up Oracle JDK or IBM JDK, be sure to apply the following settings to your product distribution.

Starting the product profiles

Open a command prompt and execute the name of the product distribution (For example, wso2ei-6.30):

<PRODUCT_DISTRIBUTION_NAME>

Stopping the server

To stop the server, press Ctrl+C in the command window, or click the Shutdown/Restart link in the navigation pane in the Management Console.

Running recommendations for security

The following are security related recommendations to be followed when running the product.

  • Running as a different user: For security reasons, it's recommended to run the product as an unprivileged user. After adding a user to the system, apply your organizational security policies to that user.
  • Running on a different port: If you want to run on a different port, like port 80, the recommended way is to add a port forwarding rule from your firewall.
  • Running as a Unix daemon: You have the option of running each product as a standard Unix service. You can start, stop, and restart the WSO2 product instances as follows.
    # sh bin/wso2server.sh [start | stop | restart]

Tuning parameters

  • The latency numbers (~50ms) are based on a two datacenter setup with a high-speed network connection. With the default configuration, you might notice intermittent behavior, so it is important to tune the system. 
  • It is not recommended to use Metrics for system monitoring (JVM, CPU, etc.) in a production deployment. You can use external monitoring tools for this purpose.

Hazelcast properties

WSO2 products use Hazelcast as its default clustering engine. The following configuration must be placed in the <PRODUCT_HOME>/repository/conf/hazelcast.properties file. Create this file if it does not exist.

hazelcast.shutdownhook.enabled=false
hazelcast.logging.type=log4j

The above configurations are explained below.

  • Hazelcast shutdown hook: This configuration disables the shutdown hook in hazelcast, which ensures that the hazelcast instance shuts down gracefully whenever the product node shuts down. If the hazelcast shutdown hook is enabled (which is the default behavior of a product), you will see errors such as "Hazelcast instance is not active!" at the time of shutting down the product node: This is because the hazelcast instance shuts down too early when the shutdown hook is enabled.
  • Hazelcast logging type: This configuration sets the hazelcast logging type to log4j, which allows hazelcast logs to be written to the wso2carbon.log file.

Once you enable log4j for hazelcast as explained above, add log4j.logger.com.hazelcast=INFO to the <PRODUCT_HOME>/repository/conf/log4j.properties file. For more information on logging, see Monitoring Logs.

Additionally, Hazelcast indicates that if all members are not mentioned in the well-known member list, there can be a split-brain (network partition) situation. If the cluster spans across data centers, it is important to add all the members to the well-known members list in the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file.


Common guidelines and checklist

The following table lists out the common guidelines and details pertaining to them. These are common to all products and are followed for making an installed WSO2 product ready for production.

GuidelineDetails
Security hardening

Guidelines for hardening the security of a WSO2 deployment in a production environment can be discussed under three high-level categories:

  • Product-level security
  • OS-level security
  • Network-level security

Related links

See Security Guidelines for Production Deployment for the detailed list of security-related recommendations.

Hostname

By default, WSO2 products identify the hostname of the current machine through the Java API. However, this value sometimes yields erroneous results on some environments. Therefore, users are recommended to configure the hostname by setting the HostName parameter in the <PRODUCT_HOME>/repository/conf/carbon.xml file.

<HostName>your.host.name</HostName>

To configure hostnames for WSDLs and endpoints, users are recommended to add the following parameter in the <transportReceiver> section in the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file as shown below.

<parameter name="WSDLEPRPrefix" locked="false">[http]://your.host.name:[port]</parameter>
Related links
Registry and governance

All WSO2 products make use of an instance of a registry to store configurations. The registry uses a database as the persistent storage. By default, the registry uses an embedded H2 database.

This embedded database might yield a lower performance and is less reliable compared to a standard database like MySQL when there are a large number of deployed artifacts. Hence, you should look at associated trade-offs, and we recommend that you switch to a database like Oracle, MySQL or MSSQL.

Moreover, it is worth noting that the default setup does not include database backup procedures. The production setup should obviously need to have regular database backup procedures configured.

When the registry database is pointed to a remote database, multiple running instances of the same product can boot up and run against the same configuration stored in the registry. This, in turn, helps with governance.

Related links

See here for more information on sharing a registry space across multiple WSO2 product instances.

User stores

WSO2 products offer three choices to store user details:

  • Using a database
  • Using an LDAP server
  • Using an Active Directory service

The default is to use the embedded H2 database, with the user store schema. Like in the case of the registry database, you can switch to a database like Oracle, MySQL or MSSQL. You can point to an existing LDAP or an Active Directory to make use of existing user bases and grant access privileges for WSO2 products based on those user stores.

Related links

See Configuring User Stores for more information on user stores, how they work, and how to configure them.

Monitoring with JMX

WSO2 Products supportJMXformonitoring. By default, JMX uses port 9999. You can configure this to the desired port by setting the JMX port parameter in the <PRODUCT_HOME>/repository/conf/carbon.xml file.

<Ports>
	<JMX>9999</JMX>
</Ports>
Related links

See JMX-Based Monitoring for information on monitoring WSO2 products using JMX.

Tuning WSO2 products

Most of the performance tuning recommendations are common to all WSO2 products. However, each WSO2 product may have additional guidelines for optimizing the performance of product-specific features.

Related links
  • See Performance Tuning for the general guidelines, which are common to all WSO2 products.
  • For performance tuning guidelines that are specific to each product, go to the product documentation for each product listed below and search for performance tuning guidelines.

    Listed below are the main WSO2 products:

The following are now legacy products of WSO2:

Firewalls

The following ports must be accessed when operating within a firewall.

  • 9443 - Used by the management console and services that use the servlet transport, and is defined in the <PRODUCT_HOME>/repository/conf/tomcat/catalina-server.xml file.
  • 9763 - Used by the services that use servlet transport, and is defined in the <PRODUCT_HOME>/repository/conf/tomcat/catalina-server.xml file.
  • 9999 - Used for JMX monitoring, defined in the <PRODUCT_HOME>/repository/conf/carbon.xml file.
  • 8280 - Default HTTP port used by ESB for proxy services, and is defined in the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file.
  • 8243 - Default HTTPS port used by ESB for proxy services, and is defined in the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file.
Related links

See Default Ports of WSO2 Products for a list of common and product-specific ports used by WSO2 products.

Proxy servers

If the product is hosted behind a proxy such as ApacheHTTPD, users can configure products to use the proxy server by providing the following system properties at the start-up.

-Dhttp.proxyHost=xxxx 
-Dhttp.proxyPort=xxxx

Alternatively, this can be done by adding the following configurations in the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file.

<parameter name="Proxy">
    <Configuration>
    <proxyhost>you.proxy.host</proxyhost>
    <proxyport>your.proxy.port</proxyport>
    </configuration>
</parameter>
High availability

For high availability, WSO2 products must run on a cluster. This enables the WSO2 products to still work in the case of failover. Databases used for the repository, user management, and business activity monitoring can also be configured in a cluster or can use replication management provided by the RDBMS.

Related links
  • See Overview for more details on clustering, what it is, how it helps and other related information.
  • See Separating the Worker and Manager Nodes for information on clustering WSO2 products by separating worker and manager concerns between the nodes.
Data backup and archivingFor data backup and for archiving of data, use the functionality provided by the RDBMS.



Backup and recovery recommendations

None of the WSO2 products persist data in the file systems or retain or generate artifacts. By default, we only store log files in the file system and data and artifacts in the databases and the repository. 

What you should back up

  1. Database backups
    • Back up of all the databases defined in <PRODUCT_HOME>/repository/conf/datasources/master-datasources.xml
    • Back up any other databases configured in any files in the <PRODUCT_HOME>/repository/conf/datasources directory. 
  2. Artifact backups:
    T
    his includes hot-deployment artifacts, web applications, synapse files, tenant directories, etc. Back up of the <PRODUCT_HOME>/repository directory periodically. The frequency of the back ups depends on your usage. For example, if you are creating or updating APIs daily, take this backup daily.
  3. WSO2 product instance backups:
    A one-time-only backup that you take of the entire server directory. This includes all the configuration files, logs, server extensions, and deployment artifacts for both tenants and super tenants. This back up is ideally done when the server is ready to be deployed in a production environment. 

Backup recommendations

We recommend that you use a proper artifact management system such as Puppet to back up and manage your artifacts before deploying them in the WSO2 Carbon runtime. Also, use the WSO2 Update Manager (WUM) tool, which is a command-line utility that allows you to get the latest updates (bug fixes and security fixes) of a particular product release.

Diagram: managing your artifacts using a configuration management system

Recovery recommendations

Be sure to determine the following depending on your business-continuity requirements:

  • Recovery Point Objective (RPO): Up to what points are you to recover. This is determined by the latest, known, good point.
  • Recovery Time Objective (RTO): How long does it take to recover to the RPO.
  • Backup Frequency: How frequently you should take backups. If your RPO is one day, your backup frequency should be daily.
  • Disaster Recovery Site: The place where the latest copy of your backup is. This can be from a different shelf in your data center to a completely different geographical location.

We also recommend the following:

  1. Align your artifact deployment and recovery processes.
  2. Schedule disaster recovery drills to test the recoverability of the system.
  3. Test your artifacts in an environment that is identical to the production environment before deploying them into production.

Recovery strategy

The following steps include how to recover your setup using the backups:

  1. Recover the hot-deployment artifacts by replacing the <PRODUCT_HOME>/repository directory with the backed up copy.
  2. Recover the entire WSO2 product by directly replacing the existing WSO2 server directory in the production setup with the backup server directory. This will ensure that all the files, logs, and configurations made to the product do not need to be redone.
  3. To recover the databases, follow the recovery strategy recommended by the databases you are using. For information on supported and tested databases, see Tested Database Management Systems.