SVN-Based Deployment Synchronizer

Deployment Synchronizer (DepSync) provides capability to synchronize deployment artifacts across the nodes of a product cluster.

A product's repository typically contains user-uploaded service archives, modules, service metadata, scheduled tasks etc. These artifacts are typically shared across multiples nodes in a cluster. Therefore, in order to maintain consistency in a clustered environment, the artifact repository (in WSO2 products it is <product-home>/repository/deployment/server by default) should be in-sync between all the cluster nodes.

The deployment synchronization activity is shown in the diagram below:

Figure: Deployment synchronization in a cluster

This synchronization functionality is provided by the following feature in the WSO2 feature repository:

Name: WSO2 Carbon - Deployment Synchronizer Feature
Identifier: org.wso2.carbon.deployment.synchronizer.feature.group

In summary, the WSO2 Deployment Synchronizer addresses the following three main use cases:

Maintaining an up-to-date backup of the artifact repository.
Sharing a single artifact repository among multiple servers in a cluster.
Enforcing artifact updates to be deployed on a server at runtime.

Currently, the WSO2 Deployment Synchronizer (DepSync) uses an SVN-based repository for storing shared deployment artifacts. The Subversion repository is used to sync the contents in the sync directory, which by default is the axis2 repo directory <PRODUCT_HOME>/repository/deployment/server. This method is efficient and high-performant. The synchronization process is as follows:

When changes to the contents of axis2 repo directory are present, the Read-Write nodes commit those to the Subversion repository.
Then, this node sends a cluster message to all other (slave) nodes specifying the repo is updated.
When the slave nodes receive the message, they update their axis2 repo directory with what's in the svn repository.

Ghost deployment for better performance

Ghost deployment is enabled in WSO2 DepSync feature, and it ensues that only ghost artifacts are loaded into memory at the time a tenant starts up. Ghost artifacts are light-weight XML references to the actual artifacts. These XML files are stored in <PRODUCT_HOME>/repository/deployment/server, with metadata references to the actual artifacts.

If DepSync updates all artifacts under its SVN repository when a tenant loads, as the number of artifacts in a repository gradually increases, the initial tenant loading time also gets higher. By loading a reference rather than checking the entire SVN repository, ghost deployment significantly improves performance of the the initial tenant loading time. It is faster, more efficient and prevents first-request timeouts and delays.

WSO2 DepSync is optimized to checkout only the repository with depth=empty when a tenant loads. For example,

svn co https://svn.wso2.org/repos/wso2/scratch/carbon-server-framework/--depth=empty

Once the top-level is checked out, the ghost meta files of the actual artifacts are loaded to it. At the time a request is made for an actual artifact, the GhostDisptcher issues a DepSync update of the required artifacts only. For example, if a request is made for a service, GhostDisptcher intercepts this request and then does the update of that service file only using following operations.

svn update axis2services --depth=empty
svn update axis2services/Version.aar

It resumes serving the request after deploying this service. The same scenario for a webapp request is handled by the GhostWebappDeployerValve.

Configuring the DepSync

Described below is how to configure Deployment Synchronizer (DepSync) for a WSO2 product cluster using two cluster nodes (manager, and one worker) with a shared registry (WSO2 Governance Registry). For information on clustering, refer to section Distributed Deployment.

SVN-based deployment synchronizer uses a Subversion repository as the central repository to sync the contents in the sync directory (by default the axis2 repo directory <PRODUCT_HOME>/repository/deployment/server). This method is efficient and transparent to the user. In SVN-based DepSync, when changes to the contents of axis2 repo directory are made, the Read-Write nodes commit those to the Subversion (central) repository. Then, this node sends an Axis2 cluster message to all other (worker) nodes specifying that the repo is updated. When the slave nodes receive the message, they update their axis2 repo directory with what is in the svn repository.

Follow the steps below to setup.

Download and install SVNKit implementation (svnClientBundle-1.0.0.jar) from http://dist.wso2.org/tools/svnClientBundle-1.0.0.jar. It is recommended that you read the svnkit license (http://svnkit.com/license.html). DepSync has the ability to use Subversion packages installed in your local machine if svnkit is unavailable, but it not recommended for production. WSO2 does not ship svnkit by default due to licensing incompatibilities in re-distributions.
You need to have an svn repository. You can use an existing svn repository, or create and setup a new one locally. To create a new svn in Linux, use following the command.
```
svnadmin create /home/test/mylocalsvnrepo
```
The file:// protocol can be used instead of setting up http:// url for the repo (by using a webserver like Apache) if the repo is in local system. For example, you can do a manual checkout of the repository by using the command,
```
svn checkout file:///home/test/mylocalsvnrepo/
```

To configure the Deployment Synchronizer, you have to simply edit the following code block in <PRODUCT_HOME>/repository/conf/carbon.xml file of the product's Master node.

<!-- Deployment Synchronizer Configuration. Uncomment the following section when running with "svn based" dep sync.
    In master nodes you need to set both AutoCommit and AutoCheckout to true 
    and in  worker nodes set only AutoCheckout to true.
    -->
<DeploymentSynchronizer>
        <Enabled>true</Enabled>
        <AutoCommit>true</AutoCommit>
        <AutoCheckout>true</AutoCheckout>
        <RepositoryType>svn</RepositoryType>
        <SvnUrl>file:///home/test/mylocalsvnrepo/</SvnUrl>
        <!-- <SvnUrl>http://svnrepo.example.com/repos/</SvnUrl> --> 
        <SvnUser>username</SvnUser>
        <SvnPassword>password</SvnPassword>
        <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>-->

Auto Commit - Allows to perform commit operations automatically.
Auto Checkout - When this option is selected, an additional option named Use Eventing will be enabled.
Use Eventing - Allows to trigger checkout actions on registry events. This is not needed most of the time.
Synchronization Period - Synchronizer will get initialized and start running periodically as specified in this parameter. Default is 10 seconds.

Add content as the following example to the product's worker node(s).

<DeploymentSynchronizer>
        <Enabled>true</Enabled>
        <AutoCommit>false</AutoCommit>
        <AutoCheckout>true</AutoCheckout>
        <RepositoryType>svn</RepositoryType>
        <SvnUrl>file:///home/test/mylocalsvnrepo/</SvnUrl>
        <!-- <SvnUrl>http://svnrepo.local.com/repos/</SvnUrl> -->                
        <SvnUser>test</SvnUser>
        <SvnPassword>password</SvnPassword>
        <SvnUrlAppendTenantId>false</SvnUrlAppendTenantId>
</DeploymentSynchronizer>

The only differences between the Master and Salve configs are that in slave nodes, we set AutoCommit to false, and AutoCheckout to true.

The Deployment Synchronizer setup is complete. Start the Governance Registry and the two product nodes and examine how uploads to the master node gets synchronized to the worker nodes accordingly.