Governing External References Across Environments

Some artifacts must change based on the environment where the application is deployed. For example, when you deploy an ESB application to Dev, QA, and Production, the service endpoints are different in each of those environments, so you must update the proxy services accordingly. This page provides details on managing environment-specific artifacts across different environments. It focuses specifically on the management of endpoints that are used as external references from within a composite application deployed in different environments.

Services are used from an application. This application is then deployed across various environments: Dev, QA, Integration, etc. For each environment, the service endpoint value is changing and the ESB proxy reference needs to change accordingly.

To attain a comprehensive understanding of this section, please view the VM image from WSO2Con. The VM image provides the pre-configured deployment setup (i.e., the configured ESB and AS servers for Dev and Test environments along with DevStudio for creating the artifacts). This enables you to easily create and deploy the projects on the relevant servers.

Certain sections of this page include detailed steps on setting up the required environment, deploying artifacts using WSO2 Developer Studio, etc. However, in the VM image, these are already setup for your convenience. See here for more information on how to setup the VM.

The following are the topics related to governing external references across environments. These topics are aimed at aiding you to do this from scratch:

Understanding the Users

Users interacting with artifacts in each environment often have different roles and have access to different resources and tools. For example:

Developer: uses Developer Studio (or Eclipse) to create services and composite applications (CApps) and to push project artifacts to a source code repository, such as Subversion (SVN). Typically, the developer has no access to QA or Production resources.
DevOps or Operations team member: uses scripts and the WSO2 Management Console to pull the applications created by the developers from the source code repository and deploy them to the QA and Production environments. These users need to update the endpoints before they deploy in the different environments. Typically, they do not have Developer Studio.

Working with the Server Registry

All WSO2 servers contain an embedded registry, which runs in-process. This registry store the server configurations, as well as artifacts such as endpoints and data sources. The registry is split into three sections: local, governance, and configuration.

local: used to store configurations and runtime data that are local to the server. This partition cannot be shared. Local metadata is stored under _system/local in the registry file system.
configuration: used to store product-specific configurations. This partition can be shared across multiple instances of the same product, which is useful for replicating server configurations. Configuration metadata is stored under _system/config in the registry file system.
governance: used to store configurations and data that is shared across the whole platform. This typically includes services, service description, endpoints, and data sources. Governance metadata is stored under _system/governance in the registry file system.

Mounting to External Registries

While the local section is always local to the server and therefore stored in the embedded registry, you can configure the governance and configuration sections to point to external registries (an approach known as mounting) instead of using the embedded registry. This is particularly useful if the same configuration or governance assets must be shared across a group of server instances. When a registry is shared across multiple WSO2 servers, the servers must be configured to mount their remote registry on the locations listed above. For example:

<mount path="/_system/governance" overwrite="true">
  <instanceId>sharedRegistryInstance</instanceId>
  <targetPath>/_system/governance/branches/qa</targetPath>
</mount>

The target path in the mounting setup above must match the location of the external references. For a tutorial on registry mounting options, see here.

You can browse the contents of the registry from any WSO2 server administration console as shown below:

Referencing Registry Entries

When an application needs to reference an entry in the registry, it uses the naming convention gov:, conf:, and local: to reference the three registry spaces.

Design Assumptions

This page is based on the following design assumptions:

Definitions of external references are stored in the governance space of the registry.
External references are stored in the location:
- _system/governance/trunk for the dev environment,
- _system/governance/branches/<env_name> for other environments, such as _system/governance/branches/qa.

When a registry is shared across multiple WSO2 servers, the servers must be configured to mount their remote registry on the locations listed above, e.g.:

<mount path="/_system/governance" overwrite="true">
	<instanceId>sharedRegistryInstance</instanceId>
	<targetPath>/_system/governance/trunk</targetPath>
</mount>

The target path in the mounting setup above must match the location of the external references. For more information about registry mounting options, see the following tutorials:
Sharing Registry Space Across Multiple Product Instances Online Tutorial and Sharing Registry Space across Multiple Product Instances

Best Practices for Migration

The following are the best practices that allow you to easily migrate applications across environments:

Whenever you need to reference an endpoint, create it in the registry. This registry can either be embedded or stand-alone, as explained above. This approach ensures that the application can be deployed unchanged from one environment to another.

Do not directly reference an endpoint or any external reference inside your application.
Reference endpoints from client applications using a key pointing to the governance registry (for example: gov:/public/endpoints/HelloWorldService). This value is constant across all environments.
Ensure the endpoint definitions are present and accurate in all environments prior to deploying an application using those endpoints. You can either manually edit the endpoint definitions in the registry prior to deploying the application, or make this an automatic part of your deployment process.

Sample Scenario

The rest of this page walks you through a sample scenario of governing external references across environments. It makes the following assumptions:

A simple HelloWorld service is created and hosted in the Application Server, in both the Dev and QA environments.
The respective endpoints URLS for the services are:

- http://10.200.3.92:9764/services/t/dev-as.com/HelloService.HelloServiceHttpSoap12Endpoint/
- http://10.200.3.92:9764/services/t/qa-as.com/HelloService.HelloServiceHttpSoap12Endpoint/

This HelloWorld service is proxied through the ESB.
The HelloWorld service endpoint URL varies from Dev to QA and therefore must be changed when the proxy service is deployed from Dev to QA.
The external references are stored in the governance space in _system/governance/trunk for the dev environment and system/governance/branches/qa for the QA environment.
WSO2 Developer Studio 2.x is used to create the composite application containing WSO2 server artifacts.
WSO2 Governance Registry is used as the central registry/repository for services and server configurations.
WSO2 Application Server is hosting the HelloWorld service.
WSO2 Enterprise Service Bus is hosting the proxy service.
SVN is used as repository and version control of the CApp projects.

Deployment Setup

In terms of deployment, you have several choices available. You may wish to rely on the embedded registry only, or share a registry instance across several environments.

One Registry Per Environment

One registry per environment is the default setup if you use the embedded registry as depicted in the first diagram below or if you decide to assign one registry instance per environment as depicted in the second diagram. In this setup, all external references can be saved in the same registry branch, such as _system/governance/trunk/references, because there is no risk of overlap between the various environments.

One Embedded Registry Per Environment:

One Standalone Registry Per Environment:

Shared Registry Instance Across Multiple Environments

Another option is to share a registry instance among multiple environments, as depicted below. In such as setup, you must store the external references in separate registry branches, such as _system/governance/trunk/references and _system/governance/branches/qa/references. Please see here for more details.

Packaging and Deploying the Application

The application and the definition of the endpoints for all environments are packaged together into a single Maven project. At a high level, the setup is similar to this:

A multi-module Maven project is used to contain all the project information. For more information on Maven multi-model projects, see http://www.sonatype.com/books/mvnex-book/reference/multimodule.html.
Inside the resources folder, one can find the resources for each of the various environments.
1. In the basic setup, environment specific files are kept in the same project, under separate folders.
2. An alternative to this setup is to create one resources project per environment. In such a setup, resources for each environment can be accessed/controlled separately.
3. You can also mix and match and use the same resources project for non-production resources and a separate one for production ones.
Independently from how you are storing resources as explained above, we recommend you create one Composite Application (CApp) for each deployment environment, namely HelloWorldResourcesDev and HellowWorldResourcesQA. This allows you to deploy and manage them separately. Additionally, you need to define a composite application for the application itself, i.e., the proxy services. Each composite application is a Maven module that can be built and deployed separately.

The following is the folder structure used:

HelloWorldApp/
   HelloWorldDynamicResources/
     pom.xml
     Development/
     	endpoints/HelloWorld/HelloWorldServiceEP.xml
    	wsdls/HelloWorld/HelloWorldServiceWSDL.xml
	policies/HelloWorld/Policy.xml
     QA/
  	endpoints/HelloWorld/HelloWorldServiceEP.xml
	wsdls/HelloWorld/HelloWorldServiceWSDL.xml
    	
HelloWorldStaticResources/ 
       pom.xml
       policies/HelloWorld/Policy.xml
 HelloWorldProxyServices/
     	pom.xml
     	src/ 		/* src for the HelloWorld ESB proxy service */
   HelloWorldCApp/
     pom.xml 	/* maven distribution project aggregating
			HelloWorldStaticResources and ProxyService and creates
			HelloWorld.car */
   HelloWorldResourcesDevCApp/
     pom.xml 	/* maven distribution project aggregating Resources
			and creates HelloWorldResourcesDev.car */
   HelloWorldResourcesQACApp/
     pom.xml	/* maven distribution project aggregating Resources
			and creates HelloWorldResourcesQA.car */

Deployment Setup

The deployment setup is as follows:

The HelloWorld Service has been deployed in two different WSO2 App Server tenants, to simulate the Dev and QA versions of the same service.
External reference definitions are stored in the governance space of the registry. The following locations are used to store external references:
1. /_system/governance/trunk for the Dev environment
2. /_system/governance/branches/qa for the QA environment
A WSO2 Governance Registry instance is shared across both ESB servers and has been configured to mount its remote registry on the locations listed above, e.g.:
```
<mount path="/_system/governance" overwrite="true">
	<instanceId>sharedRegistryInstance</instanceId>
	<targetPath>/_system/governance/trunk</targetPath>
</mount>
```
The target path in the mounting setup above must match the location of external references.
WSO2 Developer Studio 3.0.0 is used to create the composite application (CApp) containing WSO2 server artifacts.

Samples

The following samples are described in their respective sections:

Use a Pass-through Proxy Service in ESB to expose services hosted on AS. Services as well as proxy services are deployed across Dev and QA environments. The endpoint value keeps changing for these environments and the ESB proxy service references need to change accordingly.
Use a Secure Proxy Service to Expose Services hosted on AS. The endpoint values and the policies are stored in the registry and referenced by the proxy service. The endpoint values keep changing for Dev and QA environments and the ESB proxy service reference needs to change accordingly.

Building and Deploying Applications in Dev

Starting the Servers

Open a Terminal and start the following servers:

WSO2 Application Server
WSO2 Governance Registry
WSO2 Enterprise Service Bus (relevant server for the Dev environment)

Deploying from Developer Studio

To deploy via Developer Studio, the ESB and G-Reg servers need to be added in Developer Studio.

To add the servers in Developer Studio:

Open the Servers view (if you can't see it, go to Window > Show View).
Right-click on the Server's view window at the bottom of the page and select New > Server and select the relevant Carbon server in the WSO2 group.
Specify the server name and server URL along with the credentials.
The server is then added in Developer Studio.

To deploy the HelloWorldResourcesDevCApp to WSO2 Governance Registry server instance, drag and drop the HelloWorldResourcesDevCApp onto the Governance Registry server that is listed in the Servers view. The application should be automatically deployed. You can monitor the trace in the Terminal from which you started the server.

You can repeat the above steps to deploy the HelloWorldResourcesDevCApp to the Governance Registry and HelloWorldCApp to the ESB servers.

Deploying from the Server Admin Console

Building the CAR Files

To deploy via the server admin console or via Maven, you need to first build the CAR file. This can be built using Developer Studio or either using command line, via Maven.

Do not set the media type for WSDLs and WSDL resources. This is done to ensure that the WSDL handler does not intercept the Registry Put operation.

To build via Developer Studio: right-click on a CApp project and choose Export Carbon Application Project.
To build via Maven: once a project has been checked-out from SVN for example, you can navigate to the HelloWorldCApp project and enter: mvn clean install.

Deploying the CAR Files

To deploy the CAR file, you can either use the administration console or the Maven WSO2 deploy plugin.

From the Administration Console

You can deploy CAR files to any WSO2 server using the administration console. This also applies to the Governance Registry. For example, to deploy the dev resources to the registry, you need to:

Open the administration console.
Login as administrator.
Go to Manage -> Carbon Applications.
Click on Add.
Pick the CAR file you want to deploy and upload the file.

The application is deployed asynchronously (i.e., multiple deployment functions can occur at the simultaneously). Refresh the user interface regularly to check whether it has been uploaded successfully.

Using Maven Deploy

You can use the Maven deploy plugin to push CAR files to servers. To use the Maven deploy plugin, you must first define deployment profiles within the project pom.xml file. This is described in more detail here.

Note: This plugin is new and upgraded often. Make sure to refresh your Maven repository to ensure that you are using the latest version.

Testing the Service

Testing Sample 1: Pass through Proxy Service

Open the Dev ESB Server admin console from the bookmarks by navigating to the admin console URL that is listed in the terminal.
Login as an admin.
Go to Services > List.
Click on Try This Service for the HelloWorldServiceProxy (which you just deployed via the admin console).

In the dialog that appears, use the following request:

<body>
   <p:greet xmlns:p="http://www.wso2.org/types">
      <!--0 to 1 occurrence-->
      <name>YourNameGoesHere</name>
   </p:greet>
</body>

Click Send. You should see a response from the application server as seen in the image below:

Testing Sample 2: Secured Proxy Service

Secured proxy services (not including user name token) cannot be tested using the tryit tool available in the management console.

For this we need to create a Soap UI project to test the proxy service. Soap UI is shipped by default with WSO2 Developer Studio. Click on Window > Open Perspective > Other and select SoapUI from the list in the pop-up window that appears to start using SoapUI.

Details on how to test the proxy service are available at: http://blog.thilinamb.com/2011/02/invoking-secured-web-service-with.html

Editing Resources for QA

To deploy in the QA environment, you can modify the URI and generate the CAR file from Developer Studio or from the command line.

Option 1: Using Developer Studio

Open the HelloWorldServiceEP.xml file under HelloWorldResources/QA and replace the URL with the QA URL: http://192.168.139.1:9765/services/t/qa-as.com/HelloService/.
Right-click on the HelloWorldResourcesQACApp project and choose Export Carbon Application Project.
Provide a location (e.g. /home/wso2user/Desktop) for the output file (the file name is composed of the artifactID combined with the version number).
Follow steps 1 and 2 above for editing the WSDL file for the QA environment.

Option 2: From Command Line

Open a Terminal window and locate the HelloWorldServiceEP.xml file from the Eclipse workspace.
Edit the HelloWorldServiceEP.xml (e.g. using gedit or vi) under HelloWorldResources/QA and replace the URL with the QA one.

... uri="http://192.168.139.1:9765/services/t/test-as.com/HelloService/ "> ...
Go to the root of the HelloWorldResourcesQACApp and type mvn clean install (the first time this is run, you need Internet access to download the Maven repository bundles). This creates the CAR file to be deployed to the Registry.

Note: This step requires an active network connection.
The resulting CAR file can be deployed directly to the registry as mentioned above.

Updating the Endpoint URI Value

Once the endpoint has been created as a registry resource, you can update it in two ways; using the administration console or using the check-in/check-out client. You can also redeploy the CAR file, which overrides the current value.

Using the Administration Console

To edit the QA endpoint, navigate to the QA registry resource (Resources > Browse) in the management console: /_system/governance/branches/qa/endpoints/HelloWorld/HelloWorldServiceEP.xml.
Click on 'Edit as text' and modify the contents of the registry resource by replacing the value of the URI.
To edit the QA WSDL, go to the QA registry resource (Resources > Browse) in the management console: /_system/governance/branches/qa/wsdls/HelloWorld/HelloWorldServiceWSDL.xml
Click on 'Edit as text' and modify the contents of the registry resource by replacing the content with the QA WSDL content.

Using the Check-in Client

You can use the check-in client to check-in/check-out resources from the governance registry, like this:

sh checkin-client.sh  checkout https://localhost:9443/registry/_system/governance/branches/qa/endpoints/HelloWorld -u admin -p admin

The check-in client only supports check-out of collections (i.e. folders, not of individual resources)

To check the file back in after modifying it, use the following command:

sh checkin-client.sh  checkin  -u admin -p admin -l ./HelloWorldServiceEP.xml
https://localhost:9443/registry/_system/governance/qa/prod/endpoints/HelloWorld/HelloWorldServiceEP.xml

Make sure to use the full path, including the filename when checking-in the file!

Note: The check-in client is not available by default with WSO2 servers, only with the WSO2 Governance Registry full product. Please request it from the WSO2 support team if you need to use this tool against an embedded registry.

Using a Script

Alternatively you can have a CAR file with dummy values for the endpoint URLs and use a customized shell script or batch script. The script created would need to do the following:

Extract the CAR file.
Edit the URL values.
Re-create the CAR file with new values.