VFS Transport

The Virtual File System (VFS) transport is used by WSO2 Enterprise Integrator(WSO2 EI) to process files in the specified source directory. After processing the files, it moves them to a specified location or deletes them. Note that files cannot remain in the source directory after processing or they will be processed again, so if you need to maintain these files or keep track of which files have been processed, specify the option to move them instead of deleting them after processing. If you want to move files into a database, use the VFS transport and the DBReport mediator (for an example, see Sample 271: File Processing).

Note that the VFS transport is enabled by default from WSO2 EI version 6.5.0 onwards.

Tip: When you transfer a file to a remote FTP location via VFS, the ESB tries to detect the FTP location by navigating from the root folder first. If the ESB does not have at least list permission to the root (/), the file transfer fails.

Configuring the endpoint

To configure a VFS endpoint, use the vfs:file prefix in the URI. For example:

<endpoint>
    <address uri="vfs:file:///home/user/test/out"/>
</endpoint>

Security

The VFS transport supports the SFTP protocol with Secure Sockets Layer (SSL). The configuration is identical to other protocols with the only difference being the URL prefixes and parameters. For more information, see VFS URL parameters below.

Securing VFS password in a proxy service

The following instructions describe how to secure the VFS password in a proxy service configuration.

Provide configuration for the password decryption by adding the following lines in <EI_HOME>/conf/axis2/axis2.xml file:

<transportReceiver class="org.apache.synapse.transport.vfs.VFSTransportListener" name="vfs">
<parameter locked="false" name="keystore.identity.location">repository/resources/security/wso2carbon.jks</parameter>
<parameter locked="false" name="keystore.identity.type">JKS</parameter>
<parameter locked="false" name="keystore.identity.store.password">wso2carbon</parameter>
<parameter locked="false" name="keystore.identity.key.password">wso2carbon</parameter>
<parameter locked="false" name="keystore.identity.alias">wso2carbon</parameter>
</transportReceiver>

If you need to secure the passwords provided in the above configuration, you can do so using secure vault. The secured configuration would look like this:

<transportReceiver class="org.apache.synapse.transport.vfs.VFSTransportListener" name="vfs">
<parameter locked="false" name="keystore.identity.location">repository/resources/security/wso2carbon.jks</parameter>
<parameter locked="false" name="keystore.identity.type">JKS</parameter>
<parameter locked="false" name="keystore.identity.store.password" svns:secretAlias="vfs.transport.keystore.password">password</parameter>
<parameter locked="false" name="keystore.identity.key.password" svns:secretAlias="vfs.transport.key.password">password</parameter>
<parameter locked="false" name="keystore.identity.alias">wso2carbon</parameter>
</transportReceiver>

Manually encrypt the password using Cipher Tool. See, Encrypting Passwords in Cipher Tool in the administration guide.
Provide the encrypted password value in your proxy configuration by adding the following parameter:
```
<parameter name="transport.vfs.FileURI">smb://{wso2:vault-decrypt('encryptedValue')}</parameter>
```

Failure tracking

To track failures in file processing, which can occur when a resource becomes unavailable, the VFS transport creates and maintains a failed records file. This text file contains a list of files that failed to be processed. When a failure occurs, an entry with the failed file name and the timestamp is logged in the text file. When the next polling iteration occurs, the VFS transport checks each file against the failed records file, and if a file is listed as a failed record, it will skip processing and schedule a move task to move that file.

Transferring large files

If you need to transfer large files using the VFS transport, you can avoid out-of-memory failures by taking the following steps:

In <EI_HOME>/conf/axis2/axis2.xml, in the messageBuilders section, add the binary message builder as follows:

<messageBuilder contentType="application/binary" class="org.apache.axis2.format.BinaryBuilder"/>

and in the messageFormatters section, add the binary message formatter as follows:

<messageFormatter contentType="application/binary" class="org.apache.axis2.format.BinaryFormatter"/>

In the proxy service where you use the VFS transport, add the following parameter to enable streaming (see VFS service-level parameters below for more information):
```
<parameter name="transport.vfs.Streaming">true</parameter>
```
In the same proxy service, before the Send mediator, add the following property:
Note

You also need to add the following property if you want to use the VFS transport to transfer files from VFS to VFS.
```
<property name="ClientApiNonBlocking" value="true" scope="axis2" action="remove"/>
```
For more information, see Example 3 of the Send Mediator.

VFS service-level parameters

The VFS transport does not have any global parameters to be configured. Rather, it has a set of service-level parameters that must be specified for each proxy service that uses the VFS transport.

Parameter Name	Description	Required	Possible Values	Default Value
transport.vfs.FileURI	The URI of the location of your files. This should be the source location of the files (if you are configuring the ESB to read files) or the destination of the files (if you are configuring the ESB to send files). You can specify connection-level parameters on the URL (see VFS URL parameters below). When you need to access the absolute path of the URL, you can define the URL with `sftpPathFromRoot` as shown below. Also, note that transport.vfs.AvoidPermissionCheck is a mandatory parameter for this URL when SFTP is used. <parameter name="transport.vfs.FileURI">sftp://[ username[: password]@] hostname[: port][ absolute-path]?sftpPathFromRoot=true;transport.vfs.AvoidPermissionCheck=true</parameter>	Yes	A valid file URI in the following form: `file://<path>`
transport.vfs.ContentType	Content type of the files processed by the transport. To specify the encoding, follow the content type with a semi-colon and the character set. For example: `<parameter name="transport.vfs.ContentType“>text/plain;charset=UTF-32</parameter>` When writing a file, you can set a different encoding with the `CHARACTER_SET_ENCODING` property: `<property name="CHARACTER_SET_ENCODING" value="UTF-8" scope="axis2" type="STRING"/>`	Yes	A valid content type for the files (e.g., `text/xml`). You can specify the encoding after the content type, such as `text/plain;charset=UTF-32`
transport.vfs. FileNamePattern	If the VFS listener should process only a subset of the files available at the specified file URI location, use this parameter to select those files by name using a regular expression.	No	A regular expression to select files by name (e.g., `*\.xml`)
transport. PollInterval	The polling interval for the transport receiver to poll the file URI location. The value is expressed in seconds unless you add "ms" for milliseconds, e.g., "2" or "2000ms" to specify 2 seconds.	No	A positive integer.
transport.vfs. ActionAfterProcess	Whether to move, delete or take no action on the files after the transport has processed them.	No	`MOVE`, `DELETE` or `NONE`	`DELETE`
transport.vfs. ActionAfterFailure	Whether to move, delete or take no action on the files if a failure occurs.	No	`MOVE`, `DELETE` or `NONE`	`DELETE`
transport.vfs. MoveAfterProcess	Where to move the files after processing if ActionAfterProcess is MOVE.	Yes, if ActionAfterProcess is MOVE	A valid file URI
transport.vfs. MoveAfterFailure	Where to move the files after processing if ActionAfterFailure is MOVE.	Yes, if ActionAfterFailure is MOVE	A valid file URI
transport.vfs. ReplyFileURI	The location where reply files should be written by the transport.	No	A valid file URI
transport.vfs. ReplyFileName	The name for reply files written by the transport.	No	A valid file name	`response.xml`
transport.vfs. MoveTimestampFormat	The pattern/format of the timestamps added to file names as prefixes when moving files.	No	A valid timestamp pattern (e.g., `yyyy-MM-dd'T'HH:mm:ss.SSSZ` )
transport.vfs. Streaming	Whether files should be transferred in streaming mode, which is useful when transferring large files	No	`true` or `false`	`false`
transport.vfs. ReconnectTimeout	Reconnect timeout value in seconds to be used in case of an error when transferring files	No	A positive integer	30 sec
transport.vfs. MaxRetryCount	Maximum number of retry attempts to carry out in case of errors.	No	A positive integer	3
transport.vfs.Append	When writing the response to a file, whether the response should be appended to the response file instead of overwriting the file. This value should be defined as a query parameter in the out/reply file URI. For example: `"vfs:file:///home/user/test/out?` `transport.vfs.Append=true"` or: `<parameter name="` `transport.vfs.ReplyFileURI">` `file:///home/user/test/out?` `transport.vfs.Append=true` `</parameter>`	No	`true` or `false`	`false` (the response file will be completely overwritten).
transport.vfs. MoveAfterFailedMove	Where to move the failed file.	No	A valid file URI
transport.vfs. FailedRecordsFileName	The name of the file that maintains the list of failed files.	No	A valid file name	`vfs-move-failed-records.` `properties`
transport.vfs. FailedRecordsFile Destination	Where to store the failed records file.	No	A folder URI
transport.vfs. MoveFailedRecord TimestampFormat	Entries in the failed records file include the name of the file that failed and the timestamp of its failure. This property configures the time stamp format.	No	A valid timestamp pattern (e.g., `yyyy-MM-dd'T'HH:mm:ss.SSSZ` )	`dd-MM-yyyy HH:mm:ss`
transport.vfs. FailedRecordNext RetryDuration	The time in milliseconds to wait before retrying the move task.	No	A positive integer	3000 milliseconds
transport.vfs.Locking	By default, file locking is enabled in the VFS transport. This parameter lets you configure the locking behavior on a per service basis. You can also disable locking globally by specifying the parameter at the receiver level and selectively enable locking only for a set of services.	No	`enable` or `disable`	`enable`
transport.vfs. FileProcessCount	This setting allows you to throttle the VFS listener by processing files in batches. Specify the number of files you want to process in each batch.	No	A positive integer, such as 10	N/A
transport.vfs. FileProcessInterval	The interval in milliseconds between two file processes.	No	A positive integer, such as 1000	N/A
transport.vfs.ClusterAware	Whether VFS coordination support is enabled in a clustered deployment or not.	No	`true` or `false`	`false`
transport.vfs.FileSizeLimit	Only file sizes that are less than or equal to the defined limit are processed.	No	File size in bytes	-1(unlimited file size)
transport.vfs.AutoLockReleaseInterval	The timeout value for stale locks where the VFS transport will ignore those file locks once the defined time period is reached. (The time period is calculated from the time the lock is created to the time you attempt to access it.) If you need stale locks to never timeout provide -1 as the timeout value.	No	Time in milliseconds	20000
transport.vfs.SFTPIdentities	Location of the private key	No	A valid file path	N/A
transport.vfs.SFTPIdentityPassPhrase	Passphrase of the private key	No	A valid passphrase	N/A
transport.vfs.SFTPUserDirIsRoot	If the SFTP user directory should be treated as root	No	`true` or `false`	true
transport.vfs.ResolveHostsDynamically	Whether hostnames should be resolved at the time of deployment or whether it is necessary to resolve hostnames dynamically at runtime. By default hostnames are resolved at the time of deployment. If you want to resolve hostnames at runtime, set this parameter to `true`. Note Reolving hostnames at runtime is only possible for the Server Message Block (SMB) protocol.	No	`true` or `false`	`false`

VFS URL parameters

When you use the transport.vfs.FileURI parameter, you can set connection-specific VFS parameters as URL query parameters. For example, to use SFTP with SSL, you could specify the URL as shown below. Note that transport.vfs.AvoidPermissionCheck is a mandatory parameter for this URL when SFTP is used.

<parameter name="transport.vfs.FileURI">vfs:ftps://test:test123@10.200.2.63/vfs/in?vfs.ssl.keystore=/home/user/openssl/keystore.jks&amp;vfs.ssl.truststore=/home/user/openssl/vfs-truststore.jks&amp;vfs.ssl.kspassword=importkey&amp;vfs.ssl.tspassword=wso2vfs&amp;vfs.ssl.keypassword=importkey;transport.vfs.AvoidPermissionCheck=true</parameter>

Following are details on the URL parameters you can set. To configure the proxy over ftp/sftp click here.

Parameter Name	Description	Possible Values	Default Value
transport.vfs.AvoidPermissionCheck	Be sure to set this parameter to true for an SFTP connection. This is because (by default) the VFS transport checks whether the user has permission to access the location of the files (the source location or the destination). However, since the system is reading files in an external server through the SFTP connection, this permission check is not required and should be avoided.	true \| false
vfs.passive	Enable FTP passive mode. This is required when the FTP client and server are not in the same network.	true \| false	false
transport. vfs . Append	If file with same name exists, this parameter tells whether to create a new file and write content or append content to existing file	true \| false	false
vfs.protection	Set data channel protection level using FTP PROT command	C - Clear S - Safe(SSL protocol only) E - Confidential(SSL protocol only) P - Private	C
vfs.ssl.keystore	Private key store to use for mutual SSL. Your keystore must be signed by a certificate authority. For more information, see http://docs.oracle.com/cd/E19509-01/820-3503/ggfen/index.html.	String - Path of keystore
vfs.ssl.kspassword	Private key store password	String
vfs.ssl.keypassword	Private key password	String
vfs.ssl.truststore	Trust store to use for SFTP	String - Path of keystore
vfs.ssl.tspassword	Trust store password	String
transport.vfs.CreateFolder	If the directory does not exists create and write the file	true \| false	false
transport. vfs.SendFileSynchronously	Whether to send files synchronously to the file host. When this parameter is set to `true`, files will be sent one after another to the file host. This synchronous write can be configured on a per host basis.	true \| false	false

WSO2 Enterprise Integrator 6.5.0