File Inbound Protocol

The WSO2 ESB file inbound protocol is a multi-tenant capable alternative to WSO2 ESB VFS transport. The ESB file inbound protocol uses the VFS transport to process files in a specified source directory. After processing the files, it moves them to a specified location or deletes them. Note that files cannot remain in the source directory after processing or they will be processed again, so if you need to maintain these files or keep track of the files that are processed, specify the option to move them instead of deleting them after processing.

Security

The file inbound protocol supports the FTPS protocol with Secure Sockets Layer (SSL). The configuration is identical to other protocols. The only difference is the URL prefixes and parameters. For more information, see VFS URL parameters.

Failure tracking

To track failures in file processing that can occur when a resource becomes unavailable, the VFS transport creates and maintains a file for failed records. This text file contains a list of files that failed to process. When a failure occurs, an entry with the failed file name and timestamp is logged in the text file. When the next polling iteration occurs, the VFS transport checks each file against the failed records file and if a file is listed as a failed record, it will skip processing and schedule a move task to move that file.

<inboundEndpoint xmlns="http://ws.apache.org/ns/synapse" 
				 name="file" sequence="request" 
				 onError="fault" 
				 protocol="file" 
				 suspend="false">
   <parameters>
      <parameter name="interval">1000</parameter>
      <parameter name="sequential">true</parameter> 
      <parameter name="coordination">true</parameter> 
      <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
      <parameter name="transport.vfs.MoveAfterProcess">file:///home/user/test/out</parameter>
      <parameter name="transport.vfs.FileURI">file:///home/user/test/in</parameter>
      <parameter name="transport.vfs.MoveAfterFailure">file:///home/user/test/failed</parameter>
      <parameter name="transport.vfs.FileNamePattern">.*.txt</parameter>
      <parameter name="transport.vfs.ContentType">text/plain</parameter>
      <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   </parameters>
</inboundEndpoint>

VFS service-level parameters

The VFS transport does not have any global parameters to be configured. Rather, it has a set of service-level parameters that must be specified for each proxy service that uses the VFS transport. For information on how to configure the file inbound protocol for FTP, SFTP and FILE connections, see Configuring File Inbound Protocol for FTP, SFTP and FILE Connections.

Parameter	Description	Required	Possible Values	Default Value
`interval`	The time duration in milliseconds between two file scans that checks for updates.	Yes
`sequential`	Files will be processed sequentially when this parameter is set to true.	Yes		true
`coordination`	This should be true for clustered deployments in order to prevent two nodes from retrieving the same file.	Yes		true
`transport.vfs.` `FileURI`	The URI where the files you want to process are located. You can specify connection-level parameters on the URL.	Yes	A valid file URI in the following form: `file://<path>`
`transport.vfs.` `ContentType`	Content type of the files processed by the transport. To specify the encoding when reading a file, follow the content type with a semi-colon and the character set. For example: `<parameter name="transport.vfs.ContentType“>text/plain;charset=UTF-32</parameter>` When writing a file, you can set a different encoding with the `CHARACTER_SET_ENCODING` property: `<property name="CHARACTER_SET_ENCODING" value="UTF-8" scope="axis2" type="STRING"/>`	Yes	A valid content type for the files (e.g., text/xml). You can specify the encoding after ContentType, such as: text/plain;charset=UTF-32
`transport.vfs.` `FileNamePattern`	If the VFS listener should process only a subset of the files available at the specified file URI location, use this parameter to select those files by name using a regular expression.	No	A regular expression to select files by name (e.g., `*\.xml`)
t`ransport.vfs.` `ActionAfterProcess`	Whether to move or delete the files after the transport has processed them.	No	`MOVE` or `DELETE`	`DELETE`
`transport.vfs.` `ActionAfterFailure`	Whether to move or delete the files if a failure occurs.	No	`MOVE` or `DELETE`	`DELETE`
`transport.vfs.` `MoveAfterProcess`	If WSO2 ESB reads the files without any issue, it moves the files to the location specified in this parameter.	Yes, if ActionAfterProcess is MOVE	A valid FileURI
`transport.vfs.` `MoveAfterFailure`	If WSO2 ESB is unable to read a file from the location specified in `FileURI` due to an error while reading the file, it moves this file to the location specified in this parameter .	Yes, if ActionAfterFailure is MOVE	A valid FileURI
`transport.vfs.` `ReplyFileURI`	The location where reply files should be written by the transport.	No	A valid FileURI
`transport.vfs.` `ReplyFileName`	The name for reply files written by the transport.	No	A valid FileName	`response.xml`
`transport.vfs.` `MoveTimestampFormat`	The pattern/format of the timestamp added to file names as prefixes when moving files.	No	A valid timestamp pattern (e.g., `yyyy-MM-dd'T'HH:mm:ss.SSSZ` )
`transport.vfs.` `Streaming`	Whether files should be transferred in streaming mode, which is useful when transferring large files.	No	`true` or `false`	`false`
`transport.vfs.` `ReconnectTimeout`	Reconnect timeout value in seconds to be used in case of an error when transferring files.	No	A positive integer	30 sec
`transport.vfs.` `MaxRetryCount`	Maximum number of retry attempts in case of errors.	No	A positive integer	3
`transport.vfs.` `MoveAfterFailedMove`	The location to move a failed file.	No	A valid FileURI
`transport.vfs.` `FailedRecordsFileName`	The name of the file that maintains the list of failed files.	No	A valid FileName	`vfs-move-failed-records.` `properties`
`transport.vfs.` `FailedRecordsFile` `Destination`	The location to store the failed records file.	No	A folder URI	`repository/conf/`
`transport.vfs.` `MoveFailedRecord` `TimestampFormat`	The time stamp format for entries in the failed records file. The failed records file maintains the name of the file that failed and the timestamp of its failure.	No	A valid timestamp pattern (e.g., `yyyy-MM-dd'T'HH:mm:ss.SSSZ` )	`dd-MM-yyyy HH:mm:ss`
`transport.vfs.` `FailedRecordNext` `RetryDuration`	The time in milliseconds to wait before retrying the move task.	No	A positive integer	3000 milliseconds
`transport.vfs.Locking`	By default, file locking is enabled in the VFS transport. This parameter lets you configure the locking behavior on a per service basis. You can also disable locking globally by specifying the parameter at the receiver level and selectively enabling locking only for a set of services.	No	`enable` or `disable`	`enable`
`transport.vfs.` `FileProcessCount`	This parameter allows you to throttle the VFS listener by processing files in batches. Specify the number of files you want to process in each batch.	No	A positive integer, such as 10	N/A
`transport.vfs.` FileProcessInterval	The interval in milliseconds between file processing batches.	No	A positive integer, such as 1000	N/A
`transport.vfs.DistributedLock`	This applies only in cluster deployments. Set to `true` if you need to avoid multiple servers trying to process the same file simultaneously.	No	True or False	N/A
`transport.vfs.DistributedTimeout`	The timeout period in seconds for the distributed lock.	No	A positive integer, such as 10	N/A
`transport.vfs.AutoLockRelease`	Set to true if you need to release locking in order to avoid files not being processed due to faulty locking. This works together with the `transport.vfs.AutoLockReleaseInterval` and `transport.vfs.LockReleaseSameNode` parameters.	No	True or False	N/A
`transport.vfs.AutoLockReleaseInterval`	Lock release interval in milliseconds.	No	A positive integer, such as 1000	N/A
`transport.vfs.LockReleaseSameNode`	Set to `true` if you need to release the locks only accrued by the same worker node. If this is set to `false`, locks accrued by other nodes will be released according to the value specified in `transport.vfs.AutoLockReleaseInterval`.	No	True or False	true
`transport.vfs.FileSortAttribute`	The attribute by which the files should be sorted and processed.	No	NONE, Name, Size and Lastmodifiedtimestamp	N/A
`transport.vfs.FileSortAsscending`	The sort order to sort and process the files. If set to `true` files will be sorted in ascending order based on the attribute you specify in `transport.vfs.FileSortAttribute.`	No	True or False	true
`transport.vfs.CreateFolder`	Set to `True` to create a folder if a folder does not exist when moving files.	No	True or False	false
`transport.vfs.SubFolderTimestampFormat`	The pattern/format of the timestamps added to the folder structure when moving files. You need to set `transport.vfs.CreateFolder` to `true` to in order to specify a value for this parameter.	No	A valid timestamp pattern (e.g., `yyyy-MM-dd'T'HH:mm:ss.SSSZ` )	N/A
`transport.vfs.Build`	Set to `True` if you need to build the content inside the file before injecting the file to the mediation engine. If there is a build error, the file will not be injected to the mediation engine.	No	True or False	false

Note

If you specify the transport.vfs.FileProcessCount parameter you do not need to specify the transport.vfs.FileProcessInterval parameter in a configuration, and vice versa. This is because the transport.vfs.FileProcessCount parameter and the transport.vfs.FileProcessInterval parameter cannot be used at the same time.

Samples

For a sample that demonstrates how to use the file system as an input medium via the inbound file listener, see Sample 900: Inbound Endpoint File Protocol Sample (VFS).