Sample 271: File Processing

All files for this example are in sample.zip.

Objective: To pick up a file in a local directory, insert the records from the file into a database, send an email with data from the file, trace and write the log and finally move the file to another directory. The following scheme visually displays the processes running in this example.

Note

This example works with the MySQL database.

Step 1. Set up the database

1. Manually set up the database.

2. Create a table named "info" in your schema. You can use the commands as shown bellow.

delimiter $$

CREATE TABLE `info` (
  `name` varchar(45) DEFAULT '',
  `surname` varchar(45) DEFAULT NULL,
  `phone` varchar(45) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8$$

3. Make sure the "info" table has been created and it contains the following columns:

name
surname
phone

The screen-shot bellow displays the result of the query to show records from the test.info table. There is no data in the table.

Step 2. Create main and fault sequences

1. Find the main.xml and fault.xml files in the supplied archive. You can find them in the \conf\synapse-config\sequences folder.

2. Save the files to <ESB_HOME>/repository/deployment/server/synapse-configs/default/sequences folder.

Note

The "main" and "fault" sequences are created and preconfigured automatically while installing ESB.

Step 3. Configure ESB

You will now configure the ESB to use the VFS transport for processing the files and the MailTo transport for sending the email message. The scenario requires some modifications of the axis2.xml (ESB configuration file) located in <ESB_HOME>/repository/conf/axis2.

Enabling VFS transport

1. To enable the VFS transport receiver, uncomment the following string.

<transportreceiver name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportListener"/>

2. To enable the VFS transport sender, uncomment the following string.

<transportSender name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportSender"/>

Configuring the MailTo Transport

This transport can be used to send e-mail messages.

1. Configure the mail transport sender to use a mailbox for sending the messages. Use the following code.

<transportSender name="mailto">
   <parameter name="mail.smtp.host">smtp.gmail.com</parameter>
   <parameter name="mail.smtp.port">587</parameter>
   <parameter name="mail.smtp.starttls.enable">true</parameter>
   <parameter name="mail.smtp.auth">true</parameter>
   <parameter name="mail.smtp.user">username</parameter>
   <parameter name="mail.smtp.password">userpassword</parameter>
   <parameter name="mail.smtp.from">username@gmail.com</parameter>
</transportSender>

Tip

In this sample, we do not retrieve mails from a mailbox. So there is no need to enable the mail transport receiver.

Configuring Message Builders/Formatters

When a message comes thorough the wire, it first goes through a message builder and message builder is responsible for converting the message into a SOAP message. Message formatters determine the outgoing wire message format of a SOAP message inside the ESB.

1. Add the following message formatter to the axis2.xml file placed in <WSO2ESB_HOME>/repository/conf/axis2.

Message formatters are configured in axis2.xml under the section.

<messageFormatters></messageFormatters>

Add the org.apache.axis2.transport.http.ApplicationXMLFormatter under the content type "text/html".

<messageFormatter contentType="text/html" class="org.apache.axis2.transport.http.ApplicationXMLFormatter"/>

Step 4. Add Database Drivers

1. Find the MySQL database driver mysql-connector-java-5.1.10-bin.jar in the supplied archive. It is located in the lib folder.

2. Copy the driver to the <WSO2ESB_HOME>/repository/components/lib directory.

Step 5. Add Smooks Libraries

This example uses a CSV smooks library.

1. Find the CSV smooks library milyn-smooks-csv-1.2.4.jar in the supplied archive. It is located in the lib folder.

2. Copy the library to the <WSO2ESB_HOME>/repository/components/lib directory.

Notice

These configuration changes make system-wide changes to the ESB and the ESB has to be restarted for these changes to take effect.

Step 6. Create and Configure FileProxy

1. Find the FileProxy.xml in the supplied archive. It is located in the \conf\synapse-config\sequences folder.

The XML code of the sequence is provided bellow.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse" name="FileProxy" transports="vfs" startOnLoad="true" trace="disable">
    <target>
        <inSequence>
            <log level="full"/>
            <clone>
                <target sequence="fileWriteSequence"/>
                <target sequence="sendMailSequence"/>
                <target sequence="databaseSequence"/>
            </clone>
        </inSequence>
    </target>
    <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
    <parameter name="transport.PollInterval">15</parameter>
    <parameter name="transport.vfs.MoveAfterProcess">file:///home/username/test/original</parameter>
    <parameter name="transport.vfs.FileURI">file:///home/username/test/in</parameter>
    <parameter name="transport.vfs.MoveAfterFailure">file:///home/username/test/failure</parameter>
    <parameter name="transport.vfs.FileNamePattern">.*.txt</parameter>
    <parameter name="transport.vfs.ContentType">text/plain</parameter>
    <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
</proxy>

2. In the file, define the folder where the original file is moved after processing.

<parameter name="transport.vfs.MoveAfterProcess">[file:///home/]<username>/test/original</parameter>

3. Define the folder where the input file should be placed.

<parameter name="transport.vfs.FileURI">[file:///home/]<username>/test/in</parameter>

4. Define the folder where the file is moved if an error occurs:

<parameter name="transport.vfs.MoveAfterFailure">[file:///home/]<username>/test/failure</parameter>

5. Save the file to:

<WSO2ESB_HOME>/repository/deployment/server/synapse-configs/default/proxy-services/FileProxy.xml.

Step 7. Create and Configure databaseSequence

Follow the instructions below to create a sequence used to connect to the database and to insert the data there.

1. Find the databaseSequence.xml file in the supplied archive. It is located in the \conf\synapse-config\sequences folder.

The XML code of the sequence is provided bellow.

<dbreport>
	<connection>
		<pool>
			<password>databasepassword</password>
			<user>databaseuser</user>
			<url>jdbc:mysql://localhost:3306/info</url>
			<driver>com.mysql.jdbc.Driver</driver>
		</pool>
	</connection>
	<statement>
		<sql>insert into INFO values (?, ?, ?)</sql>
		<parameter expression="//csv-record/name/text()" type="VARCHAR"/>
		<parameter expression="//csv-record/surname/text()" type="VARCHAR"/>
		<parameter expression="//csv-record/phone/text()" type="VARCHAR"/>
	</statement>
</dbreport>

2. Specify your database password, username and URL in the <pool> section of the sequence.

3. Save the file to:

<WSO2ESB_HOME>/repository/deployment/server/synapse-configs/default/sequences/databaseSequence.xml.

Step 8. Create and Configure fileWriteSequence

1. Find the fileWriteSequence.xml in the supplied archive. It is located in the \conf\synapse-config\sequences folder.

The XML code of the sequence is provided bellow.

<?xml version="1.0" encoding="UTF-8"?>
<sequence xmlns="http://ws.apache.org/ns/synapse" name="fileWriteSequence">
    <log level="custom">
        <property name="sequence" value="fileWriteSequence"/>
    </log>
    <property xmlns:ns2="http://org.apache.synapse/xsd" name="transport.vfs.ReplyFileName" expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.txt')" scope="transport"/>
    <property name="OUT_ONLY" value="true"/>
    <send>
        <endpoint name="FileEpr">
            <address uri="vfs:file:///home/username/test/out"/>
        </endpoint>
    </send>
</sequence>

2. In the XML code, define the folder to move the file to.

3. Save the file to:

<WSO2ESB_HOME>/repository/deployment/server/synapse-configs/default/sequences/fileWriteSequence.xml

Step 9. Create and Configure sendMailSequence

1. Find the sendMailSequence.xml file in the supplied archive. You can find it in the \conf\synapse-config\sequences folder.

The XML code of the sequence is provided bellow.

<?xml version="1.0" encoding="UTF-8"?>
<sequence xmlns="http://ws.apache.org/ns/synapse" name="sendMailSequence">
    <log level="custom">
        <property name="sequence" value="sendMailSequence"/>
    </log>
    <property name="messageType" value="text/html" scope="axis2"/>
    <property name="ContentType" value="text/html" scope="axis2"/>
    <property name="Subject" value="File Received" scope="transport"/>
    <property name="OUT_ONLY" value="true"/>
    <send>
        <endpoint name="FileEpr">
            <address uri="mailto:username@gmail.com"/>
        </endpoint>
    </send>
</sequence>

2. In the XML code, define the e-mail address to which the notification will be sent.

3. Save the file to:

<WSO2ESB_HOME>/repository/deployment/server/synapse-configs/default/sequences/sendMailSequence.xml.

Step 10. Create an Input file

1. Create a text file in the following format.

name_1, surname_1, phone_1
name_2, surname_2, phone_2

2. Save the file in TXT format to the /home/<username>/test/in directory.

Step 11. Monitor Results

In this example, ESB listens on a local file system directory. When a file is dropped into in directory, ESB picks this file.

1. Make sure the file has appeared in the out directory.

2. ESB inserts the records from the text file to the database. Make sure the data is in the info table. The screen-shot bellow displays the content of the test.info table containing data from the file.

3. Make sure the original file has been moved to the /home/<username>/test/original directory.

4. Make sure the e-mail notification has been sent to the specified email. The message should contain the file data. The following screen-shot displays a notification received.