Web Resource

The WSO2 Data Services Server supports scraping and extracting web information and presenting it as a service.

First, create a configuration file with information regarding the web resource and the xslt file which has information regarding the response. Given below are sample configuration and xslt files.

Sample Configuration File:

<?xml version="1.0" encoding="UTF-8"?>
<config>
 <var-def name='weatherInfo'>
  <xslt>
   <xml>
    <html-to-xml>
     <http method='get' url='http://weather.yahoo.com/'/>
    </html-to-xml>
   </xml>
   <stylesheet>
    <file path="/media/data/web/template.xsl"/>
   </stylesheet>
  </xslt>
 </var-def>
</config>

Sample XSLT File:

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" omit-xml-declaration="yes" indent="yes"/>
<xsl:template match="/">
  <WeatherInfo>
   <xsl:for-each select="//div[@id='myLocContainer']/ul/li">
     <Weather>
      <Region><xsl:value-of select="a/@title"/></Region>
      <Temp><xsl:value-of select="em"/></Temp>
     </Weather>
   </xsl:for-each>
  </WeatherInfo>
</xsl:template>
</xsl:stylesheet>

Follow the instructions below to create a data service. In this example we use Yahoo Weather Forecasts to extract weather information.

1. Log on to the Data Services Server Management Console.

2. Click the Main tab and select the Create link under Data Services.

3. The Create Data Service page appears. Fill in the fields and click Next. The Data Service name is mandatory.

4. The Data Sources page appears. Click on the Add New Data Source link and add your Data Source details. Select the data source type as Web Data Source.

Example 1:

You can either save the configuration to an xml and provide the file path, or can enter as an inline configuration. In example 1, we provide the sample configuration mentioned above as a file path.

Example 2:

In example 2, we use an inline configuration.

Click Save. You have added a Data Source to your Web Service. Click Next to proceed.

6. The Queries page appears.

Initially your service does not include any queries. Click on 'Add New Query' to add a new query, input/output mappings, events etc. to your Data Service. Enter query details according to the structure of the response you want.

Scraper variable should be the same as the output name in the configuration, which actually returns the output from the configuration. Eg: It should be 'weatherInfo' which is the var-def name in the configuration.

In this example, we have used Grouped By Element, Row Name and Row namespace as follows.

Click the Add New Output Mapping button as shown in the screenshot above to create how the output looks like. Once you have entered the output mapping details, click Main Configuration and then the Save button as follows:

For information on adding validations to input mappings in the query, if any, refer to section Input Validators.

Once the query details are added click Save.You have added one query to your data service configuration. To continue, click Next.

7. The Operations page appears. Click the Add new operation link to create an operation. Select a query to be executed when this operation is called.

Click Save once done. For example,

8. The Data Service configuration is done. Click Finish. (You can continue to add resources by clicking the Next button if you like)

9. You will be navigated to the "Deployed services" page.

From here, you can manage your service.