Exposing a Web Resource as a Data Service

This tutorial will guide you on how to expose a web resource as a data service using the ESB profile of WSO2 Enterprise Integrator (WSO2 EI). You can create a data service that can fetch selected data from a web resource and display the data in XML, JSON or RDF form. 

Follow the instructions below. Also, see the samples in Data Integration Samples.

Start the Create New Data Service wizard

Now, let's start creating the data service from scratch:

1. Log in to the management console using the following URL on your browser:  "https://localhost:9443/carbon/".

2. Click Create under the Data Service menu to open the Create Data Service window.

3. Enter WebResource as the data service name as shown below.
   

4. Click Next to go to the Datasources screen.
   

Add a Web Resource as the datasource

You can add a web resource as the datasource by following the steps given below.

1. Click Add New Datasource to open the following screen:
   
2.  Enter the following values:
   1. In the Datasource ID field, enter WebHarvestDataSource as the datasource Id.
   2. In the Datasource Type field, specifies the type of datasource for which you will create the data service. Select Web Datasource from the list. You will now get the following screen:
      

   3. Select Inline Web Harvest Config and enter the configuration as shown below:
      

      <config>
       <var-def name='AppInfo'>
        <xslt>
         <xml>
          <html-to-xml>
           <http method='get' url='https://play.google.com/store/apps'/>
          </html-to-xml>
         </xml>
         <stylesheet>
         <![CDATA[ <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
      		<xsl:output method="xml" omit-xml-declaration="yes" indent="yes"/>
      		<xsl:template match="/">
                       <AppInfo>
                          <xsl:for-each select="//*[@class='details']">
                          <App>
                           <Title><xsl:value-of select="a[@class='title']"/></Title>
                           <Description><xsl:value-of select="div[@class='description']"/></Description>
                          </App>
                        </xsl:for-each>
                       </AppInfo>
                      </xsl:template>
      	     </xsl:stylesheet>
         ]]>
         </stylesheet>
        </xslt>
       </var-def>
      </config>

3. Save the datasource. 
4. Click Next to go to the Queries screen.
   

Define a query for the datasource

Now let's start writing a query for getting data from the web resource. The query will specify the data that should be fetched by this query, and the format that should be used to display the data.

1. Click Add New Query to open the Add New Query screen.
2. Enter the following values:
   • Query ID: In this field, enter webquery as the query ID.
   • Datasource: In this field, select the datasource for which you are going to write a query. Select the web datasource that you created previously.
     
   • Scrapper Variable: When you add a query to a Web datasource, you must enter a Scraper Variable. This scraper variable must be the same as the output name in the web datasource configuration, which returns the output from the configuration. In this example, the var-def name in the configuration is AppInfo (< var-def name = 'AppInfo' >).
3. Define Output Mapping: Now, let's specify how the data fetched from the datasource should be displayed in the output. We will create output mappings for the following data in the web resource: Title and Description. 
   1. In the Output type field, specify the format in which the query results should be presented. You can select XML, JSON or RDF. We will use XML for this tutorial.
   2. In the Grouped by element field, specify a grouping for all the output mappings. This will be the XML element that will group the query result. Enter AppInfo in this field. 
   3. In the Row Name field, specify the XML element that should group each individual result.  Enter App in this field.
   4. Click Add New Output Mapping to start creating the output mapping for the Title field. Enter values as shown below:
      
   5. Click Add to save the output mapping. 

   6. Now, add another output mapping for the Description column.

   7. You will now have the following output mappings listed for the webquery query:
      

   8. Once you have created all the output mappings, click Main Configuration to return to the Edit Query screen.

      Find out more about defining Output Mappings.

4. Click Save, to save the query.
5. Click Next to go to the Operations screen.
   

Defining and operation to invoke the query

You need to define an operation, to expose data as a SOAP service.

1. Click Add New Operation to open the Add New Operation screen. 
2. In the Operation Name field, enter GetResource.
3. In the Query ID field, select WebQuery.
4. Save the operation.

Finish creating the data service

In this tutorial, we are only defining a SOAP service to expose the data. Therefore, you can click Finish to complete the data service creation process, after defining the operation. You will now be taken to the  Deployed Services screen, which shows all the data services deployed on the server.

Invoking your data service

You can try the data service you created by using the TryIt tool that is in your product by default.

1. Go to the Deployed Services screen.
2. Click the Try this service link for the WebResource data service. The TryIt Tool will open with the data service.
3. Select the GetResource operation and click Send.
4. The response will be published.