Complex JSON Processing

In this tutorial, let's look at how to process a complex JSON queryvia Siddhi Query Language and use extracted values from the JSON query in the streaming integration logic of the Siddhi Query Language.

At the end of this tutorial, you will know:

• How to extract the string value of a JSON corresponding to a given path using json:getString.
• How to extract the object of a JSON corresponding to a given path using json:getObject.
• How to tokenize the given JSON based on a provided path and get the response as an object using the json:tokenizeAsObject.
• How to use a sink of the file type to write event data that is processed within Siddhi to files.

Let’s consider a scenario where Sam, the factory foreman receives an inventory list from the Sweet Factory for required stocks. This list is in JSON format.  This JSON contains information on required materials, required quantities and corresponding suppliers for those materials. 

{
 "InventoryList": [
   {
     "materialName": "Flour",
     "RequiredStocks": {
       "Amount": "5500",
       "Unit": "kg",
       "Suppliers": [
         "ABC Distributors",
         "Foodies (pvt) Ltd",
         "Sherry Flour Mills"
       ]
     }
   },
   {
     "materialName": "Sugar",
     "RequiredStocks": {
       "Amount": "1000",
       "Unit": "kg",
       "Suppliers": [
         "Asia Suppliers",
         "Sweet Stores (pvt) Ltd",
         "Don Bakery Needs",
         "Jaden Suppliers"
       ]
     }
   },
   {
     "materialName": "Honey",
     "RequiredStocks": {
       "Amount": "4500",
       "Unit": "litre",
       "Suppliers": [
         "Busy Bee Distributors",
         "Sweet Stores (pvt) Ltd",
         "Honey Wholesale (pvt) Ltd"
       ]
     }
   }
 ]
}

Tutorial steps

To process the inventory list sent by supplier in JSON format, let's create a Siddhi application as follows:

1. Access the Stream Processor Studio and start creating a new Siddhi application.

2. Let's enter a name and a description for your new Siddhi application as follows.

   @App:name('ComplexJsonTutorial')

   @App:description('Demonstrate the features of Siddhi Execution JSON extension')

3. First, let's define the input stream you need in order to capture the inventory lists.

   define stream InventoryStream (json string);

4. Now let's see how to split the input JSON into separate events for each material type. For example the sample JSON array provided above needs to be divided into three  events for the material types Flour, Honey, and Sugar as shown below.

   Event 1	Event 2	Event 3
   {     "materialName": "Flour",     "RequiredStocks": {       "Amount": "5500",       "Unit": "kg",       "Suppliers": [         "ABC Distributors",         "Foodies (pvt) Ltd",         "Sherry Flour Mills"       ]     }   }	{     "materialName": "Sugar",     "RequiredStocks": {       "Amount": "1000",       "Unit": "kg",       "Suppliers": [         "Asia Suppliers",         "Sweet Stores (pvt) Ltd",         "Don Bakery Needs",         "Jaden Suppliers"       ]     }   }	{     "materialName": "Honey",     "RequiredStocks": {       "Amount": "4500",       "Unit": "litre",       "Suppliers": [         "Busy Bee Distributors",         "Sweet Stores (pvt) Ltd",         "Honey Wholesale (pvt) Ltd"       ]     }   }

   To do this, you need to use the json:tokenizeAsObject function available in the siddhi-execution-json extension. This function generates separate events for elements in the InventoryList array.

   To use this extension, include it ion the from clause as shown below:

   from InventoryStream#json:tokenizeAsObject ( json, '$.InventoryList')

   The above single line is indicated as an error in the Stream Processor Studio. This is because you have not completed the query at this stage. This error no longer appears once you perform the next stepo and complete the query.

   The first parameter of the function indicates the input JSON that you need to tokenize. The second parameter indicates the path that is used to tokenize.

5. After splitting the input json into separate events, let’s see how you can extract the values from each event.
   
   You can extract the materialName as a string as follows.
   
   

   json:getString(jsonElement, '$.materialName')

   Here, the first parameter jsonElement is the return attribute name of the tokenizeAsObject function. $.materialName is the path of the JSON from where the material name is extracted.If you want to extract elements from your JSON as boolean, double, float, int, or long values, the siddhi-execution-json extension provides support to do so via  json:getBool, json:getDouble, json:getFloat, json:getInt, and json:getLong respectively.

   In your sample JSON, the supplier list is arrives as an array. Assume that the factory foreman needs to extract it as an object for further processing. You can do it as shown below.

   json:getObject(jsonElement,'$.RequiredStocks.Suppliers')

   Then you can extract all the required attributes from the input JSON and push that into the EachMaterialStream as follows.
   
   

   from InventoryStream#json:tokenizeAsObject ( json, '$.InventoryList')

   select

      json:getString(jsonElement, '$.materialName') as materialName,

      str:concat(

   json:getString(jsonElement, '$.RequiredStocks.Amount') ,    json:getString(jsonElement, '$.RequiredStocks.Unit')

   ) as quotingAmount,

    json:getObject(jsonElement,'$.RequiredStocks.Suppliers') as supplierList

   insert into EachMaterialStream;

   
   In the above query, you have extracted required stock amount and its unit, and later concatenated those by specifying str:concat as the siddhi execution string.

6. Now you need to further split the events in the EachMaterialStream so that a single event only contains the information relating to one supplier. Therefore, let's tokenize the EachMaterialStream using the supplierList object.

   from EachMaterialStream#json:tokenize(supplierList, "$")

   select materialName, quotingAmount,

           json:getString(jsonElement, '$') as supplier

   insert into EachSupplierStream;

   You are using the json:tokenize function here instead of json:tokenizeAsObject because the supplier list does not have any further nesting and therefore, receiving the supplier name as a string is sufficient.

7. Now, let's create a file with the quotation request for each supplier. For this, you can use a sink of the file type and extract information from the EachSupplierStream stream.

   @sink(type = 'file', append = "false",

   file.uri = "/home/tanya/Desktop/quotationLetters/{{materialName}}_{{supplier}}.html",

   @map(type = 'text',

   @payload("Hi {{supplier}},<br/> Please provide a quotation for {{quotingAmount}} of {{materialName}}. <br/>Thank you")))

   define stream EachSupplierStream (materialName string, quotingAmount string, supplier string);

   The sink configuration needs to be added with the stream definition of the EachSupplierStream stream as shown above to connect the stream to the sink. This enables the sink to get the events to be published from the EachSupplierStream stream.