Home

Paged Web Reader

Icon
Use When	Use this component to break up requests for data into multiple smaller requests(pages). This is helpful when querying large data sets. Each smaller paged response is sent as a text message to downstream components increasing the speed of your flow while reducing memory requirements.
Description	The Paged Web Reader component extends the functionality of the built-in Web component. The components are configured the same with the exception of two additional parameters on the Paged Web Reader, "Request XPath" and "Result XPath". There are different ways to implement pagination via web services. The paged web reader is designed to work with services that accept a page size and the ID of the last record received. An initial request is sent using the provided body or an incoming text message. The received response is then parsed for the last record ID using the Result XPath expression. If a last record ID is found, a subsequent web service request is made by updating the last record ID using the Request XPath expression. This process continues to repeat until the last record ID is not found. Each response is sent as a new message to downstream components for processing. As a result, downstream components can begin processing the response before the paged web reader has completed.
Inbound Message Type	Text Based Message
Output Message Type	Text Based Message

Parameters

Resource	A web (http url) resource where the service resides and can be called
Enabled
Run When
Append to URL	Text that will be appended to to the Connection url forming the base url for the web service call
Body From	Either 'Message' or 'Provided'. If Message the body of the web service request message will come from the inbound message. If 'Provided" the body of the web service request message will come from the Body Text parameter
Body Text	If the Body From parameter is 'Provided', this parameter is used to specify the body of the text message for the web service request
Parameter replacement
Request XPath	An XPath expression used to identify the template element to be populated with the last received record ID.
Response XPath	An XPath expression used to locate the last received record ID.
Log Input
Log Output
Inbound Queue Capacity
Shared

Example 1.

Request

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:sec="http://schemas.xmlsoap.org/ws/2002/04/secext">
   <soapenv:Body>
      <urn:select_users>
         <pagination>
            <previousPageLastRecordPredicate></previousPageLastRecordPredicate>
            <pageSize>100</pageSize>
         </pagination>
      </urn:select_users>
   </soapenv:Body>
</soapenv:Envelope>

Response

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
   <soapenv:Body>
      <ns1:select_usersResponse>
         <data>
            <directory>
               <users>
                  <login>jdoe</login>
                  <firstName>john</firstName>
                  <lastName>doe</lastName>
               </users>
               ...
            </directory>
         </data>
         <lastRecordPredicate>./login='jdoe'</lastRecordPredicate>
      </ns1:select_usersResponse>
   </soapenv:Body>
</soapenv:Envelope>

Settings

Request XPath: //previousPageLastRecordPredicate
Result XPath:  //lastRecordPredicate

Component Operation

1. The Paged Web Component receives a control message and sends the initial request.

   1. In this example, the initial request is provided as a parameter on the Paged Web Reader component.

   2. The request element <pageSize> specifies the response should include 100 records.

   3. The request element <previousPageLastRecordPredicate> is blank so the results will return the first page.

2. The XML response from the remote server is returned after receiving our request.

   1. The response is forwarded as text to the next downstream component. This allows the downstream component to begin processing before this component has received all pages.

   2. The response XML is parsed using the Result XPath expression "//lastRecordPredicate". The value "./login='jdoe'" is found.

   3. The XML request is resent whenever the Result XPath expression returns a value. The Request XPath expression is used to find the element to replace in the Request XML with the result found using the Result XPath expression.

   4. The result is then sent to the remote server. This time the element <previousPageLastRecordPredicate> is populated with the //lastRecordPredicate value, "./login='jdoe'".

3. Steps 1 and 2 repeat until the Request XPath expression result is not found or empty.