Question

I’m playing around with a XML API where the search doesn’t support paging/limit. The recommended way is to just request all the IDs and then in a second request get the data and handle paging on your own.

First request:

http://example.com?search=foobar&columns=ID

<results>
  <item><id>1</id></item>
  <item><id>2</id></item>
  <item><id>3</id></item>
  <item><id>4</id></item>
  <item><id>5</id></item>
  <item><id>6</id></item>
  <item><id>7</id></item>
  <item><id>8</id></item>
  <item><id>9</id></item>
  <item><id>10</id></item>
</results>

Second request:

http://example.com?search=1,2,3,4,5&columns=ID,title,description

<results>
  <item><id>1</id><title>foobar</title><description /></item>
  <item><id>2</id><title>foobar</title><description /></item>
  <item><id>3</id><title>foobar</title><description /></item>
  <item><id>4</id><title>foobar</title><description /></item>
  <item><id>5</id><title>foobar</title><description /></item>
</results>

Is it possible with YQL to combine this into a single request with a search result count and paging support?

Answer 1

Create a YQL table with paging:

<paging model="offset" matrix="true">  
    <start id="internalIndex" default="0" />  
    <pagesize id="internalPerPage" max="250" />
</paging>

Use Javascript to handle the two fetches and the paging:

var internalIndex = parseInt( request.matrixParams['internalIndex'] );
var internalPerPage = parseInt( request.matrixParams['internalPerPage'] );

var interimURL = 'http://example.com?columns=ID';
interimURL += '&search=' + request.queryParams['search'];
var interimQueryParameter = {url:interimURL};
var interimQuery = y.query("SELECT * FROM xml WHERE url=@url", interimQueryParameter);

var rows = interimQuery.results.*;

// get subset
var xml    = rows;//this.copy(); // clone XML
var from = internalIndex;
var to = ((from + internalPerPage) < xml.length()) ? (from + internalPerPage) : xml.length();
var sliced = [];
for (; from < to; from++) {
    sliced.push(xml[from].@["ID"]);
}

var finalURL = 'http://example.com?';
finalURL += '&search=' + sliced.join(",");
finalURL += "&columns=ID,title,description"
var finalQueryParameter = {url:finalURL};
var finalQuery = y.query("SELECT * FROM xml WHERE url=@url", finalQueryParameter);

var finalResults = finalQuery.results.response;
finalResults.node += <internalCurserPositon>{internalIndex}</internalCurserPositon>
finalResults.node += <internalCount>{rows.length()}</internalCount>

response.object = finalResults;