Question

I am returning a large amount of data via a rest service. The query returns about 200,000 rows of data then gets converted to XML. When I run this service in IE8 I get an error that “there is not enough storage to complete this operation”.

Is there any best practice for returning a large amout of data this way?

The database run the query in about 5 seconds, so I’m guessing that the problem is JAXB converting it to xml.

Does anyone have any ideas to improve this?

Answer 1

The Problem

I’m assuming that you are using JPA to get your data as objects that can be handled by JAXB. If this is the case the JPA objects may be using lazy loading meaning that a query may not realize all the data at once. However as the JAXB implementation traverses the object graph, more and more of it is brought into memory until you run out.

Option #1 – Provide Data in Chunks

One approach is to return your data in chunks and offer a URI like the one below:

• http://www.example.com/customers?start=10&size=100

These parameters tie in very nicely to JPA query settings:

namedQuery.setFirstResult(10);
namedQuery.setMaxResults(100);

Option #2 – Provide Links to Get More Data

Alternatively, instead of including all the data you can provide links to get more. For example instead of:

<purchase-order>
    <product id="1">
        <name>...</name>
        <price>...</price>
        ...
    </product>
    <product id="2">
        <name>...</name>
        <price>...</price>
        ...
    </product>
    ...
</purchase-order>

You could return the following, the client can then request details on products using the provided URI. You can map this in JAXB using an XmlAdapter.

<purchase-order>
    <product>http://www.example.com/products/1</product>
    <product>http://www.example.com/products/2</product>
    ...
</purchase-order>

For more information on XmlAdapter see:

• http://bdoughan.blogspot.com/2010/07/xmladapter-jaxbs-secret-weapon.html