Question

I’m trying to generate a sitemap dynamically for a large web site with thousands of pages.

Yes, I have considered generating the sitemap file offline and simply serving it statically, and I might end up doing exactly that. But I think this is a generally useful question:

How can I stream large data from a DB in Wicket?

I followed the instructions at the Wicket SEO page, and was able to get a dynamic sitemap implementation working using a DataProvider. But it doesn’t scale- it runs out of memory when it calls my DataProvider’s iterator() method with a count arg equal to the total number of objects I’m returning, rather than iterating over them in chunks.

I think the solution lies somewhere with WebResource/ResourceStreamingRequestTarget. But those classes expect an IResourceStream, which ultimately boils down to providing an InputStream implementation, which deals in bytes, rather than DB records. I wouldn’t know how to implement the length() method in such a case, as that would require visiting every record ahead of time to compute the overall length.

Answer 1

I ended up using an AbsractResourceStreamWriter subclass:

public class SitemapStreamWriter extends AbstractResourceStreamWriter
{
    @Override
    public void write(OutputStream output)
    {
        String HEAD = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" +
                      "<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\"\n" +
                       "        xmlns:wicket=\"http://wicket.apache.org/dtds.data/wicket-xhtml1.4-strict.dtd\">\n";
        try
        {
            output.write(HEAD.getBytes());

            // write out a <loc> entry for each of my pages here

            output.write("</urlset>\n".getBytes());
        }
        catch (IOException e)
        {
            throw new RuntimeException(e.getMessage(), e);
        }
    }
}