Question

Basically I do not want to be penalized by search engine for “duplicate contents”. I’m concerned that, in some cases, the webapp server may be generating exception on pages that would otherwise display fine.

For example, users may post links to the following two pages on the Internet:

www.example.org/nice-article

and

www.example.org/even-nicer-article

Now my webapp typically shall return two different pages and they shall contain great content that is going to be indexed fine by Google etc.

My concern is that in some cases my webapp server may encounter an exception when trying to serve these pages: there may be a bug that we didn’t caught on a previous release, the servlet may throw an exception because a resource it depends on is down, etc.

In that case, as far as I understand, there’s some exception catching going on but I’m not sure what I’m doing is correct.

In my web.xml, I’ve got something like this:

<error-page>
  <exception-type>java.lang.Throwable</exception-type>
  <location>/WEB-INF/jsp/error.jsp</location>
</error-page>

and

<error-page>
  <error-code>404</error-code>
  <location>/WEB-INF/jsp/error.jsp</location>
</error-page>

I’m not sure my web.xml is correct here.

My concern is that in the case of an exception getting caught, I display a message from error.jsp telling something like “An internal error occured, please try again later” but that for whatever reason I’d be sending this as a normal page and not as an error-page (and that hence all the broken page would get referenced as “duplicate contents” because they’d all be displaying more or less the same message).

Basically I’d like to know two things:

• what status code should I throw if a resource that is normally available is down when the crawling bots (and the clients’ browsers) tries to access it (it’s a 500 right?)

• how do I configure Tomcat to generate such a status code

Additionally it would be great if you could explain a way I could test this is working: for example can I telnet in and see if I get the correct error code?

Answer 1

You may want to use 503 error for temporary unavailable pages. This will tell any fair bot that the unavailability is temporary and they need to come back later to crawl the page.

Your error part of web.xml seems ok to me – how you checked that you’re sending that message as a “normal page”? (which I think – correct me if I’m wrong – means you’re getting HTTP 200 instead of HTTP 404)