Question

Selenium (RC) is being used to test an ASP.NET 1.1 site.

When we make a request via Selenium RC (which in turn automates the request via a configured browser – in this case Firefox) the http verb is “HEAD”. We have several form action methods that have separate GET and POST methods decorated with AcceptVerbs(HttpVerbs.Get) or HttpVerbs.Post respectively. These methods are returning a 404 and logging a “a public action method could not be found” error message.

Questions:

• When writing separate Get/Post action methods what is the best practice for handling the Head verb? Should we always decorate with an AcceptVerbs(HttpVerbs.Get | HttpVerbs.Head)?

• Why is the HEAD verb being generated when Selenium RC is automating the browser in lieu of an If-Modified-Since header?

• We’ve also seen log entries from (non-mainstream) crawlers that are using the HEAD verb. We created robots.txt entries to stop these crawlers from indexing the site, but now we’re wondering what the best practice from an SEO perspective is as well. Is it important to respond to HEAD for crawlers? Are there mainstream crawlers that use it? Does it impact SEO rank?

Answer 1

1. Yes, I think that whenever you are restricting your requests to be GET only, you should always allow HEAD on them as well – in fact, I do think it should be built into the MVC framework (next thing on my todo list: raise the issue in MVC bug tracker that [HttpGet] attribute should somehow support HEAD verb)
2. I would like to know an answer to this too. In the meantime, there is a suggested workaround – pass ‘true’ as a second param to Selenium’s open().
3. I don’t think it impacts SERP ranking per se, however I can see how crawlers would not request the full page if HEAD gives a 404. According to the HTTP spec (RFC2616), “The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response”, so, if you are doing things right – it shouldn’t be a problem to allow the method and avoid getting unlisted.