Question

we are working on reducing some ‘pointless’ traffic.

if we send valid cache control headers (a-la “do cache”) with a 404 response, will modern browsers cache that the requested resource does not exist?

we use a custom php based 404 handler that was sending do not cache headers due to session creation, so we removed those. but now are wondering if for resources we are CERTAIN are invalid, should we send proper cache headers?

these are public resources btw accessed by the same uri by all users.

thanks!

Answer 1

Modern Browsers have an implementation of the HTTP Protocol ideally close to the standard, same for proxy servers which you should take into account if you like to have an effect on user- and non-intentional-bot-traffic. The current version is 1.1, some excerpts:

10.4 Client Error 4xx

The 4xx class of status code is intended for cases in which the client seems to have erred. Except when responding to a HEAD request, the server SHOULD include an entity containing an explanation of the error situation, and whether it is a temporary or permanent condition. These status codes are applicable to any request method. User agents SHOULD display any included entity to the user.

If the client is sending data, a server implementation using TCP SHOULD be careful to ensure that the client acknowledges receipt of the packet(s) containing the response, before the server closes the input connection. If the client continues sending data to the server after the close, the server’s TCP stack will send a reset packet to the client, which may erase the client’s unacknowledged input buffers before they can be read and interpreted by the HTTP application.

More specifically for 404:

10.4.5 404 Not Found

The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent. The 410 (Gone) status code SHOULD be used if the server knows, through some internally configurable mechanism, that an old resource is permanently unavailable and has no forwarding address. This status code is commonly used when the server does not wish to reveal exactly why the request has been refused, or when no other response is applicable.

So much for the 404 response code.

You asked:

now [we] are wondering if for resources we are CERTAIN are invalid, should we send proper cache headers?

You’ve not given any specific cache headers you’d like to use, so it’s a bit unprecise to answer your question. Caching is a section of it’s own in the protocol specification: 13 Caching in HTTP.

Generally the following should be what you’re looking for, excerpt from 13.1.1 Cache Correctness:

A correct cache MUST respond to a request with the most up-to-date response held by the cache that is appropriate to the request (see sections 13.2.5, 13.2.6, and 13.12) which meets one of the following conditions:

…

 3. It is an appropriate 304 (Not Modified), 305 (Proxy Redirect),
    or error (4xx or 5xx) response message.

So you can cache 404 response messages and signal them to cache. Proxies and Clients should handle it.