Due to some rather bizarre architectural considerations I’ve had to set up something that really ought to run as a console application as a web page. It does the job of writing a large variety of text files and xml feeds from our site data for various other services to pick up so obviously it takes a little while to run and is pretty processor intensive.
However, before I deploy it I’m rather worried that it might get hit repeatedly by spiders and the like. It’s fine for the data to be re-written but continual hits on this page are going to trigger performance issues for obvious reasons.
Is this something I ought to worry about? Or in reality is spider traffic unlikely to be intensive enough to cause problems?
You can tell the big ones not to spider you; http://www.robotstxt.org.
You could also implement some form of authentication/ip address criteria that would prevent it from running.