Question

I am using MVC to develop my website. I am getting the following errors.

URL:http://www.abc.com/robots.txt
1. The controller for path ‘/robots.txt’ was not found or does not implement IController.

URL:http://www.abc.com/blogs/post/whats-new-in-mvc
2. The controller for path ‘/blogs/post/whats-new-in-mvc’ was not found or does not implement IController..

But i dont have the above mentioned url in my website. How the above url are generated? Can you please let me know the solution to fix the above issue?

Answer 1

The first one is used by search engines to index your website. It’s a good practice to have a robots.txt file to your application. So you could add this file to the root of your site. As far as the second url is concerned, I have strictly no idea who is querying it. Maybe somewhere inside your site you have a link to this url?

But if you don’t want to use this file you could exclude it from routing:

routes.IgnoreRoute("robots.txt");

Now when a search engine sends a request to this file he will also get a 404 but the request won’t be routed through the MVC pipeline.