Question

I am using htaccess in my site, such that all the request to my site will be redirected to index page in my root directory. No other file in my site can be accessed because my htaccess will restrict it. My doubt is, when I use robots.txt file, will the search engines be able to reach the robots.txt file in my domain?. Or must i modify my htaccess file to allow the search engines to read the robots.txt file. If so help me in finding that specific code for htaccess.

Answer 1

I suppose you’re using some sort of rewriting. You can exclude files from beeing processed by mod_rewrite with the following rule:

RewriteCond %{REQUEST_FILENAME} -f
RewriteRule ^(.*) $1 [L]

In this case all files that do exist in the webroot and are called directly will not be processed any further. Replace _%{REQUEST_FILENAME} with robots.txt and you should be fine.

Best wishes,
Fabian