Question

Will this robots.txt file only allow googlebot to index my site’s index.php file? CAVEAT, I have an htaccess redirect that people who type in

http://www.example.com/index.php

are redirected to simply

http://www.example.com/

So, this is my robots.txt file content…

User-agent: Googlebot
Allow: /index.php
Disallow: /

User-agent: *
Disallow: /

Thanks in advance!

Answer 1

Not really.

Good bots
Only “good” bots follow the robots.txt instructions (not all robots and spiders bother to read/follow robots.txt). That might not even include all the main search engine’s bots, but it definitely mean that some web crawlers will just completely ignore your requests (you should look at using .htaccess or password protection if you really want to stop bots/crawlers from seeing parts of your site).

Second checks
Google makes multiple visits to your website, including appearing as a browsing user. This second visit will ignore the robots.txt file. The second visit probably doesn’t actually index (if that’s your worry) but it does check to make sure you’re not trying to fool the indexing bot (for SEO etc).

That being said your syntax is right… if that’s all you’re asking, then yes it’ll work, just not as well as you might hope.