Question

I’m building this website with many effects that don’t work properly on mobile devices, so we built a simple html version and we redirect devices on it when they don’t support all the effects.

Normal version URL: www.mysite.com/#some_page_name

Simple version URL:www.mysite.com/static.php?p=some_page_name

The thing is we don’t want Google, Bing and Yahoo to give the simple version URLs in the search results, we always want people to land on the normal version of the site.

Is it possible to tell the search engine robots with Apache, PHP or JS, to ignore all the URLs that start with static.php ?

Answer 1

Put this in your robots.txt file:

User-agent: *
Disallow: /static.php

This will not only keep the pages out of the robots indexes, but also keep them from crawling them all together.