Question

I have some pages names index2.php, index3.php,..
they are copies of my index.php, for testing purposes
should I disallow these pages in robots.txt because of the double content?
or it doesn’t matter because these pages are not linked to on my website?
if I get punished by google for the double content, how bad is it?

Answer 1

Dupe content or anything that can knock you down, can be very hard to recover from. Overall I would say it depends on what your doing with your site/service. If its all about content, then yea its going to matter to some extent or another. Once google and other engines list you in a pool of domains for dupe content or other possible flags that would be an attempt to falsely boost your rank, getting out of that pool is tough. Its like trust, once you lose it for someone its hard for them to gain it back.

Then again its hard to say, Ive known some who have been plauged for months on end trying to fix similar issues, and I know some that fixed it within a week.

Again it all boils down to what your site/service is doing, and how its doing it. So many factors, that not just one alone will kill..

As for that whole robots text thing, if your that concerned go for it, theres no harm in it. Engines like google pay mind to the robots text and respect it enough. Without it, despite not being linked directly google and or other engines sometimes find a way to find it. Ive had whole sub domains never seen by the public end up getting indexed in search engines, I have had thee most obsucre file names never linked publicly end up in search engines.. its hit or miss