Question

My web development team has a version control system setup with two branches. One of these branches are for development, the other has the stable code which is displayed on our primary domain. For ease of testing during development, we have a second domain from which the development branch is served.

I want to maintain our robots.txt file in the version control system, but I don’t want our development website showing up in search results. If I put a robots.txt file in the development branch that disallows all indexing, the development domain will serve that file solving one of my problems. But, as soon as I merge that change with the stable branch, that same file will get served from the primary domain, which isn’t what I want. What would you do in this situation to resolve both issues?

Answer 1

Another approach, if you’re using Apache:

1. Create two robots.txt files instead of one: robots-production.txt, and robots-development.txt

2. Add two .htaccess rules:

   RewriteCond %{HTTP_HOST} ^your_development_domain$
   RewriteRule ^robots.txt robots-development.txt
   
   RewriteCond %{HTTP_HOST} ^your_production_domain$
   RewriteRule ^robots.txt robots-production.txt
   

This serves a different robots.txt file for each domain.