Question

For some reason, I want to serve my robots.txt via a PHP script. I have setup apache so that the robots.txt file request (infact all file requests) come to a single PHP script.

The code I am using to render robots.txt is:

echo "User-agent: wget\n";
echo "Disallow: /\n";

However, it is not processing the newlines. How to server robots.txt correctly, so search engines (or any client) see it properly? Do I have to send some special headers for txt files?

EDIT 1:

Now I have the following code:

header("Content-Type: text/plain");
echo "User-agent: wget\n";
echo "Disallow: /\n";

which still does not display newlines (see http://sarcastic-quotes.com/robots.txt ).

EDIT 2:

Some people mentioned its just fine and not displayed in browser. Was just curious how does this one display correctly: http://en.wikipedia.org/robots.txt

EDIT 3:

I downloaded both mine and wikipedia’s through wget, and see this:

$ file en.wikipedia.org/robots.txt
en.wikipedia.org/robots.txt: UTF-8 Unicode English text

$ file sarcastic-quotes.com/robots.txt
sarcastic-quotes.com/robots.txt: ASCII text

FINAL SUMMARY:

Main issue was I was not setting the header. However, there is another internal bug, which is making the Content-Type as html. (this is because my request is actually served through an internal proxy but thats another issue).

Some comments that browsers don’t display newline were only half-correct -> modern browsers correctly display newline if content-type is text/plain. I am selecting the answer that closely matched the real problem and was void of the above slightly misleading misconception :). Thanks everyone for the help and your time!

thanks

JP

Answer 1

Yes, you forgot to set the Content Type of your output to text/plain:

header("Content-Type: text/plain");

Your output is probably being sent as HTML, where a newline is truncated into a space, and to actually display a newline, you would need the <br /> tag.