I am a bit new at search engine optimisation, and I am just trying to write a robot.txt file. I want to disallow pages in the terms and conditions folder, which has a space in it.
Should I write:
Disallow: /Page TermsAndConditions/
Or:
Disallow: /Page%20TermsAndConditions/
Or both, or does it not matter?
(I want to disallow this folder because this is full of technical jargon that I think is messing up the content keywords found by the googlebot – or do I have this wrong as well?).
Edit:
I found this page from googe Robots.txt Specifications. Which says: “Non-7-bit ASCII characters in a path may be included as UTF-8 characters or as percent-escaped UTF-8 encoded characters per RFC 3986”. So I guess the answer to this question is that it doesn’t matter.
Rename the folder without a space and avoid the issue.