Question

I have a project that lies in a folder below the main domain, and I dont have access to the root of the domain itself.

http://mydomain.com/myproject/

I want to disallow indexing on the subfolder “forbidden”

http://mydomain.com/myproject/forbidden/

Can I simply put a robots.txt in the myproject folder? Will it get read even if there is no robots.txt in the root?

What is the correct syntax for disallowing the forbidden folder?

User-agent: *
Disallow: /forbidden/

or

User-agent: *
Disallow: forbidden/

Answer 1

From robotstxt.org:

Where to put it

The short answer: in the top-level
directory of your web server.

The longer answer:

When a robot looks for the
“/robots.txt” file for URL, it strips
the path component from the URL
(everything from the first single
slash), and puts “/robots.txt” in its
place.

For example, for
“http://www.example.com/shop/index.html,
it will remove the “/shop/index.html”,
and replace it with “/robots.txt”, and
will end up with
“http://www.example.com/robots.txt“.

So, as a web site owner you need to
put it in the right place on your web
server for that resulting URL to work.
Usually that is the same place where
you put your web site’s main
“index.html” welcome page. Where
exactly that is, and how to put the
file there, depends on your web server
software.

Remember to use all lower case for the
filename: “robots.txt”, not
“Robots.TXT.

So I’m afraid the answer is that you have to put it in the root folder 🙁

With regards to your second question, I believe the correct syntax is the one starting with a forward slash (eg. /forbidden/).