Home/ Questions/Q 4621218
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-22T02:40:08+00:00

I have a site with the following structure: http://www.example.com/folder1/folder2/folder3 I would like to disallow

  • 0

I have a site with the following structure:

http://www.example.com/folder1/folder2/folder3

I would like to disallow indexing in folder1, and folder2.
But I would like the robots to index everything under folder3.

Is there a way to do this with the robots.txt?

For what I read I think that everything inside a specified folder is disallowed.

Would the following achieve my goal?

user-agent: *
Crawl-delay: 0

Sitemap: <Sitemap url>

Allow: /folder1/folder2/folder3
Disallow: /folder1/folder2/
Disallow: /folder1/
Allow: /
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Yes, it works… however google has a tool to test your robots.txt file

    you only need to go on google webmaster tools (https://www.google.com/webmasters/tools/)

    and open the section “site configuration -> crawler access”

    • 0