Home/ Questions/Q 9038937
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-16T09:36:45+00:00

my robots.txt looks like this: User-agent: * Disallow: /admin Disallow: /test User-Agent: Googlebot Disallow:

  • 0

my robots.txt looks like this:

User-agent: *
Disallow: /admin
Disallow: /test

User-Agent: Googlebot
Disallow: /maps

Now Google ignores the user-agent * part and only obeys the specific Googlebot directives (/maps). Is this normal behaviour and shouldn’t also obey the useragent * directives (/admin, /test)?

It seems strange to have to add every line for every useragent?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Never mind, Google states this:

    Each section in the robots.txt file is separate and does not build
    upon previous sections. For example:

    User-agent: * Disallow: /folder1/

    User-Agent: Googlebot Disallow: /folder2/

    In this example only the URLs matching /folder2/ would be disallowed
    for Googlebot.

    • 0