Home/ Questions/Q 154101
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-11T09:55:09+00:00

I have links with this structure: http://www.example.com/tags/ blah http://www.example.com/tags/ blubb http://www.example.com/tags/ blah / blubb

  • 0

I have links with this structure:

I want google & co to spider all links that have ONE tag in the URL, but NOT the URLs that have two or more tags.

Currently I use the html meta tag ‘robots’ -> ‘noindex, nofollow’ to solve the problem.

Is there a robots.txt solution (that works at least for some search bots) or do I need to continue with ‘noindex, nofollow’ and live with the additional traffic?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. I don’t think you can do it using robots.txt. The standard is pretty narrow (no wildcards, must be at the top level, etc.).

    What about disallowing them based on user-agent in your server?

    • 0