Home/ Questions/Q 927845
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-15T19:55:03+00:00

My URL structure is set up in two parallels (both lead to the same

  • 0

My URL structure is set up in two parallels (both lead to the same place ):

  • www.example.com/subname
  • www.example.com/123

The trouble is is that, the spiders are crawling into things like:

  • www.example.com/subname/default_media_function
  • www.example.com/subname/map_function

Note that the name “subname” represents thousands of different pages on my site that all have that same function.

And they are throwing out errors because those links are strictly for JSON or AJAX purposes and not actual links. I would like to block them from accessing those pages, but how would I do that if the URL contains a variable?

Would this work in robots.txt?

Disallow: /map_function
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You are going to have to do 

    Disallow: /subname/map_function

    The robots will look for the robots.txt at root level. Also there they evaluate URLs left to right with no wildcards.

    So, you will either need to make one location for all the map_function and exclude that, or exclude all locations.

    • 0