Home/ Questions/Q 5976211
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-22T21:11:45+00:00

How do I prevent spriders crawling pages that start with mydomain.com/abc… For example mydomain.com/abcSGGSHS

  • 0

How do I prevent spriders crawling pages that start with mydomain.com/abc…

For example mydomain.com/abcSGGSHS or mydomain.com/abc6bNNha

I think I need to add some sort of regular expression to the web root’s .htaccess, right?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    With mod_rewrite enabled, you can do the following

    RewriteEngine on 
RewriteBase / 
RewriteCond %{HTTP_USER_AGENT} ^YourBadSpiderName [OR]
RewriteCond %{HTTP_USER_AGENT} ^AotherBadSpider
RewriteCond %{REQUEST_URI} ^abc
RewriteRule ^$ http://mydomain.com/404.html [NC,L]

    You’ll have to update the spider names accordingly. If a bot changes his user agent, let’s say to ‘Mozilla/Firefox’, you’re out of luck..

    • 0