Question

I guess all everybody else would like the opposite but i need to hide the “directory” structure of an ASP.NET website.

For this reason i am thinking of

1. Use the robots.txt as follows…

   User-agent: *
   Disallow: /

2. Use Url rewriting, to make ghost paths

3. Disable directory browsing. (Directory listing denied…)

4. Use .ashx to serve images.

5. Other ways that you may suggest.

In other words i would not like someone with a “downloader-structrure reader”, strip my site.

As you see the sekurity tag is missing 🙂

P.S. I do not care about SEO

Answer 1

A site downloader like wget -r will work anyway. It follows links and doesn’t care about directories (except the fact that you can limit the depth). If you want to exclude legit crawlers like Google, using robots.txt is fine. wget and rogue crawlers don’t care about it though.

The only really good solution is either requriring a login (but that still doesn’t protect you against those people who’d just use wget to download your whole site; they’ll just provide it with the login information/session id) or cluttering your content with annoying CAPTCHAs (pisses of legit users quickly) or making the whole site use JavaScript/AJAX to display/load content. That makes the user experience even better (if it’s done properly) and effectively locks out most/all crawlers.