We’ve now got plenty of sites which all use a log4net base error loging framework and we receive error from site from anywhere it append. We’ve notice that some of them catch error because of “Bot” like google, bing, yahoo, etc. But there’s a things we’ve not sure about how to resolve. I’ve two questions about it :
- Is “Java/1.6.0_24” a Bot? Because the user-agent of my question #2 is about this.
- The “Java/1.6.0_24” still calling subfolder on our site that just do not exists! Like, if we have a page called “Page1.aspx”, instead of calling “~/Page1.aspx”, he calls it “~/minisite/Page1.aspx”. How can I tell him he’s wrong? Is there a way to do it?
Thanks you
It’s most likely a bot but it could as well be some kind of browser based on Java that sends that user-agent string – you can’t trust it 100% but it can give you an estimate idea of what the connecting entity is. Depending on the kind of bot it might as well just ignore your robots.txt so I’d just impement some handling stuff somewhere.
Did those folders ever exist? If so, you could use HTTP’s permanent redirect (code 301) to tell him to no longer look there – however that doesn’t guarantee it will do so.