I need to provide access to a private wiki to a crawler.
The wiki is closed to all anonymous users – you have to log in in order to see the contents, but I need to provide a single crawler (identified by a user-agent string and a single IP) full access so the contents can be indexed. It’s an internal crawler so access to its resources will only be available upon successful login.
Any suggestions on how to enable access to a single client (and not user, since a crawler is not able to log itself into the wiki)?
There actually is a solution to this problem.
As I mentioned a crawler will be using a specific IP and it will only be the crawler to use it. So quick and dirty but still a civilised way to do it is:
Simple, huh? 🙂