Question

I am actually using PHP but such crawling can be done by any programming languages. It will be a bit difficult to cater a lot of situations. Please help me look through the problem, and please give me some suggestion on whether I am going to the right direction.

What I know is the current url address from which I can get a list of links from <a href=" or from <frame src=".

What I am doing is: from current url address, I can firstly get root url, for example, from http://www.abc.com/def, I can get http://www.abc.com first. This is to cater the situation <a href="/fff.html", so I have to know the root url first.

Secondly, I need to get url directory from current url, this is a little difficult and I still have no idea how to get it done perfectly. For example, from http://www.abc.com/def/xyz.htm, it’s url directory is http://www.abc.com/def. This is to cater the situation <a href="../../xyz.html">.

The problem I am facing is, how to get the current url directory? For example, if the current url is http://www.abc.com/def, how can I actually know that def is a directory or a file? If def is a file, then the url directory would be http://www.abc.com. But if def is a directory, then the url directory would just be http://www.abc.com/def.

You can say that if there is “/” at last, then it would be directory. But from my point of view, when I am crawling a webpage, I can’t really ensure that the webpage builder will add “/” at the end of a directory url. A directory url is perfect valid, for example, if def is a directory, then http://www.abc.com/def would probably stands for http://www.abc.com/def/index.html.

Since it’s hard to know whether http://www.abc.com/def is a directory or a script file, then it is hard to make full url from relative href such as <a href="xyz.html">.

Am I over complicating the problem? Is there any solution to this?

There are other situations for example href=”# means anchor then I’ll just append it to the end of current url. Is that correct and valid for any current url situation? Meaning that, is that valid for the situation where current url is http://www.abc.com/def (def is a directory), will http://www.abc.com/def#xyz be converted to http://www.abc.com/def/index.html#xyz ?

And for href=”javascript: or href=”vbscript: etc, I’ll just ignore it.

And for href=”xyz.???”, and if ??? is an image file, exe file, or anything that is not valid html, I’ll just ignore them?

Thanks.

The question might be a little messy, I hope I explained it clearly.

Answer 1

Anything after the domain name can map to whatever the person configuring the domain wants.

There is no guarantee that a URL ending in .html refers to a real file on a filesystem somewhere, or that it will return valid HTML, or anything else.

You can arbitrarily decide to count def/ as a directory or part of a filename, whatever floats your boat, as any choice is equally correct.