Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I am writing a web crawler to search for files and download. My problem is I do not want to download the same files that are downloaded already to the local drive. I know it’s possible to use the MD5 hash to compare but how can I do this on HTTP URL without downloading them to the local disk?
If this approach is wrong. Please advice on a better solution
Unless the webserver has some sort of service that shares the MD5, then No.
Computing a file hash requires every byte in the file. This is why changing a single byte changes the hash, to prevent getting modified files.