Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I have a script which runs through a database table and downloads a file for each row, adds to a results table in memory, then bulk uploads all the results back to the database once finished.
The problem I have is that there could be thousands of files to download and the script could timeout or error half way through.
Is there a better approach to this, maybe involving threading or asynchronous calls?
Threading seems to be the way to go..
you should have one, or many threads that reads rows from the db(if you want many threads, you should partition the read accordingly) and putting them in some sort of concurrent collection(either .net 4 build-in ones, or built/download a custom one).
then you should have a thread collection who will get items from that list and get the file, if he timeouts, he should put the task back to the collection..
this is a basic producer-consumer threading pattern.
you can easly find many examples in google.