I’m using Quartz.NET to schedule a job which loads a bunch of data from external sources and persists in a database. After downloading them some processing have to be on them too, which will create additional records in a different table in the database.
The downloader job is a stateful job and runs in every minute. The problem I’ve faced is that, after downloading the data the processing part of it could take much longer time than I expected.
How should I manage this? I’ve though about creating another job (which will run for one time only) when the data downloading finishes. In this case the downloader job can run in every minute (this was the original plan), because the download part takes 5-20 secs only, and the other job can process these records after the downloader one has finished. The processer job would fetch records marked as unprocessed from database and do the work on them.
Is this a correct approach to handle the processing? The other idea I came is to set up a WCF service which would process one downloaded element. This would be called on each downloaded element. However I don’t think this would perform better than the other job approach.
Your approach to having the download job schedule another job to do the processing is feasible and used commonly.
One thing to keep in mind though is that this works as long as the download job doesn’t always schedule a processing job. If your download job always schedules a job (say every minute) and the scheduled job takes longer than that to run, you will eventually run out of processing threads and your download job will have to wait for a thread to be available.