Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I’m working an application that periodically fetch data from a web service. The problem is, the web service does not expose a method to fetch data with a range of time. So I have to fetch all the data and drop the out-dated entries. I think this will be a problem when the scale of return data become large. Is there anyway to enhance the performance? Thanks in advance!
If possible you should have a overloaded method wherein you can pass startDate and endDate.
If this is not possible then your best bet is to find the fastest way to get the data and filter it.
If I assume that you get a data table from the WS method , you can write a LINQ query to filter based on the date range columns and then bulk load to you destination.if for example it is MS Sql Server where you loading data , you can do SQLBulkCopy in first step. There must be similar meathods to do this bulk load.
Once the initial Load is done you can load data to your tables by doing proper indexing on the source tables.
The question is very much centric around the volume of data you might get from WS , what is you destination where data will be loaded , what are the most optimized drivers that you can use for dataread/write and if you can modify the WS or not (I presume not from your question).
If you let me more about these things I might be able to give to a specific answer to improve the performance of your load.