Question

I have some bad performance issues in my application. One of the big operations is comparing strings.
I download a list of strings, approximately 1000 – 10000. These are all unique strings.
Then I need to check if these strings already exists in the database.
The linq query that I’m using looks like this:

IEnumerable<string> allNewStrings = DownloadAllStrings();

var selection = from a in allNewStrings
                where !(from o in context.Items
                        select o.TheUniqueString).Contains(a)
                select a;

Am I doing something wrong or how could I make this process faster preferably with Linq?

Thanks.

Answer 1

You did query the same unique strings 1000 – 10000 times for every element in allNewStrings, so it’s extremely inefficient.

Try to query unique strings separately in order that it is executed once:

IEnumerable<string> allNewStrings = DownloadAllStrings();

var uniqueStrings = from o in context.Items
                    select o.TheUniqueString;

var selection = from a in allNewStrings
                where !uniqueStrings.Contains(a)
                select a;

Now you can see that the last query could be written using Except which is more efficient for the case of set operators like your example:

var selection = allNewStrings.Except(uniqueStrings);