Question

In some library code, I have a List that can contain 50,000 items or more.

Callers of the library can invoke methods that result in strings being added to the list. How do I efficiently check for uniqueness of the strings being added?

Currently, just before adding a string, I scan the entire list and compare each string to the to-be-added string. This starts showing scale problems above 10,000 items.

I will benchmark this, but interested in insight.

• if I replace the List<> with a Dictionary<> , will ContainsKey() be appreciably faster as the list grows to 10,000 items and beyond?
• if I defer the uniqueness check until after all items have been added, will it be faster? At that point I would need to check every element against every other element, still an n^^2 operation.

---

EDIT

Some basic benchmark results. I created an abstract class that exposes 2 methods: Fill and Scan. Fill just fills the collection with n items (I used 50,000). Scan scans the list m times (I used 5000) to see if a given value is present. Then I built an implementation of that class for List, and another for HashSet.

The strings used were uniformly 11 characters in length, and randomly generated via a method in the abstract class.

A very basic micro-benchmark.

Hello from Cheeso.Tests.ListTester
filling 50000 items...
scanning 5000 items...
Time to fill: 00:00:00.4428266
Time to scan: 00:00:13.0291180

Hello from Cheeso.Tests.HashSetTester
filling 50000 items...
scanning 5000 items...
Time to fill: 00:00:00.3797751
Time to scan: 00:00:00.4364431

So, for strings of that length, HashSet is roughly 25x faster than List , when scanning for uniqueness. Also, for this size of collection, HashSet has zero penalty over List when adding items to the collection.

The results are interesting and not valid. To get valid results, I’d need to do warmup intervals, multiple trials, with random selection of the implementation. But I feel confident that that would move the bar only slightly.

Thanks everyone.

EDIT2

After adding randomization and multple trials, HashSet consistently outperforms List in this case, by about 20x.

These results don’t necessarily hold for strings of variable length, more complex objects, or different collection sizes.

Answer 1

You should use the HashSet<T> class, which is specifically designed for what you’re doing.