Question

I have a List<> of objects containing two strings and a DateTime. I want to build another list of the same objects containing only the last unique items using the two strings as keys and the last DateTime value. In SQL think the following:

SELECT col1, col2, MAX(datetime) FROM table GROUP BY col1, col2

This gives the unique list of col1, col2 and the last datetime. So.. I’m trying to do this in code with two lists. One with duplicates in it which parse and grab only the last unique items out of it to populate a second list.

The data sets I have are huge, so just going through the duplicate list then checking if the item is in the unique list, if it’s not adding it, if it is, comparing the dates etc.. is pretty slow. So I thought I could recursively go through the duplicate list and grab the unique items find their max datetime and delete the non max ones as I loop through, making my duplicate list smaller and smaller, thus speeding things up. (i hope your still following me..)

So anyway. I wrote a recursive loop with two lists, but when I loop through I get a System.StackOverflowException on about the 3000th iteration.

Here’s my code. Imagine the ListWithDuplicates is full of data. The actual ListDataItem has more properties I’ve left out. But my main question is why can’t I loop through the public list in this manner without causing the StackOverflowException?

using System;
using System.Net;
using System.IO;
using System.Collections.Generic;
using System.Linq;

public class RecursionTest
{
    public List<listDataItem> ListWithDuplicates { get; set; }
    public List<listDataItem> ListWithUniques { get; set; }

    public RecursionTest()
    {
        Process();
    }

    public void Process()
    {
        int rowcount = 0;
        int duplicates = 0;
        int total = 0;
        RecursiveLoopForUnique(ref rowcount, ref duplicates, ref total, "", "");
    }

    private void RecursiveLoopForUnique(ref int rowcount, ref int duplicates, ref int total, string col1, string col2)
    {
        if (rowcount > 0)
            duplicates += ListWithDuplicates.RemoveAll(z => z.COL1 == col1 && z.COL2 == col2);
        if (ListWithDuplicates.Count > 0)
        {
            foreach (listDataItem item in ListWithDuplicates)
            {
                rowcount++;
                if (ListWithUniques.FindAll(z => z.COL1 == item.COL1 && z.COL2 == item.COL2).Count < 1)
                {
                    ListWithUniques.Add(ListWithDuplicates.FindAll(z => z.COL1 == item.COL1 && z.COL2 == item.COL2).OrderByDescending(z => z.DATETIME).First());
                    col1 = item.COL1;
                    col2 = item.COL2;
                    break;
                }
            }
            RecursiveLoopForUnique(ref rowcount, ref duplicates, ref total, col1, col2);
        }
        else
            return;
    }

    public class listDataItem
    {
        public string COL1 { get; set; }
        public string COL2 { get; set; }
        public DateTime DATETIME { get; set; }            

        public listDataItem(string col1, string col2, DateTime datetime)
        {
            COL1 = col1;
            COL2 = col2;
            DATETIME = datetime;
        }
    }
}

Answer 1

How about this:

Dictionary<string, item> destDict = new Dictionary<string, item>();

foreach (item curr in items)
{
    string key = curr.col1 + curr.col2;
    if (!destDict.Keys.Contains(key))
    {
        destDict.Add(key, curr);
    }
    else
    {
        if (destDict[key].date < curr.date)
        {
            destDict[key].date = curr.date;
        }
    }
}

I tested this on a list containing 1000 each of 2 unique col1/col2 pairs. Worked fine and was faster than a LINQ groupby/select.