Question

Read the edit below for more information.

I have some code below that I use to split a generic list of Object when the item is of a certain type.

    public static IEnumerable<object>[] Split(this  IEnumerable<object> tokens, TokenType type) {

        List<List<object>> t = new List<List<object>>();
        int currentT = 0;
        t.Add(new List<object>());
        foreach (object list in tokens) {
            if ((list is Token) && (list as Token).TokenType == type) {
                currentT++;
                t.Add(new List<object>());

            }
            else if ((list is TokenType) && ((TokenType)list )== type) {
                currentT++;
                t.Add(new List<object>());

            }
            else {
                t[currentT].Add(list);
            }
        }

        return t.ToArray();
    }

I dont have a clear question as much as I am curious if anyone knows of any ways I can optimize this code. I call it many times and it seems to be quite the beast as far as clock cycles go. Any ideas? I can also make it a Wiki if anyone is interested, maybe we can keep track of the latest changes.

Update: Im trying to parse out specific tokens. Its a list of some other class and Token classes. Token has a property (enum) of TokenType. I need to find all the Token classes and split on each of them.

{a b c T d e T f g h T i j k l T m} 

would split like

{a b c}{d e}{f g h}{i j k l}{m}

EDIT UPDATE:
It seems like all of my speed problems come into the constant creation and addition of Generic Lists. Does anyone know how I can go about this without that?
This is the profile of what is happening if it helps anyone.

Answer 1

This is the best I could do to eliminate as much of the allocation times as possible for the function (should only allocate when it goes over the capacity, which should be no more than what is required to create the largest sub list in the results). I’ve tested this implementation and it works as you described.

Please note that the results of the prior sub list are destroyed when the next list in the group is accessed.

public static IEnumerable<IEnumerable> Split(this  IEnumerable tokens, TokenType type)
{
    ArrayList currentT = new ArrayList();
    foreach (object list in tokens)
    {
        Token token = list as Token;
        if ((token != null) && token.TokenType == type)
        {
            yield return currentT;
            currentT.Clear();
            //currentT = new ArrayList(); <-- Use this instead of 'currentT.Clear();' if you want the returned lists to be a different instance
        }
        else if ((list is TokenType) && ((TokenType)list) == type)
        {
            yield return currentT;
            currentT.Clear();
            //currentT = new ArrayList(); <-- Use this instead of 'currentT.Clear();' if you want the returned lists to be a different instance
        }
        else
        {
            currentT.Add(list);
        }
    }
}

EDIT
Here’s another version that doesn’t make use of another list at all (shouldn’t be doing any allocations). Not sure how well this will compare, but it does work as requested (also I’ve got no idea how this one will go if you try to cache a sub ‘array’).

Also, both of these require a “using System.Collections” statement (in addition to the Generic namespace).

private static IEnumerable SplitInnerLoop(IEnumerator iter, TokenType type)
{
    do
    {
        Token token = iter.Current as Token;
        if ((token != null) && token.TokenType == type)
        {
            break;
        }
        else if ((iter.Current is TokenType) && ((TokenType)iter.Current) == type)
        {
            break;
        }
        else
        {
            yield return iter.Current;
        }
    } while (iter.MoveNext());
}

public static IEnumerable<IEnumerable> Split(this  IEnumerable tokens, TokenType type)
{
    IEnumerator iter = tokens.GetEnumerator();
    while (iter.MoveNext())
    {
        yield return SplitInnerLoop(iter, type);
    }
}