Question

I am building a natural language processor in C#, and many ‘words’ in our database are actually multiple-word phrases that refer to one noun or action. Please, no discussion on this design call, suffice it to say it is not changeable at this time. I have string arrays of related words (chunks) of the sentence that I need to test for these phrases and words. What is an appropriately idiomatic way to handle sub-array extraction so I run the least risk of overflow errors and the like?

To give an example of the desired logic, let me step through a run with a sample chunk. For our purposes, assume that the only multiple-word phrase from the database is ‘quick brown’.

Full phrase: The quick brown fox -> encoded as {"The", "quick", "brown", "fox"}
First iteration: Test "The quick brown fox" -> returns nothing
Second iteration: Test "The quick brown" -> returns nothing
Third iteration: Test "The quick" -> returns nothing
Fourth iteration: Test "The" -> returns value
Fifth iteration: Test "quick brown fox" -> returns nothing
Sixth iteration: Test "quick brown" -> returns value
Seventh iteration: Test "fox" -> returns value

Sum all returned values and return.

I have some ideas of how to go about this but the more I look at things the more I am really getting worried about array addressing errors and other such horrors plaguing my code. The phrase is coming in as a string array, but I’m fine with putting it to IEnumerable. My only concern there lies in an Enumerable’s lack of an index.

Answer 1

The path forward here lay in combining Mark’s and Philipp’s answers. Under ideal circumstances I would have edited one of their posts with it but it appears as though my edits were denied.

Anyway, I took the DelimitedArray that Mark linked and changed a few things in it:

Constructor changed to:

    public DelimitedArray(T[] array, int offset, int count, bool throwErrors = false)
    {
        this.array = array;
        this.offset = offset;
        this.count = count;
        this.throwErrors = throwErrors;
    }

Index reference changed to:

public T this[int index]
    {
        get
        {
            int idx = this.offset + index;
            if (idx > this.Count - 1 || idx < 0)
            {
                if (throwErrors == true)
                    throw new IndexOutOfRangeException("Index '" + idx + "' was outside the bounds of the array.");
                return default(T);
            }
            return this.array[idx];
        }
    }

I then worked that in to Philipp’s loop usage. This becomes:

        for (var start = 0; start < words.Length - 2; start++) // at least one word
        {
            for (var end = start + 1; end < words.Length - 1; end++)
            {
                var segment = new DelimitedArray<string>(words, start, end - start);
                lemma = string.Join(" ", segment.GetEnumerator()); // get the word/phrase to test
                result = this.DoTheTest(lemma);

                if (result > 0)
                {
                    // Add the new result
                    ret = ret + result;

                    // Move the start sentinel up, mindful of the +1 that will happen at the end of the loop
                    start = start + segment.Count - 1;
                    // And instantly finish the end sentinel; we're done here.
                    end = words.Length;
                }
            }
        }

If I could accept more than one answer I’d mark both of their answers but as both of them are incomplete I will have to accept my own when I am able to do so tomorrow.