Question

The code below is designed to take a string in and remove any of a set of arbitrary words that are considered non-essential to a search phrase.

I didn’t write the code, but need to incorporate it into something else. It works, and that’s good, but it just feels wrong to me. However, I can’t seem to get my head outside the box that this method has created to think of another approach.

Maybe I’m just making it more complicated than it needs to be, but I feel like this might be cleaner with a different technique, perhaps by using LINQ.

I would welcome any suggestions; including the suggestion that I’m over thinking it and that the existing code is perfectly clear, concise and performant.

So, here’s the code:

private string RemoveNonEssentialWords(string phrase)
{
    //This array is being created manually for demo purposes.  In production code it's passed in from elsewhere.
    string[] nonessentials = {"left", "right", "acute", "chronic", "excessive", "extensive", 
                                    "upper", "lower", "complete", "partial", "subacute", "severe",
                                    "moderate", "total", "small", "large", "minor", "multiple", "early",
                                    "major", "bilateral", "progressive"};
    int index = -1;

    for (int i = 0; i < nonessentials.Length; i++)
    {
        index = phrase.ToLower().IndexOf(nonessentials[i]);
        while (index >= 0)
        {
            phrase = phrase.Remove(index, nonessentials[i].Length);
            phrase = phrase.Trim().Replace("  ", " ");
            index = phrase.IndexOf(nonessentials[i]);
        }
    }

    return phrase;
}

Thanks in advance for your help.

Cheers,

Steve

Answer 1

This appears to be an algorithm for removing stop words from a search phrase.

Here’s one thought: If this is in fact being used for a search, do you need the resulting phrase to be a perfect representation of the original (with all original whitespace intact), but with stop words removed, or can it be “close enough” so that the results are still effectively the same?

One approach would be to tokenize the phrase (using the approach of your choice – could be a regex, I’ll use a simple split) and then reassemble it with the stop words removed. Example:

public static string RemoveStopWords(string phrase, IEnumerable<string> stop)
{
    var tokens = Tokenize(phrase);
    var filteredTokens = tokens.Where(s => !stop.Contains(s));
    return string.Join(" ", filteredTokens.ToArray());
}

public static IEnumerable<string> Tokenize(string phrase)
{
    return string.Split(phrase, ' ');
    // Or use a regex, such as:
    //    return Regex.Split(phrase, @"\W+");
}

This won’t give you exactly the same result, but I’ll bet that it’s close enough and it will definitely run a lot more efficiently. Actual search engines use an approach similar to this, since everything is indexed and searched at the word level, not the character level.