Question

I’m going to write a program that takes a URL and counts the occurrences of EVERY single 1-word, 2-word, and 3-word phrases in the webpage (and possibly x-word phrases).

Here’s the best algorithm I could come up with:

1). strip html tags

2) make everything lowercase

3) split the text on space and put them all into an array

4) iterate over each word, and for each word you must: put word[i], word[i+1], word[i+2] into a hashtable.

Every time u have a collision you increase the word count for that word or 2-3 letter word phrase.

My questions are:

1) Can anyone provide any more efficient solutions in terms of space and runtime?

2) Are there any easy ways to do #1 in C#?
I can probably use a dom parser and parse out all the inner text maybe.

Answer 1

Depending on your case, You might be oversimplifying the problem and/or You may end up putting a lot of effort implementing functionalites that already exist in some libraries. So this will not be an direct answer but suggestion on what path to take in tackling this problem.

Process You want to implement is called information retrieval. It is very broad and complex but luckily there is a lot of research in this area. Part of it is extracting word ngrams (ngram is set of consecutive letters or words in sequence).

Let me show you some additional problems you should think of ahead:

• is the capitalization of letters in word important?
• is dot the only sign that You want to use to mark the end of sentence?
• do You want to exclude stop words? Stop words are words You don’t want to include in phrase like ‘a’, ‘the’, ‘I’, ‘my’ and so on.
• do you want to stem words? Convert words from their original form to their root form, like plural to singular form: basketballs -> basketball

And for extracting pure text from HTML:

• extract only text shown on page?
• extract hints also? (like those shown when hovering mouse over picture)
• Any other non-visible text (meta tag and so on)

There are libraries that perform searching and extracting information from raw material. “Raw material” means that You have to process document (html, doc, pdf, image, …) and turn it into text in order for search engine to index it (extract phrases, for instance). Once document is indexed it can be searched. One such library for .NET is Lucene.NET. It supports different stemmers, analyzers, filters.

I am not sure but i believe there are libraries for extracting text from html also.

Basically, your approach may work in some simpler scenarios where not so small error-level is acceptable. I recently gain interest in information retrieval and found it really complex and interesting. You may get benefits researching this topic depending on your goals. There is a lot of info here on stackoverflow as well as the rest of Internet.

And if You decide to go this way, there is much more info on Lucene (orioginal Lucene JAVA version, Lucene.NET is port to .NET) than on Lucene.NET. So if You don’t find answer for Lucene.NET immediately do a search on Lucene discussions.