Question

We are using dtSearch to index some external web pages. It grabs the entire HTML content of the page.

When a page shows up in a list of search results on our web site, we want to show an excerpt of the content that contains their highlighted/bold search term as part of the result (in other words, the same thing everyone is used to seeing under each Google result).

What it the best way to accomplish this? Do you have to parse and remove the HTML tags? If so, how do you do that effectively?

We have a proof of concept working, showing the excerpt with the search terms highlighted, but we have to either render the tags, or try to strip them out (as we have tried) and end up with some garbage information that’s not really content.

The fact that we are using dtSearch is incidental, I think. If an alternative search tool is capable of doing this type of thing in our behalf, we’d consider using that instead.

We are basically trying to decide if we need to author our own regular expressions to accomplish this or if it’s a well-known problem that’s already solved by some library or tool.

We happen to be using .NET/C#. I don’t think it’s central to the problem but might impact what libraries we can use.

Answer 1

Google uses meta description tags where present, and will also use rich snippet information where available.

Beyond that, you may need to perform custom parsing, but don’t use regular expressions to perform the whole task. Rather, use a proper parser (such as HTML Aglity Pack) and find tags which make semantic sense (perhaps headings, paragraphs, etc.) Once you have located such elements, you might use a regex to determine which of the matched tags would give you the best snippet, where to truncate it, etc.

A simple flow:

1. parse the document and locate all elements with a significant amount
   of textual content.
2. strip inner tags (e.g. strong inside of a p)
3. prefer elements near the beginning of the document.
4. run an algorithm (possibly using regex(es), and possibly culture-aware) to try to extract sentences.
5. strongly prefer sentences with words matching one or more search terms (based on your stated requirements).
6. prefer sentences with few noise words.
7. (advanced) prefer sentences which have words occurring regularly in the document.
8. (advanced) combine multiple, potentially useful sentences into a single description snippet.

It’s not an exact science, even for Google.