Question

Let’s say I have the following text (from wiki):

Java is a programming language
originally developed by James Gosling
at Sun Microsystems (which is now a
subsidiary of Oracle Corporation) and
released in 1995 as a core component
of Sun Microsystems’ Java platform.
The language derives much of its
syntax from C and C++ but has a
simpler object model and fewer
low-level facilities. Java
applications are typically compiled to
bytecode (class file) that can run on
any Java Virtual Machine (JVM)
regardless of computer architecture.
Java is a general-purpose, concurrent,
class-based …

And I would like to parse out “java” and “programming” matches into google style result like this:

Java is a programming language
originally developed by James Gosling
at Sun Microsystems … Java
applications are typically compiled to
bytecode (class file) that can run on
any Java Virtual Machine (JVM)…

1. What tools could I use and how should I used them to get the above result. Commons, Lucene, Compass?

2. If there is an algorithm that will highlight the keywords and take care of cutting the strings and adding “…” at the end, please share it.

3. How do you decide how many and which keywords to display in search result preview?

Answer 1

I don’t know of any tools that will help with this, but I can offer an algorithm that will give you fairly decent results. *Edit: OP asked for sample code for the index. I am using a Trove TIntObjectHashMap to store this information, but you could do the same with a Java HashMap.

Step 1: Search the text for each of the search words and make an index of the offset within the text that each of them appears.

TIntObjectHashMap<String> matchIndex = new TIntObjectHashMap<String>();
// for each word or other string to highlight
// find each instance of each word in the string
// this is pseudocode -v
for (each instance of String searchString appearing at index int x)
  matchIndex.put(x, searchString);

Step 2: Go through each combination of pairs of indices in Step 1 and record the number of characters between indices and number of hits.

// class to hold a match
private class Match implements Comparable {
  private int x1, x2;
  private int hitCount;

  public Match(int x1, int x2, int hitCount); // does the obvious

  private double sortValue() {
    return (double) hitCount / Math.abs(x1, x2);
  }  

  @Override
  public int compareTo(Match m) {
    double diff = this.sortValue() - m.sortValue();
    if (diff == 0.0) return 0;
    return (diff < 0.0) ? -1 : 1;
  }
}

// go through every combination of keys (string offsets) and record them
// the treeset will automatically sort the results
TreeSet<Match> matches = new TreeSet<Match>();
int[] keys = matchIndex.keys();
for (int x1 = 0; x1 < keys.length; x1++)
  for (int x2 = x1 + 1; x2 < keys.length; x2++)
    matches.put(new Match(keys[x1],
                          keys[x2] + matchIndex.get(keys[x2]).length(),
                          1 + x2 - x1));

Step 3: Take the list generated in Step 2 and sort them by number of hits per character of length.

// nicely done by the TreeSet

Step 4: Start at the top of the list in Step 3 and mark each item as being included. Be sure to combine overlapping results into one larger result. Stop when the next item would push the total length of the string over 255 (or so) characters.

Step 5: Display each of the selected items in Step 4, in order, with “…” between them. Be sure to include whatever markup is necessary to highlight the actual search words themselves within each item.