Question

Suppose i have a post which is something like

• TITLE: “WEB: SEO in 2011”
• DESCRIPTION: “A conference talking about SEO in the web of the 2011”

also, i have a list of categories with keywords associated:

• “IT” (cat) -> “Web design”, “seo”, “developing”, “web developing” (keywords)

i have multiple categories (it, arts, medicals, literature, machinery etc…)

i need to use java to automatically upgrade my posts with those categories and keywords (a sort of tagging) to improve future searching.

The example above: should match “seo” and “web” so the main_category field should be filled up with “IT” and the subfield_category should be filled up with “seo” or “web” (or maybe both, which isn’t bad too)

my problem is that the only solution i can come up with is waaaaay into the bruteforcing (test all the words, when one matches you have the category and the list of the keywords associated with it) and it will slower my performances…

is there any way i can do the search in a better way? also i can modify my category->keywords structure to do something better (i still don’t know how…)

thanks all in advance!

EDIT: accuracy isn’t so much important, as amit asked in a comment. i don’t need 100% accuracy on tagging, since i know i can have an honest amount of correctness based on raw matching of the strings.

Also, the logic i was thinkinking about is: look at post title/description, search for any keywords matching, tag with category, search for more keywords into this category, save 3 to 5 matching keywords

Answer 1

You might want to try a different approach, using Machine Learning.

Algorithm Description:

First, create a learning samples [documents you know for sure how they should be tagged, you can tag a sample manually and give it as input to the algorithm]. Then, create Bag Of Words for these samples, using k bag of words [you will need to decide which k is optimal, by benchmarking the quality, I’ll explain later on].

Every word is a ‘feature’, and next, for each new document, you will try to find which document from the learning sample is the nearest neighbor [i.e. has most ‘words’ in common in your Bag Of Words], the new document will be tagged as its nearest neighbor.

How to Benchmark Quality?

you can check for quality by taking 10% documents out of the learning sample, and learn only on the remaining 90%. after done learning, you can evaluate how accurate your algorithm is by checking the accuracy of the remaining 10%. Note that you will probably need to do this a few times to find optimal k [Bag Of Words size] as mentioned above.