Home/ Questions/Q 8653943
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-12T14:39:50+00:00

I am trying to build a recommender system which would recommend webpages to the

  • 0

I am trying to build a recommender system which would recommend webpages to the user based on his actions(google search, clicks, he can also explicitly rate webpages). To get an idea the way google news does it, it displays news articles from the web on a particular topic. In technical terms that is clustering, but my aim is similar. It will be content based recommendation based on user’s action.

So my questions are:

  1. How can I possibly trawl the internet to find related web-pages?
  2. And what algorithm should I use to extract data from web-page is textual analysis and word frequency the only way to do it?
  3. Lastly what platform is best suited for this problem. I have heard of Apache mahout and it comes with some re-usable algos, does it sound like a good fit?
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    as Thomas Jungblut said, one could write several books on your questions 😉
    I will try to give you a list of brief pointers – but be aware there will be no ready-to-use off-the-shelf solution …

    1. Crawling the internet: There are plenty of toolkits for doing this, like Scrapy for Python , crawler4j and Heritrix for Java, or WWW::Robot for Perl. For extracting the actual content from web pages, have a look at boilerpipe.

      http://scrapy.org/

      http://crawler.archive.org/

      http://code.google.com/p/crawler4j/

      https://metacpan.org/module/WWW::Robot

      http://code.google.com/p/boilerpipe/

    2. First of all, often you can use collaborative filtering instead of content-based approaches. But if you want to have good coverage, especially in the long tail, there will be no way around analyzing the text. One thing to look at is topic modelling, e.g. LDA. Several LDA approaches are implemented in Mallet, Apache Mahout, and Vowpal Wabbit.
      For indexing, search, and text processing, have a look at Lucene. It is an awesome, mature piece of software.

      http://mallet.cs.umass.edu/

      http://mahout.apache.org/

      http://hunch.net/~vw/

      http://lucene.apache.org/

    3. Besides Apache Mahout which also contains things like LDA (see above), clustering, and text processing, there are also other toolkits available if you want to focus on collaborative filtering: LensKit, which is also implemented in Java, and MyMediaLite (disclaimer: I am the main author), which is implemented in C#, but also has a Java port.

      http://lenskit.grouplens.org/

      http://ismll.de/mymedialite

      https://github.com/jcnewell/MyMediaLiteJava

    • 0