Home/ Questions/Q 7761009
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-01T14:02:34+00:00

Given 2 html sources, I want to first extract the main content out of

  • 0

Given 2 html sources, I want to first extract the main content out of it using something like this. Are there any other better libraries – I am specifically looking for Python/Javascript ones?

Once I have the two extracted contents, I want to return a score between 0 and 1 denoting how similar they are e.g. news articles on the same topic from CNN and BBC would have higher similarity scores since they are on the same topic or webpages pertaining to the same product on Amazon.com and Walmart.com would have a high score too. How can I do this? Are there existing libraries that do this already? What are some good libraries I can use? Basically I am looking for a combination of automatic summarization, keyword extraction, named-entity recognition and sentiment-analysis.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    There are many things embedded in your question. I will try to provide you with a library or else will suggest you Algorithms that can solve your tasks (which you can Google and you will get many python implementations)

    Point 1. To extract main content out of html (http://nltk.googlecode.com/svn/trunk/doc/book/ch03.html) & for other NLP related stuff you can check out NLTK. Its written in Python. You can also check out for a library called BeautifulSoup, its awesome (http://www.crummy.com/software/BeautifulSoup/)

    Point 2. When you say:

    Once I have the two extracted contents, I want to return a score between 0 and 1 denoting how similar they are….

    For this I suggest you can cluster your document set using any unsupervised learning clustering technique. Since your problem falls under the distance-metric based clustering so it should be really easy for you to cluster similar documents and then assign a score to them based on their similarity with the cluster centroid. Try either K-Means or Adaptive Resonance Theory. In the latter you dont need to define the number of clusters in advance. OR as larsman points out in his comments you can simply use TF-IDF (http://www.miislita.com/term-vector/term-vector-3.html)

    Point 3.When you say:

    Basically I am looking for a combination of automatic summarization, keyword extraction, named-entity recognition and sentiment-analysis

    For Automatic Summarization use Non Negative Matrix Factorization

    For Keyword extraction use NLTK

    For Named-Entity Recognition use NLTK

    For Sentiment Analysis use NLTK

    • 0