Home/ Questions/Q 8088175
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-05T19:00:40+00:00

I am building a web application as college project (using Python), where I need

  • 0

I am building a web application as college project (using Python), where I need to read content from websites. It could be any website on internet.

At first I thought of using Screen Scrapers like BeautifulSoup, lxml to read content(data written by authors) but I am unable to search content based upon one logic as each website is developed on different standards.

Thus I thought of using RSS/ Atom (using Universal Feed Parser) but I could only get content summary! But I want all the content, not just summary.

So, is there a way to have one logic by which we can read a website’s content using lib’s like BeautifulSoup, lxml etc?

Or I should use API’s provided by the websites.

My job becomes easy if its a blogger’s blog as I can use Google Data API but the trouble is, should I need to write code for every different API for the same job?

What is the best solution?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Using the website’s public API, when it exists, is by far the best solution. That is quite why the API exists, it is the way that the website administrators say “use our content”. Scraping may work one day and break the next, and it does not imply the website administrator’s consent to have their content reused.

    • 0