Question

DO you know a library for Web page scraping for Delphi. Like Beautiful Soup or Scrapy for Python ?

Answer 1

Well, it’s not for Delphi, but for FreePascal, since I do not have a recent Delphi version, but porting between them is supposed to be not so difficult.

Anyways, my Internet Tools are probably the best Pascal web scraping library that are out there.

You can, e.g. print all links on a page with:

uses simpleinternet, xquery;

var a: IXQValue;
begin
  for a in process('http://stackoverflow.com', '//a/@href') do
    writeln(a.toString);
end.

They are platform independent; have full support for XPath 2, XQuery, CSS 3 selectors (those are not so well tested through, XPath is better anyways) and pattern-matching; parse xml and html; and download over http and https.