Home/ Questions/Q 4125068
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-20T23:49:18+00:00

I have some random HTML layouts that contain important text I would like to

  • 0

I have some random HTML layouts that contain important text I would like to extract. I cannot just strip_tags() as that will leave a bunch of extra junk from the sidebar/footer/header/etc.

I found a method built in Python and I was wondering if there is anything like this in PHP.

The concept is rather simple: use
information about the density of text
vs. HTML code to work out if a line of
text is worth outputting. (This isn’t
a novel idea, but it works!) The basic
process works as follows:

  1. Parse the HTML code and keep track of the number of bytes processed.
  2. Store the text output on a per-line, or per-paragraph basis.
  3. Associate with each text line the number of bytes of HTML required to
    describe it.
  4. Compute the text density of each line by calculating the ratio of text
    t> o bytes.
  5. Then decide if the line is part of the content by using a neural network.

You can get pretty good results just
by checking if the line’s density is
above a fixed threshold (or the
average), but the system makes fewer
mistakes if you use machine learning –
not to mention that it’s easier to
implement!

Update: I started a bounty for an answer that could pull main content from a random HTML template. Since I can’t share the documents I will be using – just pick any random blog sites and try to extract the body text from the layout. Remember that the header, sidebar(s), and footer may contain text also. See the link above for ideas.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team
    • phpQuery is a server-side, chainable, CSS3 selector driven Document Object Model (DOM) API based on jQuery JavaScript Library.

    UPDATE 2

    1. many blogs make use of CMS;
    2. blogs html structure is the same almost the time.
    3. avoid common selectors like #sidebar, #header, #footer, #comments, etc..
    4. avoid any widget by tag name script, iframe
    5. clear well know content like:
      1. /\d+\scomment(?:[s])/im
      2. /(read the rest|read more).*/im
      3. /(?:.*(?:by|post|submitt?)(?:ed)?.*\s(at|am|pm))/im
      4. /[^a-z0-9]+/im

    search for well know classes and ids:

    • typepad.com .entry-content
    • wordpress.org .post-entry .entry .post
    • movabletype.com .post
    • blogger.com .post-body .entry-content
    • drupal.com .content
    • tumblr.com .post
    • squarespace.com .journal-entry-text
    • expressionengine.com .entry

    • gawker.com .post-body

    • Ref: The blog platforms of choice among the top 100 blogs

    $selectors = array('.post-body','.post','.journal-entry-text','.entry-content','.content');
$doc = phpQuery::newDocumentFile('http://blog.com')->find($selectors)->children('p,div');

    search based on common html structure that look like this:

    <div>
<h1|h2|h3|h4|a />
<p|div />
</div>

    $doc = phpQuery::newDocumentFile('http://blog.com')->find('h1,h2,h3,h4')->parent()->children('p,div');
    • 0