Home/ Questions/Q 4029292
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-20T11:19:38+00:00

Context The case is screen scraping web content using QuotaXML SDK 1.6 to finally

  • 0

Context
The case is screen scraping web content using QuotaXML SDK 1.6 to finally display the data on the dashboard and on the iPhone.
This QuotaXML tool offers regex for extracting table data only.
QuotaXML does parse html tables using a three step approach.
1. First it identifies the table, for example using “(?si)<table.*?>(.*?)</table>”
2. Second within this parsed table it identifies rows, like “(?si)<tr.*?>(.*?)</tr>”
3. Third within this row scope, individual cells are identified like “(?si)<tr.*?>(.*?)</tr>“

The problem
The source html contains some rows that are not relevant data like lines or images that span full table width using a colspan.
Or tables contain data cells which are not relevant to the data lines needed, like call detail records which also contain calls to freephones which are not substracted from the minutes in your plan, in this case 0800 and 00800 numbers.
In other words (.*?) may not match ‘ colspan=”‘ neither ‘>0800’ neither ‘>00800’.

In code:

exclude:<tr><td colspan="2"></td></tr>
include:<tr><td><strong>Date</strong></td><td><strong>Time</strong></td></tr>
exclude:<tr><td>05-01-2011</td><td>08004913</td></tr>
include:<tr><td>05-01-2011</td><td>0123456789</td></tr>

Homework done
Even trying my first (start simple) tries to only exclude colspan are all failing:

  1. (?si)<tr.*?>(?!colspan)(.*?)</tr>
  2. (?si)<tr.*?>(.*?)(?!colspan)</tr>
  3. (?si)<tr.*?>.*?[^colspan].*?</tr>
  4. (?si)<tr(\s[^>]*)?>.*?(?!colspan).*?</tr>
  5. (?si)<tr(\s[^>]*)?>.*?(!colspan).*?</tr>
  6. (?si)<tr(\s[^>]*)?>(.*?)(?!colspan)</tr>
  7. (?si)<tr.*?>^(?!.*?colspan=").*?</tr>
    How to negate specific word in regex? seems related though these suggestions don’t result in a match at all.
  8. (?si)<tr.*?>(.(?<!colspan))*?</tr>
  9. (?si)<tr.*?>(?!.*colspan).*</tr>
    Neither do give do positive and negative lookarounds using http://www.regular-expressions.info/lookaround.html the clue.

How should I correctly write this regex?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    The first problem you’re having is that your original expressions are very fragile, because of the “.*?>” intended to match everything up to the earliest “>” — but which will actually match to the following “>” if the rest of the expression fails and backtracks.

    Use a construct like “[^>]*>” instead.

    The second problem is that you’re misinterpreting the meaning of the negative lookahead: it’s not checking that the given pattern does not occur ahead of its position — it’s looking ahead of its position to check that the pattern does not occur AT THAT POSITION.

    With these two changes, your first attempt was very close to solving your test cases:

    $ pcretest
PCRE version 7.8 2008-09-05

  re> /(?si)<tr[^>]*>(?!.*(colspan|>0?0800))(.*?)<\/tr>/
data> <tr><td colspan="2"></td></tr>
No match
data> <tr><td><strong>Date</strong></td><td><strong>Time</strong></td></tr>
 0: <tr><td><strong>Date</strong></td><td><strong>Time</strong></td></tr>
 1: <unset>
 2: <td><strong>Date</strong></td><td><strong>Time</strong></td>
data> <tr><td>05-01-2011</td><td>08004913</td></tr>
No match
data> <tr><td>05-01-2011</td><td>0123456789</td></tr>
 0: <tr><td>05-01-2011</td><td>0123456789</td></tr>
 1: <unset>
 2: <td>05-01-2011</td><td>0123456789</td>

    Note this will still fail to solve the whole problem because the presence of a “colspan” or 800 number later in the string will block the match. You need further test cases, such as:

    $ pcretest
PCRE version 7.8 2008-09-05

  re> /(?si)<tr[^>]*>(?!.*(colspan|>0?0800))(.*?)<\/tr>/
data> <tr><td>05-01-2011</td><td>0123456789</td></tr><tr><td colspan="2"></td></tr>
No match

    So you need to ensure that the negative lookahead never crosses to the next :

    $ pcretest
PCRE version 7.8 2008-09-05

  re> /(?si)<tr[^>]*>(?!((?!<\/tr).)*(colspan|>0?0800))(.*?)<\/tr>/
data> <tr><td colspan="2"></td></tr>
No match
data> <tr><td><strong>Date</strong></td><td><strong>Time</strong></td></tr>
 0: <tr><td><strong>Date</strong></td><td><strong>Time</strong></td></tr>
 1: <unset>
 2: <unset>
 3: <td><strong>Date</strong></td><td><strong>Time</strong></td>
data> <tr><td>05-01-2011</td><td>08004913</td></tr>
No match
data> <tr><td>05-01-2011</td><td>0123456789</td></tr>
 0: <tr><td>05-01-2011</td><td>0123456789</td></tr>
 1: <unset>
 2: <unset>
 3: <td>05-01-2011</td><td>0123456789</td>
data> <tr><td>05-01-2011</td><td>0123456789</td></tr><tr><td colspan="2"></td></tr>
 0: <tr><td>05-01-2011</td><td>0123456789</td></tr>
 1: <unset>
 2: <unset>
 3: <td>05-01-2011</td><td>0123456789</td>

    At which point one may wonder whether RegExps are the right tool for this particular problem 🙂

    • 0