Question

I have some html files that I want to convert to text. I have played around with BeautifulSoup and made some progress on understanding how to use the instructions and can submit html and get back text.

However, my files have a lot of text that is formatted using table structures. For example I might have a paragraph of text that resides in a td tag within set of table tags

<table>
<td> here is some really useful information and there might be other markup tags but
     this information is really textual in my eyes-I want to preserve it
 </td>
</table>

And then there are the ‘classic tables’ that have data within the body of the table.

I want to be able to apply an algorithm to the table and set some rules that determine whether the table is ripped out before I convert the document to text.

I have figured out how to get the characteristics of my tables- for example to get the number of cols in each table:

numbCols=[]
for table in soup.findAll('table'):
    rows=[]
    for row in table.findAll('tr'):
        columns=0
        for column in row.findAll('td'):
            columns+=1
        rows.append(columns)
    numbCols.append(rows)

so I can operate on numbCols and use the len of each item in the list and the values in each item in the list to analyze the characteristics of my tables and identify the ones I want to keep or discard.

I am not seeing an elegant way to the use this information with BeautifulSoup to get the text. I guess what I am trying to get at is suppose I analyze numbCols and decide that of the ten tables in a particular document I want to exclude tables 2, 4, 6, & 9. So the part of the html document includes everything but those tables. How can I segment my soup that way?

The solution I have come up with is first identify the position of each of the open and close table tags using finditer and getting the spans and then zipping the spans with the numbCols. I can then use this list to snip and join the pieces of my string together. Once this is finished I can then use BeautifulSoup to convert the html to text.

I feel sure that I should be able to do all of this in BeautifulSoup. Any suggestions or links to existing examples would be great. I should mention that my source files can be large and I have thousands to handle.

Didn’t have the answer but I am getting closer

Answer 1

Man I love this stuff
Assuming in a naive case that I want to delete all of the tables that have any rows with a column length greater than 3 My answer is

for table in soup.findAll('table'):
    rows=[]
    for row in table.findAll('tr'):
        columns=0
        for column in row.findAll('td'):
            columns+=1
            rows.append(columns)
        if max(rows)>3:
          table.delete()

You can do any processing you want at any level in that loop, it is only necessary to identify the test and get the right instance to test.