Question

I am using lxml to convert html into txt. I almost get to where I wanted with parsing, converting and some parts of the cleanup (tabs, spaces, empty lines) functions ready and a program up and running.

However, after I tried my code with about a hundred htmls (all from different sites), I noticed some exceptions, i.e. lines like:

#wrapper #PrimaryNav {margin:0;*overflow:hidden;}
a.scbbtnred{background-position:right -44px;}
a.scbbtnblack{background-position:right -176px;}
.ghsearch{width:58px;height:21px;line-height:21px;background-position:0 -80px;}
a.scbbtnred span span{background-color:#f00;background-position:0 -22px;}

I assume these are CSS? or other web programming things. But I am totally unfamiliar with these.

Questions: What are these lines? And any suggestions for how to get ride of these lines?

---

Edit: Here is how I did the parts before this question for reference for anyone who drops into this post in the future (new to python, a lot of things here can be improved, but it works ok for me):

# Function for html2txt using lxml
# Author:
# http://groups.google.com/group/cn.bbs.comp.lang.python/browse_thread/thread/781a357e2ce66ce8
def html2text(html):
    tree = lxml.etree.fromstring(html, lxml.etree.HTMLParser()) if isinstance(html, basestring) else html 
    for skiptag in ('//script', '//iframe', '//style'):    
        for node in tree.xpath(skiptag):
            node.getparent().remove(node)
    # return lxml.etree.tounicode(tree, method='text')
    return lxml.etree.tostring(tree, encoding=unicode, method='text')



#Function for cleanup the text:
# 1: clearnup: 1)tabs, 2)spaces, 3)empty lines;
# 2: remove short lines
def textcleanup(text):
    # temp list for process
    text_list = []
    for s in text.splitlines():
        # Strip out meaningless spaces and tabs
        s = s.strip()
        # Set length limit
        if s.__len__() > 35:
            text_list.append(s)
    cleaned = os.linesep.join(text_list)
    # Get rid of empty lines
    cleaned = os.linesep.join([s for s in cleaned.splitlines() if s])
    return cleaned

Answer 1

That is indeed CSS. You’re getting a document like this:

<style>
#wrapper #PrimaryNav {margin:0;*overflow:hidden;}
a.scbbtnred{background-position:right -44px;}
a.scbbtnblack{background-position:right -176px;}
.ghsearch{width:58px;height:21px;line-height:21px;background-position:0 -80px;}
a.scbbtnred span span{background-color:#f00;background-position:0 -22px;}
</style>
<div>
    <p>This bit is HTML</p>
</div>

You need to remove all style tags before parsing out the text.