Question

We have a code which extract the service list from our tomcat server and insert it into a database. Basically, the code will receive a huge output ( 100+ results ) from a html result with the following structure:

<TABLE bgcolor=#dddddd border=1>
<TR>
<TD valign="top"><B>name</B></TD>
<TD>Loader</TD>
</TR>
<TR>
<TD valign="top"><B>enabled</B></TD>
<TD>true</TD>
</TR>
<TR>
<TD valign="top"><B>loadok</B></TD>
<TD>13</TD>
</TR>
</TABLE>
<TABLE bgcolor=#dddddd border=1>
<TR>
<TD valign="top"><B>name</B></TD>
<TD>tester</TD>
</TR>
<TR>
<TD valign="top"><B>enabled</B></TD>
<TD>false</TD>
</TR>
<TR>
<TD valign="top"><B>loadok</B></TD>
<TD>13</TD>
</TR>
</TABLE>

Following, the actual code ( which works )

#!/usr/bin/env python

import psycopg2
import urllib2
import base64
import sys
import re
import lxml.html as LH

con = None

try:

    con = psycopg2.connect(database='xx', user='xx',password='xx',host='vxx')
    cur = con.cursor()
    qry ="select iservers.env,iservers.family,iservers.prefix,iservers.iserver,iservers.login,iservers.password,services.service,"   +\
    "proxy_access.proxy_server,proxy_access.proxy_user,proxy_access.proxy_pass "        +\
    "from services,iservers,proxy_access where iservers.env='TEST' and services.id='2' "
    cur.execute(qry)
    data = cur.fetchall()

    for result in data:
    
        env      = result[0]
        family   = result[1]
        prefix   = result[2]
        iserver  = result[3]
        login    = result[4]
        password = result[5]
        service  = result[6]
        proxyHost = result[7]
        proxyUser = result[8]
        proxyPass = result[9]

        proxy_auth = "http://"+proxyUser+":"+proxyPass+"@"+proxyHost
        proxy_handler = urllib2.ProxyHandler({"http": proxy_auth})

        opener = urllib2.build_opener(proxy_handler)
        urllib2.install_opener(opener)
        request = urllib2.Request("http://"+iserver+service)
        base64string = base64.encodestring('%s:%s' % (login, password)).replace('\n', '')
        request.add_header("Authorization", "Basic %s" % base64string)
        response = urllib2.urlopen(request)
        html = response.read()
        
        ###################### CHANGE THIS TO USE A HTML PARSER
        regex = r"name</B></TD>\s<TD>(.*?)</TD>\s</TR>\s<TR>\s(.*)enabled</B></TD>\s<TD>(.*)</TD>"
        for m in re.finditer(regex,html):
            print "inserting\t"+iserver+"\t"+m.group(1)
            cur.execute("INSERT INTO pereirtc.package_status (env,family,iserver,prefix,package,status) values (%s,%s,%s,%s,%s,%s)",(env,family,iserver,prefix,m.group(1),m.group(3)))
            con.commit()
        ###################### END
except psycopg2.DatabaseError, e:
    print 'Error %s' % e
    sys.exit(1)

finally:

    if con:
        con.close()
        

After some help from Stack Overflow, I was advised to change the block on "change this …." to libxml. So I got the following block to use:

  doc = LH.fromstring(html)
  tds = (td.text_content() for td in doc.xpath("//td"))
  for td, val in zip(*[tds]*2):
      if td in ("name","enabled"):
          print (td,val)
      

With the example above, I have the result:

('name', 'Loader')
('enabled', 'true')

And what I want is, to insert the result using xpath to insert it into a database. Since I’m starting on python, I’m blocked on how I can do that with xpath/libxml.

Answer 1

I’m not sure if you meant the following simple operation, but splitting the result into two variables is achieved as follows:

doc = LH.fromstring(html)
tds = (td.text_content() for td in doc.xpath("//td"))
for td, val in zip(*[tds]*2):
    if td == "name":
        name = val
    elif td == "enabled":
        enabled = val

print name
print enabled