Question

I’m working on a project that requires input from data that’s displayed in a live flash graph (a data logging chart at http://137.205.144.34/flash/index.html#menuIndex=1&accordionIndex=2&menuId=mimic1&menuStruct=S1R2M3C1H1. As the html couldn’t be accessed directly, I used Firebug to monitor my activity, and found the data I wanted stored at http://137.205.144.34/services/unload.cmd?format=csvx&sched=&start=-240:00:00&id=75631&step=864. However, when I try and access this url, it automatically saves a file (containing the data) to my pc, so I can’t access the html source code. Using the url, I have used beautifulsoup to import the data, but i can’t search or manipulate it using html tags as they are unknown. The only data i actually want is the latest hourly reading – one of ~ 1300 lines. And of that line, i only need the last value. Is there a way I could find the html tags? If not, what would be the best way to extract the bit of data I need?

Any help would be greatly appreciated,
Thanks.

Answer 1

The file you are downloading has no HTML in it. It is a comma-separated fle and you should use the csv module to parse it.

This code will print the first item in each row (the item that contains the date and time):

import csv

with open('unload.cmd', 'r') as csvfile:
    reader = csv.reader(csvfile)
    for row in reader:
        print row[0]

This works assuming that you are using the file downloaded with the default name.

In order to first download the file programmatically, import it into a string, and then use it as a source file for the csv.reader():

import urllib
import csv
import StringIO

url = 'http://137.205.144.34/services/unload.cmd?format=csvx&sched=&start=-240:00:00&id=75631&step=864'
f = urllib.urlopen(url)
data = f.read()

reader = csv.reader(StringIO.StringIO(data))
for row in reader:
    if row: print row[0]