Question

I’m using Scrapy and xpaths to grab a bunch of sold property data from a website. There are 9 different ‘items’ in total (sale price, date sold, agent, agency, address, type of property, bedrooms, bathrooms, and full URL) and 20 records per page. I then deposit the results into an SQLite3 database.

Everything works perfectly well until I hit a page with slightly incomplete data. If a variable is missing from even a single record, it screws everything up and nothing from the page gets written to the database.

I’m pretty sure this is because I have coded things in an ‘unpythonic’ way but can’t figure out a solution (pythonic or otherwise) to this problem.

This is the part of my pipelines.py file where things seem to be going wrong:

def process_item(self, item, spider):

    self.cur.execute("CREATE TABLE IF NOT EXISTS Diditwork(Id INTEGER PRIMARY KEY, SalePrice TEXT, Address TEXT, Agent TEXT, Agency TEXT, DateSold TEXT, TypeOfProperty TEXT, Bedrooms TEXT, Bathrooms TEXT, FullURL TEXT)")
    n=int(0)
    for data in item['address']:

        self.cur.execute("INSERT INTO Diditwork (Saleprice, Agent, Agency, Address, DateSold, TypeOfProperty, Bedrooms, Bathrooms, FullURL) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", ((item['saleprice'][n]), (item['agent'][n]), (item['agency'][n]), (item['address'][n]), (item['datesold'][n]), (item['typeofproperty'][n]), (item['bedrooms'][n]), (item['bathrooms'][n]), (item['fullurl'][n])))

        self.conn.commit()
        n +=1

    return item

The error I get when a variable is missing from a record is ‘exceptions.IndexError: list index out of range’

I think this is because of the way my loop is written. It will look for lists with 20 variables in them, but on some pages if one or more records are incomplete or missing it tries to call a list index that doesn’t exist.

For example, on one page there may be 20 adress records but only 15 agent records. It tries to call item[‘agent’][16] and then spits back an error because the list isn’t that long.

Anyway, my apologies for the poor explanation of my problem. I’m not sure if I should be trying to implement some kind of error handling like

if len(item['address']) != len(item['agent']): 
            #item['agent'] = ["not available"] * 20 

Or whether my whole approach is wrong.

Any assistance would be most appreciated – I’m a bit out of my depth here and have been trying for the better part of a day and a half to resolve this problem on my own.

EDIT: Thanks guys. I still have more testing to do, but I think I actually ended up figuring this one out. Here’s the code I’m using. I’ll leave this one as unanswered for 24 hours in case someone comes up with a more elegant solution.

if len(item['address']) != len(item['agent']): #error checking
            difference = len(item['address']) - len(item['agent']) #find the disparity
            item['agent'].extend(["not available"] * difference) #append/extend the list by an appropriate number 

Answer 1

Your approach is compact, requires less code and less processing so it is fine. All you are lacking is proper validation. Any data entered to the db must be validated , its not only the length but also field types that you should validate and fix before parsing and storing.