Question

To keep things organized I determined there are three item classes that a spider will populate.

Each item class has a variety of fields that are populated.

class item_01(Item):
    item1 = Field()
    item2 = Field()
    item3 = Field()

class item_02(Item):
    item4 = Field()
    item5 = Field()

class item_03(Item):
    item6 = Field()
    item7 = Field()
    item8 = Field()

There are multiple pages to crawl with the same items.
In the spider I use XPathItemLoader to populate the ‘containers’.

The goal is to pass the items to a mysql pipeline to populate a single table. But here is the problem.

When I yield the three containers (per page) they are passed as such into the pipeline, as three separate containers.
They go through the pipeline as their own BaseItem and populate only their section of the mysql table, leaving the other columns ‘NULL’.

What I would like to do is repackage these three containers into a single BaseItem so that they are passed into the pipeline as a single ITEM.

Does anyone have any suggestions as to repackage the items? Either in the spider or pipeline?

Thanks

Answer 1

I did this hack to get things moving but if someone can improve or hint at a better solution please share it.

Loading my items in the spider like this:

items = [item1.load_item(), item2.load_item(), item3.load_item()]  

I then defined a function outside the spider:

def rePackIt(items):
    rePackage = rePackageItems()
    rePack = {}
    for item in items:
        rePack.update(dict(item))

    for key, value in rePack.items():
        rePackage.fields[key] = value
    return rePackage  

Where in the items.py I added:

class rePackageItems(Item):
    """Repackage the items"""
    pass  

After the spider is done crawling the page and loading items I yield:

yield rePackIt(items)  

which takes me to the pipelines.py.

In the process_item to unpack the item I did the following:

def process_item(self, item, spider):
        items = item.fields

items is now a dictionary that contains all the extracted fields from the spider which I then used to insert into a single database table