Question

I have written a scrapy spider that successfully scrapes data and saves it in a standalone database…one table.

I want to display this data on a website.

I have recently decided to use wordpress as the basis for my site, then realised that it itself uses mysql so why not just add another table to it to hold the scraped data instead?

So my questions are these.

1. Is it advisable to do this or is it better to have a stand alone database.
2. How do I display the rows of data from a query/table within wordpress. Do I use a post or page. What is the php I should use? (I assume it would be different connecting to the wp database as opposed to an external one)
3. In the scrapy pipeline, how do I connect to the wordpress database to store the scraped data?

Thanks

Answer 1

I can’t answer to the wordpress specific topics but I will try the rest:

ad 1)
I would use seperate datebase. This way, you can backup your wordpress site more easily as well as your scraped results. I would only make the exception if one data cannot exist without the other one because there are (for some reasons) dependencies between them. But I assume this is not the case so I would seperate them.

ad 3)
Pretty much the same way as you already do. I assume you are already connecting to a database since you mentioned it. Depending on how you have done it, you only need to alter the connection strings of the database you want to connect to.