I’m working on creating a python program to interact with many different types of conceptual objects. For example, it might represent a person, in which case it’d have something like this:
type = "person"
name = "Bono"
profession = "performer"
nationality = "Irish"
However, it might also represent a magazine, in which case it’d look something like this
type = "publication"
name = "Rolling Stone"
editor = ("Jann Wenner" , "Will Dana")
founding_year = "1967"
Aside from type and name, all of the other fields are optional. Here’s the tricky bit — it’s part of code written for a scraper, so all of the other fields are determined/created on the fly. In other words, we won’t know that we need an “editor” field until the scraper spits back “editor” to the code
Ideally, this would be implemented fairly straightforwardly as a python dictionary of lists. However, we’ll be working with a large number of records — too many to keep in memory at the same time. As a result, I’d like to have database compatibility — something like Django’s MVC, so we can easily query the record set.
One option I had considered was Django fieldsets, but it looks like they’re still in beta and I worry that I’ll lose some generality in what I can store — ideally, I’d be able to store any type of data with a key, (value_list) pair. I’d love any input on the feasibility of fieldsets or example code.
Another option I had considered was a combination of the Django MVC and JSON. In this case, I’d have three columns for each object — type, name, and attributes. Attributes would be a JSON serialization (or other appropriate pickling method) of all of the other attributes, so that once you had the object, you could reconstitute it’s attributes and query the set. I’d store something like this or this (links). With this method, I’d lose the ability to easily do a search for any of the attributes in the dict.
I’d very much appreciate any input or guidance. If anyone knows of similar projects, I’d love to know.
This seems like an excellent opportunity to use NoSQL databases. Something like MongoDB doesn’t rely on a fixed schema, so it might be suitable for your scenario.