Question

I have a Django database, in which I have some JSON stored. I wish to extract this JSON, parse it, and then apply filters / queries to it. More specifically I want to have a dynamic webpage which displays the JSON sorted, filtered or grouped by the users preference.

From reading the Django documentation I can only see how to use pre-made filters like starts_with, but not how to pre-process the data. I did see the option for applying a regex, but I don’t think I can implement a full JSON parser by hand. I currently parse the JSON and then search it using python classes, but that seems to defeat the purpose of using a database, particularly because it is limited in the types of query it can handle.

The JSON is coming from some other source, and has no guarantees about it’s fields or structure.

Thanks in advance,

jhoyla

Answer 1

The best answer I have been able to come up with is to add the JSON to the database as tables on the fly to get two strings of results (I do it this way because dictionaries aren’t guaranteed to maintain order.)

TABLECONFIG = 'data_result' 
def jsontodb(conn, jsonarray):
    inits = reduce(lambda x, y: x.union(y.keys()), jsonarray, set())

conn.execute("CREATE TABLE " + TABLECONFIG + " (" + 
str(map(lambda x: x.encode('UTF-8'), inits)).encode('UTF-8').strip('[]')  + ");")
for i in jsonarray:
        d = map(lambda y: str(map(lambda x: x.encode('UTF-8'), y)).strip('[]'), zip(*i.items()))    
        conn.execute("INSERT INTO " + TABLECONFIG + " (" + d[0] + ") VALUES (" + d[1] + ");")
conn.commit()

DBCONFIG = '/work/django.sqlite3'
def pullquery(query):
    p  = loads(oqr.main(
        [str(x) for x in shlex.split(query.encode('UTF-8'))]
    ))
    conn = sqlite3.connect(DBCONFIG)
    jsontodb(conn, p)
    return conn

Giving the result I want, however this only works if the JSON is at most 1 layer deep (although I imagine that this could be fixed it’s pretty hacky already.