Question

I’m trying to load fixtures into my Django (1.4) app via converting CSVs with the data into json files, and then feeding those into Django with loaddata. But I keep getting a max recursion depth error. I’ve looked around, and it seems to be connected to non-ASCII characters in the json files. However, I’m pretty sure I’m not doing that? Here’s my code:

    def csv_to_json(self,csv,dest,model,keys,sub):
    #keys is a dict formatted x:f where x = index of value v;
    #don't include pk
    with open(csv) as f:
        l = f.readlines()
    right = len(l[0].split(","))
    out = []
    for x in xrange(1,len(l)):
        if sub:
            line = re.sub(", "," ",l[x])
        line = re.sub(r'(,[\'\"*.]+|[\'\"*.]+,)','',l[x])
        line = unicodedata.normalize('NFKD',unicode(line,'utf-8','ignore')).encode('ASCII','ignore')
        line.encode('ASCII')
        splt = line.split(",")
        print len(splt) 
        if len(splt) == right:
            for y in xrange(len(splt)):
                if re.match(r'\d{4}-\d{2}-\d{2}',str(splt[y])):
                    splt[y] = int(splt[y][:4])
                try:
                    if splt[y].isupper():
                        splt[y] = splt[y].title()
                    splt[y] = splt[y].rstrip()
                except AttributeError:
                    continue
                try:    
                    splt[y] = int(splt[y])
                except ValueError:
                    try:
                        splt[y] = float(splt[y])
                    except ValueError:
                        continue

            d = {}
            d["model"] = "dishes.%s"%(model)
            d["pk"] = splt[0]
            d["fields"] = {}
            for k,v in keys.items():
                print v,k
                try:
                    d["fields"][v] = splt[k]
                except IndexError:
                    continue
            out.append(d)
    with open(dest,"w") as f:
        json_out = simplejson.dump(out,f,separators=(",",":"))

As you can see, I AM briefly turning the contents of the CSVs into UTF-8 for the purposes of using unicodedata to scrub them of diacritics (which I had to do because I was getting “invalid continuation character” messages when I tried to load them with loaddata), but then I’m converting everything back into ASCII. Or am I? If I am, then what’s causing the recursion problem?

ETA: Here’s the error message in full:

Problem installing fixture
'/Users/samuelraker/django/menus/menus/dishes/fixtures/Classification.json':                
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 190, in handle
for obj in objects:
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/core/serializers/json.py", line 47, in Deserializer
raise DeserializationError(e)
DeserializationError: maximum recursion depth exceeded while calling a Python object

Re-edited to further add:
Here’s a snippet of one of the JSON files I’m trying to load.

[
{
    "pk":1,
    "model":"dishes.dish",
    "fields":{
        "name":"Consomme printaniere royal"
    }
},
{
    "pk":2,
    "model":"dishes.dish",
    "fields":{
        "name":"Chicken gumbo"
    }
},
{
    "pk":3,
    "model":"dishes.dish",
    "fields":{
        "name":"Tomato aux croutons"
    }
},
{
    "pk":4,
    "model":"dishes.dish",
    "fields":{
        "name":"Onion au gratin"
    }
},
{
    "pk":5,
    "model":"dishes.dish",
    "fields":{
        "name":"St. Emilion"
    }
},
{
    "pk":7,
    "model":"dishes.dish",
    "fields":{
        "name":"Radishes"
    }
},
{
    "pk":8,
    "model":"dishes.dish",
    "fields":{
        "name":"Chicken soup with rice"
    }
},
{
    "pk":9,
    "model":"dishes.dish",
    "fields":{
        "name":"Clam broth (cup)"
    }
},
{
    "pk":11,
    "model":"dishes.dish",
    "fields":{
        "name":"Clear green turtle"
    }
},
{
    "pk":13,
    "model":"dishes.dish",
    "fields":{
        "name":"Anchovies"
    }
},
{
    "pk":14,
    "model":"dishes.dish",
    "fields":{
        "name":"Fresh lobsters in every style"
    }
},
{
    "pk":15,
    "model":"dishes.dish",
    "fields":{
        "name":"Celery"
    }
},
{
    "pk":16,
    "model":"dishes.dish",
    "fields":{
        "name":"Pim-olas"
    }
},
{
    "pk":17,
    "model":"dishes.dish",
    "fields":{
        "name":"Caviar"
    }
},
{
    "pk":18,
    "model":"dishes.dish",
    "fields":{
        "name":"Sardines"
    }
},
{
    "pk":19,
    "model":"dishes.dish",
    "fields":{
        "name":"India chutney"
    }
},
{
    "pk":20,
    "model":"dishes.dish",
    "fields":{
        "name":"Pickles"
    }
},
{
    "pk":21,
    "model":"dishes.dish",
    "fields":{
        "name":"English walnuts"
    }
},
{
    "pk":22,
    "model":"dishes.dish",
    "fields":{
        "name":"Pate de foies-gras"
    }
},
{
    "pk":23,
    "model":"dishes.dish",
    "fields":{
        "name":"Pomard"
    }
},
{
    "pk":26,
    "model":"dishes.dish",
    "fields":{
        "name":"Clams"
    }
},
{
    "pk":27,
    "model":"dishes.dish",
    "fields":{
        "name":"Oysters"
    }
},
{
    "pk":28,
    "model":"dishes.dish",
    "fields":{
        "name":"Claremont planked shad"
    }
},
...

EDIT 3: Happy Holidays! I just pasted a small snippet of one of the JSON files into a separate file, ran it through JSONLint just to make sure, and then passed it to loaddata…and I got the same darn error message. Could it be a problem with my models?

EDIT 4: So I tried encoding my data as YAML instead of JSON, and it didn’t work. I’m really stumped here guys. Someone PLEASE help!

EDIT 5: I even tried changing my db backend to django-mysql-pymysql (http://pypi.python.org/pypi/django-mysql-pymysql/0.1), but that didn’t work either. Anyone? Please?

Answer 1

Turns out, the problem was that I had a field in my models called ‘pk’ with ‘primary_key=True’. It’d be nice if there was a list of ‘reserved words’ for Django, since my mistake seems fairly easy to make, and, without using the shell for debugging, is tricky to catch. I also had problems with manually-defined fields ending in “_id,” which isn’t documented either, AFAIK.