Question

I would like to convert flat form data to recursive JSON data in python or javascript. This JSON data can later be interpreted by a template engine (google for tempest, it has django like syntax). There are plenty examples to convert flat data to recursive data, but the problem is it can’t be a dict or list only.

I tried to do it in many ways, but didn’t succeed yet. So after scratching my head for at least two weeks, I decided to ask a question here.

The formdata is like this (key names may be different):

formdata = [
    {"formname": "name", "formvalue": "Roel Kramer"},
    {"formname": "email", "formvalue": "[e-mail]"},
    {"formname": "paragraph-0.title", "formvalue": "test titel 1"},
    {"formname": "paragraph-0.body", "formvalue": "bla bla body 1"},
    {"formname": "paragraph-0.image-0.src", "formvalue": "src 1"},
    {"formname": "paragraph-0.image-1.src", "formvalue": "src 2"},
    {"formname": "paragraph-1.title", "formvalue": "test titel 2"},
    {"formname": "paragraph-1.body", "formvalue": "bla bla body 2"},
    {"formname": "paragraph-1.image-0.src", "formvalue": "src 3"},
    {"formname": "paragraph-1.image-1.src", "formvalue": "src 4"},
    {"formname": "paragraph-1.image-2.src", "formvalue": "src 5"},
    {"formname": "paragraph-2.title", "formvalue": "test titel 3"},
    {"formname": "paragraph-2.body", "formvalue": "bla bla body 3"},
    {"formname": "paragraph-2.image-0.src", "formvalue": "src 6"},
    {"formname": "paragraph-2.image-1.src", "formvalue": "src 7"},
]

I would like to convert it to this format:

{'paragraph':
    [
        {
        'image': [{'src': 'src 1'}, {'src': 'src 2'}],
        'body': 'body 2',
        'title': 'titel 2'
        },
        {
        'image': [{'src': 'src 3'}, {'src': 'src 4'}, {'src': 'src 5'}],
        'body': 'body 2',
        'title': 'titel 2'
        },
        {
        'image': [{'src': 'src 6'}, {'src': 'src 7'},
        'body': 'body 3',
        'title': 'titel 3'
        },
    ],
}

As you can see I mix dicts with lists, which makes it a bit harder. In my last attempt I got to the point where the script figures out where to add lists and where to add dicts. This results in this:

{'paragraph': [{'image': []}, {'image': []}, {'image': []}]}

But when I add data the result is not what I expected.

{
  "paragraph": [{
    "body": "bla bla body 1",
    "image": {
      "src": "src 7"
    },
    "title": "test titel 1"
  }, {
    "body": "bla bla body 2",
    "image": {
      "src": "src 5"
    },
    "title": "test titel 2"
  }, {
    "body": "bla bla body 3",
    "image": {
      "src": "src 3"
    },
    "title": "test titel 3"
  }, {
    "image": {
      "src": "src 6"
    }
  }],
  "name": "Roel Kramer",
  "email": "[e-mail]"
}

The total script can be seen at github gist. I know it can be much cleaner, but I will refactor it when it works.

What am I doing wrong? Am I totally missing something?
Thanks a lot!

Answer 1

Well, if you know the format will be consistent then something like this will work:

def add_data(node, name, value):
    if '-' not in name:
        node[name] = value
    else:
        key = name[:name.index('-')]
        node_index = int(name[len(key) + 1:name.index('.')])
        node.setdefault(key, [])
        if node_index >= len(node[key]):
            node[key].append({})
        add_data(node[key][node_index],
                 name[name.index('.') + 1:],
                 value)

Then to use it, just do something like this:

root_node = {}
for data in formdata:
    add_data(root_node, data['formname'], data['formvalue'])

The function makes the following assumptions:

1. The – character is used to specify which node of a particular node type, and is followed
   by a number.
2. The . character separates nodes in the tree, and always follows the index number.
3. The form data will always go in order. (paragraph-0, paragraph-1, paragraph-2) instead of (paragraph-1, paragraph-0, paragraph-3).

So, here’s the code with comments explaining it:

def add_data(node, name, value):
    # We're at a parent node (ex: paragraph-0), so we need to drill down until
    # we find a leaf node
    if '-' in name:
        key = name[:name.index('-')]
        node_index = int(name[len(key) + 1:name.index('.')])

        # Initialize the parent node if needed by giving it a dict to store it's
        # information nodes
        node.setdefault(key, [])
        if node_index >= len(node[key]):
            node[key].append({})

        # Drill down the tree by calling this function again, making this
        # parent node the root
        add_data(node[key][node_index],
                 name[name.index('.') + 1:],
                 value)

    # We're at a leaf node, so just add it to the parent node's information
    # ex:  The first formdata item would make the root_node dict look like
    # { 'name': 'Roel Kramer' }
    else:
        node[name] = value

Here’s a working example: http://pastebin.com/wpMPXs1r