Question

I am trying to get the pricing information about devices from a website which uses javascripts to load the data. I am getting the data in a JSON dictionary structure. Below is the code I am using:

# -*- coding: cp1252 -*-
import csv
import urllib2
import sys
import time
import re
import json
from bs4 import BeautifulSoup
from itertools import islice
from pprint import pprint

page = urllib2.urlopen('http://www.vodafone.de/privat/tarife/red-smartphone-tarife.html').read()
soup = BeautifulSoup(page)
x = [None]*1000

i = 0
j = 0
k = 0
scripts = soup.find_all('script')
script = next(s.text for s in scripts if s.string and 'window.rates' in s.string)
datastring = script.split('phones=')[1].split(';window.')[0]
datastring = re.sub(ur'([{,])([a-z]\w*):', ur'\1"\2":', datastring)
data = json.loads(datastring)

for d in data:
    for j in data[d]:
        pprint(data[d]['name']) 
        x[i] = data[d]['name']
        i = i + 1

This part gives me the list of devices from the web page.

However, when I use the same method to get the pricing information(below mentioned) which is one more level inside in the JSON data dictionary, I get a TypeError: string indices must be integers –

for d in data:
    for j in data[d]:
        for k in data[d][j]:
            pprint(data[d][j][k]) 
            x[i] = data[d][j][k]
            i = i + 1

Here is the snippet of JSON data structure which I am scraping and from which I need to extract information:

{u'deliveryTime': u'Lieferbar innerhalb 48 Stunden',
 u'image': u'/images/m707491_300465.jpg',
 u'name': u'BlackBerry Bold 9900',
 u'sku1104261': {u'e': u'169.90', u'p': u'prod974431'},
 u'sku1444275': {u'e': u'129.90', u'p': u'prod974431'},
 u'sku1444283': {u'e': u'89.90', u'p': u'prod974431'},
 u'sku1444286': {u'e': u'49.90', u'p': u'prod974431'},
 u'sku1444291': {u'e': u'49.90', u'p': u'prod974431'}}
{u'deliveryTime': u'Vorauss. verfügbar ab Mitte Januar',
 u'image': u'/images/m1327474_300658.jpg',
 u'name': u'HTC One X+ mit limitiertem Beats-Headset',
 u'sku1444277': {u'e': u'249.90', u'p': u'prod1624433'},
 u'sku1444285': {u'e': u'119.90', u'p': u'prod1624433'},
 u'sku1444287': {u'e': u'99.90', u'p': u'prod1624433'},
 u'sku1444292': {u'e': u'99.90', u'p': u'prod1624433'},
 u'sku1474223': {u'e': u'399.90', u'p': u'prod1624433'}}

Please help me in solving this issue.

Answer 1

According to JSON structure you’ve posted, it should be something like this:

price = 0
for key, value in data.items():
    if key.startswith(u'sku') and isinstance(value, dict): 
        price += value.get(u'e', 0)

Actually, few basic concepts here:

• validate that you have needed keys (skipping keys not starting with “sku”)
• validate that sub-keys are dicts
• use .get() method for dicts instead of direct-referencing. this method allows you to specify default value that will be returned if there is no such key in dict (and no exceptions will be rised)