Question

This is a problem I’ve been racking my brains on for a long time, so any help would be great. I have a file which contains several lines in the following format (word, time that the word occurred in, and frequency of documents containing the given word within the given instance in time). Below is an example of what the inputfile looks like.

#inputfile
<word, time, frequency>
apple, 1, 3
banana, 1, 2
apple, 2, 1
banana, 2, 4
orange, 3, 1

I have Python class below that I used to create 2-D dictionaries to store the above file using as the key, and frequency as the value:

class Ddict(dict):
    '''
    2D dictionary class
    '''
    def __init__(self, default=None):
            self.default = default

    def __getitem__(self, key):
            if not self.has_key(key):
                self[key] = self.default()
            return dict.__getitem__(self, key)


wordtime=Ddict(dict) # Store each inputfile entry with a <word,time> key
timeword=Ddict(dict) # Store each inputfile entry with a <time,word> key

# Loop over every line of the inputfile
for line in open('inputfile'):
    word,time,count=line.split(',')

    # If <word,time> already a key, increment count
    try:
        wordtime[word][time]+=count
    # Otherwise, create the key
    except KeyError:
        wordtime[word][time]=count

    # If <time,word> already a key, increment count     
    try:
        timeword[time][word]+=count
    # Otherwise, create the key
    except KeyError:
        timeword[time][word]=count

The question that I have pertains to calculating certain things while iterating over the entries in this 2D dictionary. For each word ‘w’ at each time ‘t’, calculate:

1. The number of documents with
   word ‘w’ within time ‘t’. (a)
2. The number of documents without
   word ‘w’ within time ‘t’. (b)
3. The number of documents with
   word ‘w’ outside time ‘t’. (c)
4. The number of documents without
   word ‘w’ outside time ‘t’. (d)

Each of the items above represents one of the cells of a chi-square contingency table for each word and time. Can all of these be calculated within a single loop or do they need to be done one at a time?

Ideally, I would like the output to be what’s below, where a,b,c,d are all the items calculated above:

print "%s, %s, %s, %s" %(a,b,c,d)

In the case of the input file above, the result of trying to find the contingency table for the word ‘apple’ at time ‘1’ would be (3,2,1,6). I’ll explain how each cell is calculated:

• ‘3’ documents contain ‘apple’ within
  time ‘1’.
• There are ‘2’ documents within time
  ‘1’ that don’t contain ‘apple’.
• There is ‘1’ document containing
  ‘apple’ outside time ‘1’.
• There are 6 documents outside time
  ‘1’ that don’t contain the word
  ‘apple’ (1+4+1).

Answer 1

Your 4 numbers for apple/1 add up to 12, more than the total number of observations (11)! There are only 5 documents outside time ‘1’ that don’t contain the word ‘apple’.

You need to partition the observations into 4 disjoint subsets:
a: apple and 1 => 3
b: not-apple and 1 => 2
c: apple and not-1 => 1
d: not-apple and not-1 => 5

Here is some code that shows one way of doing it:

from collections import defaultdict

class Crosstab(object):

    def __init__(self):
        self.count = defaultdict(lambda: defaultdict(int))
        self.row_tot = defaultdict(int)
        self.col_tot = defaultdict(int)
        self.grand_tot = 0

    def add(self, r, c, n):
        self.count[r][c] += n
        self.row_tot[r] += n
        self.col_tot[c] += n
        self.grand_tot += n

def load_data(line_iterator, conv_funcs):
    ct = Crosstab()
    for line in line_iterator:
        r, c, n = [func(s) for func, s in zip(conv_funcs, line.split(','))]
        ct.add(r, c, n)
    return ct

def display_all_2x2_tables(crosstab):
    for rx in crosstab.row_tot:
        for cx in crosstab.col_tot:
            a = crosstab.count[rx][cx]
            b = crosstab.col_tot[cx] - a
            c = crosstab.row_tot[rx] - a
            d = crosstab.grand_tot - a - b - c
            assert all(x >= 0 for x in (a, b, c, d))
            print ",".join(str(x) for x in (rx, cx, a, b, c, d))

if __name__ == "__main__":

    # inputfile
    # <word, time, frequency>
    lines = """\
    apple, 1, 3
    banana, 1, 2
    apple, 2, 1
    banana, 2, 4
    orange, 3, 1""".splitlines()

    ct = load_data(lines, (str.strip, int, int))
    display_all_2x2_tables(ct)

and here is the output:

orange,1,0,5,1,5
orange,2,0,5,1,5
orange,3,1,0,0,10
apple,1,3,2,1,5
apple,2,1,4,3,3
apple,3,0,1,4,6
banana,1,2,3,4,2
banana,2,4,1,2,4
banana,3,0,1,6,4