Question

i’m trying to generate unigram from a text file. But only the bigram for the first line of the given file is shown. I want to show unigram for all the sentences in the file.

import string;
import sys;
import tokenize;

f = open("data.txt", 'r');
line=f.readline();
while line:
    line = line.rstrip();
    list = line.split();
    for word in list:
         print word
    line = f.readline();

Why it is not showing unigram for the sentences and also how can i turn this into a bigram?

Thanks in advance.

data.txt is the text file which contains the sentences.
It has two sentences –

        Hello world this is a test code
        today is 29th november 2011

im getting the output:

    Hello
    world
    this
    is
    a
    test

code

Answer 1

There are some obvious problems with that code snippet.

1. ; are not required
2. None of the imported modules (i.e. tokenize) are used. This is valid, but pointless.
3. The loop over the file lines uses while, which works but is odd.

You do not show the structure of the text file, but I’m assuming each sentence is on a separate line (i.e. a text file with two sentences will contain two lines).

I’m unsure exactly what a bigram is in this case, so you may need to replace the bigram function.

from itertools import tee, izip

def bigrams(iterable):
    a, b = tee(iterable)
    next(b, None)
    return izip(a, b)

with open("data.txt", 'r') as f:
    for line in f:
        words = line.strip().split()
        uni = words
        bi = bigrams(words)
        print uni
        print list(bi)