Question

Where I am going wrong with this for loop which is meant to take as input a specific corpus, sample size and number of samples and then give the average and standard deviation of expected no. of sentiment tokens?

def test_iterate(corpus_reader, sample_size, number_of_samples):
for i in xrange(number_of_samples):
    tokens = corpus_reader.sample_words_by_sents(sample_size)
    sents = corpus_reader.sample_sents(sample_size)
    print expected_sentiment_tokens(tokens)
    s = []
    s.append(expected_sentiment_tokens(tokens))
    s = array(s)
print "Average expected no of sentiment tokens: %s" % average(s)
print "Standard deviation of sentiment tokens: %s" % std(s)

test_iterate(rcr, 500, 3)

returns

181.166666667
186.277777778
185.5
Average expected no of sentiment tokens: 185.5
Standard deviation of sentiment tokens: 0.0

For some reason the average is being set to the last sample instead of averaging and standard deviating all of the samples together.

Answer 1

As noted above, indentation is significant in Python, so always make sure that the code in your questions looks just like the code on your screen. Otherwise we have to guess, and if we’re wrong, we can run down false trails. Anyway, this:

s = []
s.append(expected_sentiment_tokens(tokens))

will keep making an empty list and appending one value each time inside your loop. Doing just the expected sentiment to start with, you probably want something like

def test_iterate(corpus_reader, sample_size, number_of_samples):
    s = []
    for i in xrange(number_of_samples):
        tokens = corpus_reader.sample_words_by_sents(sample_size)
        exp_sent = expected_sentiment_tokens(tokens)
        print exp_sent
        s.append(exp_sent)
    print "s =", s    
    print "Average expected no of sentiment tokens: %s" % average(s)
    print "Standard deviation of sentiment tokens: %s" % std(s)

test_iterate(rcr, 500, 3)

[and remember, indentation matters]