Question

I am trying to process various texts by regex and NLTK of python -which is at http://www.nltk.org/book-. I am trying to create a random text generator and I am having a slight problem. Firstly, here is my code flow:

1. Enter a sentence as input -this is called trigger string, is assigned to a variable-

2. Get longest word in trigger string

3. Search all Project Gutenberg database for sentences that contain this word -regardless of uppercase lowercase-

4. Return the longest sentence that has the word I spoke about in step 3

5. Append the sentence in Step 1 and Step4 together

6. Assign the sentence in Step 4 as the new ‘trigger’ sentence and repeat the process. Note that I have to get the longest word in second sentence and continue like that and so on-

So far, I have been able to do this only once. When I try to keep this to continue, the program only keeps printing the first sentence my search yields. It should actually look for the longest word in this new sentence and keep applying my code flow described above.

Below is my code along with a sample input/output :

Sample input

“Thane of code”

Sample output

“Thane of code Norway himselfe , with terrible numbers , Assisted by that most disloyall Traytor , The Thane of Cawdor , began a dismall Conflict , Till that Bellona ‘ s Bridegroome , lapt in proofe , Confronted him with selfe – comparisons , Point against Point , rebellious Arme ‘ gainst Arme , Curbing his lauish spirit : and to conclude , The Victorie fell on vs”

Now this should actually take the sentence that starts with ‘Norway himselfe….’ and look for the longest word in it and do the steps above and so on but it doesn’t. Any suggestions? Thanks.

import nltk

from nltk.corpus import gutenberg

triggerSentence = raw_input("Please enter the trigger sentence: ")#get input str

split_str = triggerSentence.split()#split the sentence into words

longestLength = 0

longestString = ""

montyPython = 1

while montyPython:

    #code to find the longest word in the trigger sentence input
    for piece in split_str:
        if len(piece) > longestLength:
            longestString = piece
            longestLength = len(piece)


    listOfSents = gutenberg.sents() #all sentences of gutenberg are assigned -list of list format-

    listOfWords = gutenberg.words()# all words in gutenberg books -list format-
    # I tip my hat to Mr.Alex Martelli for this part, which helps me find the longest sentence
    lt = longestString.lower() #this line tells you whether word list has the longest word in a case-insensitive way. 

    longestSentence = max((listOfWords for listOfWords in listOfSents if any(lt == word.lower() for word in listOfWords)), key = len)
    #get longest sentence -list format with every word of sentence being an actual element-

    longestSent=[longestSentence]

    for word in longestSent:#convert the list longestSentence to an actual string
        sstr = " ".join(word)
    print triggerSentence + " "+ sstr
    triggerSentence = sstr

Answer 1

Mr. Hankin’s answer is more elegant, but the following is more in keeping with the approach you began with:

import sys
import string
import nltk
from nltk.corpus import gutenberg

def longest_element(p):
    """return the first element of p which has the greatest len()"""
    max_len = 0
    elem = None
    for e in p:
        if len(e) > max_len:
            elem = e
            max_len = len(e)
    return elem

def downcase(p):
    """returns a list of words in p shifted to lower case"""
    return map(string.lower, p)


def unique_words():
    """it turns out unique_words was never referenced so this is here
       for pedagogy"""
    # there are 2.6 million words in the gutenburg corpus but only ~42k unique
    # ignoring case, let's pare that down a bit
    for word in gutenberg.words():
        words.add(word.lower())
    print 'gutenberg.words() has', len(words), 'unique caseless words'
    return words

print 'loading gutenburg corpus...'
sentences = []
for sentence in gutenberg.sents():
    sentences.append(downcase(sentence))

trigger = sys.argv[1:]
target = longest_element(trigger).lower()
last_target = None

while target != last_target:
    matched_sentences = []
    for sentence in sentences:
        if target in sentence:
            matched_sentences.append(sentence)

    print '===', target, 'matched', len(matched_sentences), 'sentences'
    longestSentence = longest_element(matched_sentences)
    print ' '.join(longestSentence)

    trigger = longestSentence
    last_target = target
    target = longest_element(trigger).lower()

Given your sample sentence though, it reaches fixation in two cycles:

$ python nltkgut.py Thane of code
loading gutenburg corpus…
=== target thane matched 24 sentences
norway himselfe , with terrible
numbers , assisted by that most
disloyall traytor , the thane of
cawdor , began a dismall conflict ,
till that bellona ‘ s bridegroome ,
lapt in proofe , confronted him with
selfe – comparisons , point against
point , rebellious arme ‘ gainst arme
, curbing his lauish spirit : and to
conclude , the victorie fell on vs
=== target bridegroome matched 1 sentences
norway himselfe , with
terrible numbers , assisted by that
most disloyall traytor , the thane of
cawdor , began a dismall conflict ,
till that bellona ‘ s bridegroome ,
lapt in proofe , confronted him with
selfe – comparisons , point against
point , rebellious arme ‘ gainst arme
, curbing his lauish spirit : and to
conclude , the victorie fell on vs

Part of the trouble with the response to the last problem is that it did what you asked, but you asked a more specific question than you wanted an answer to. Thus the response got bogged down in some rather complicated list expressions that I’m not sure you understood. I suggest that you make more liberal use of print statements and don’t import code if you don’t know what it does. While unwrapping the list expressions I found (as noted) that you never used the corpus wordlist. Functions are a help also.