Question

I am trying to loop through some 50-odd files in a directory. Each file has some text for which i am trying to find the keywords using Yahoo Term Extractor. I am able to extract text from each file, but I am not able to iteratively call the API using the text as input. Only the keywords for the first file is displayed.

Here is my code snippet:
in ‘comments’ list, I have extracted and stored the text from each file.

for c in comments:

    print "building query"
    dataDict = [ ('appid', appid), ('context', c)]
    queryData = urllib.urlencode(dataDict)
    request.add_data(queryData)
    print "fetching result"
    result = OPENER.open(request).read()
    print result
    time.sleep(1)

Answer 1

Well I don’t know anything about the Yahoo Term Extractor, but I’d presume that your call request.add_data(queryData) simply tacks on another data set with each iteration of your loop. And then the call to OPENER.open(request).read() would probably only process the results of the first data set. So either your request object can only hold one query, or your OPENER object’s inner workings can only process one query, it’s as simple as that.

Actually a third reason comes to mind now that I read the documentation provided at your link, and this is probably the true one:

RATE LIMITS

The Term Extraction service is limited to 5,000 queries per IP address per day and to noncommercial use. See information on rate limiting.

So it would make sense that the API would limit your usage to one query at a time, and not allow you to flood a bunch of queries in a single request.

In any event, I’d assume you could fix your problem in a "naive" way by having many request variables instead of just one, or maybe just creating a new request with every iteration of your loop. If you’re not worried about storing your results, and just trying to debug, you could try:

for c in comments:
    print "building query"
    dataDict = [ ('appid', appid), ('context', c)]
    queryData = urllib.urlencode(dataDict)
    request = urllib2.Request() # I don't know how to initialize this variable, do it yourself
    request.add_data(queryData)
    print "fetching result"
    result = OPENER.open(request).read()
    print result
    time.sleep(1)

Again, I don’t know about the Yahoo Term Extractor (nor do I really have time to research it) so there may very well be a better, more native way to do this. If you post more details of your code (i.e. what classes are the request and OPENER objects coming from) then I might be able to elaborate on this.