Question

I’m continuing writing my twitter crawler and am running into more problems. Take a look at the code below:

from BeautifulSoup import BeautifulSoup
import re
import urllib2

url = 'http://mobile.twitter.com/NYTimesKrugman'

def gettweets(soup):
    tags = soup.findAll('div', {'class' : "list-tweet"})#to obtain tweet of a follower
    for tag in tags: 
        print tag.renderContents()
        print ('\n\n')

def are_more_tweets(soup):#to check whether there is more than one page on mobile 
    links = soup.findAll('a', {'href': True}, {id: 'more_link'})
    for link in links:
        b = link.renderContents()
        test_b = str(b)
        if test_b.find('more'):
            return True
        else:
            return False

def getnewlink(soup): #to get the link to go to the next page of tweets on twitter 
    links = soup.findAll('a', {'href': True}, {id : 'more_link'})
    for link in links:
        b = link.renderContents()
        if str(b) == 'more':
            c = link['href']
            d = 'http://mobile.twitter.com' +c
            return d

def checkforstamp(soup): # the parser scans a webpage to check if any of the tweets are   
    times = soup.findAll('a', {'href': True}, {'class': 'status_link'})
    for time in times:
        stamp = time.renderContents()
        test_stamp = str(stamp)
        if test_stamp.find('month'): 
            return True
        else:
            return False


response = urllib2.urlopen(url)
html = response.read()
soup = BeautifulSoup(html)
gettweets(soup)
stamp = checkforstamp(soup)
tweets = are_more_tweets(soup)
print 'stamp' + str(stamp)
print 'tweets' +str (tweets)
while (not stamp) and tweets: 
    b = getnewlink(soup)
    print b
    red = urllib2.urlopen(b)
    html = red.read()
    soup = BeautifulSoup(html)
    gettweets(soup)
    stamp = checkforstamp(soup)
    tweets = are_more_tweets(soup)
print 'done' 

The code works in the following way:
For a single user NYTimesKrugman
-I obtain all tweets on a single page(gettweets)
-provided more tweets exist(are more tweets) and that I haven’t obtained a month of tweets yet(checkforstamp), I get the link for the next page of tweets
-I go to the next page of tweets (entering the while loop) and continue the process until one of the above conditions is violated

However, I have done extensive testing and determined that I am not actually able to enter the while loop. Rather, the program is not doing so. This is strange, because my code is written such that tweets are true and stamp should yield false. However, I’m getting the below results: I am truly baffled!

<div>
<span>
<strong><a href="http://mobile.twitter.com/nytimeskrugman">NYTimeskrugman</a></strong>
<span class="status">What Would I Have Done? <a rel="nofollow"   href="http://nyti.ms/nHxb8L" target="_blank" class="twitter_external_link">http://nyti.ms/nHxb8L</a></span>
</span>
<div class="list-tweet-status">
<a href="/nytimeskrugman/status/98046724089716739" class="status_link">3 days ago</a>
</div>
<div class="list-tweet-actions">
</div>
</div>




stampTrue
tweetsTrue
done
>>> 

If someone could help that’d be great. Why can I not get more than 1 page of tweets? Is my parsing in checkstamp being done incorrectly? Thanx.

Answer 1

if test_stamp.find('month'): 

will evaluate to True if it doesn’t find month, because it returns -1 when it doesn’t find the substring. It would only evaluate to False here if month was at the beginning of the string, so its position was 0.

You need

if test_stamp.find('month') != -1: 

or just

return test_stamp.find('month') != -1