Question

I’m trying to construct a social network graph of twitter users who have mentioned a particular topic. My strategy to do this goes roughly like this:

1. Query twitter for a topic. Collect the first 100 tweets that come up and add those users to the graph.
2. For each user:
3. Retrieve friends and followers.
4. Query each friend/follower for the topic. If they turn up a result (meaning they’ve discussed the topic), add them to the graph.
5. For each user that was added to the graph, return to step 2 until the desired search depth is reached.

My problem is two-fold. First of all, this approach quickly exceeds my search API rate limit. Even with a search depth of 2, it’s quite likely that I’ll find people with 100+ friends/followers and I am unable to query them all before hitting the rate limit.

Secondly, this all takes quite awhile. Twitter API is not fast. In the hypothetical event that I was not rate limited, I could submit the requests asynchronously, but I can’t help wondering if there is a more efficient way.

I’ve tried aggregating the requests into one query per search depth:
topic AND from:name1 OR from:name2 …. OR from:namei

This basically explodes. I get a connection reset error from the twitter API. If I copy the query into the twitter web page, it just sits for awhile and then says “loading tweets seems to be taking awhile.”

I also emailed [e-mail] to ask for suggestions / access increase, but no response so far.

If anyone has any suggestions on how to go about gathering this type of information through the twitter API, I would very much appreciate it. I am currently using twitter4j and java.

Answer 1

Have you tried just using a filtered stream for a topic, and building the graph using mentions and retweets? This is quite indirect, and will still be slow, but won’t hit any rate limits.

See http://truthy.indiana.edu/ and http://cnets.indiana.edu/groups/nan/truthy