Question

I’m trying to write a program that will retrieve all of the tweets the user has seen on their twitter home timeline (ie, from people their following, as they would see at twitter.com). I realise this is a lot of data, and the rest API has limitations.

What would be the be best way to do this? Slowly retrieve the last 200 or whatever the limit is tweets, keeping in mind the 350 requests per hour limit? Or is there some hard limit to how far back I can go even with that?

The streaming API only streams from current point on I believe, so I don’t think this is an option. This is a personal project so I can’t pay very much for any elevated access or anything.

Answer 1

Yes, there is a limit to how far back you can go:

Clients may access a theoretical
maximum of 3,200 statuses via the page
and count parameters for the
user_timeline REST API methods. Other
timeline methods have a theoretical
maximum of 800 statuses. Requests for
more than the limit will result in a
reply with a status code of 200 and an
empty result in the format requested.
Twitter still maintains a database of
all the tweets sent by a user.
However, to ensure performance of the
site, this artificial limit is
temporarily in place.

Source: http://dev.twitter.com/pages/every_developer

As you mentioned, you will need to go page by page through each of the 200 max results that come back until you hit that limit and get the empty result set, making sure not to hit the 350 requests per hour. There might also be gaps depending on how many tweets the user has on their timeline.