Question

I’m working on a Ruby script to automatically collect tweets from a list of users or terms, and it works except when Twitter is over capacity. Then, Twitter returns an HTML page with no error code for me to capture. The HTML throws off the parser and the script fails. How can I check for that HTML response and handle it gracefully? I tried using an “until success” approach (commented out in my code below), but that ended up in the rescue every time.

I’m using the logger, twitter and twitter4r gems and authenticating via OAuth. The code below works unless Twitter is over capacity. I only get to this section if no RestErrors are returned.

Here’s my code:

  #        until success
  #          begin
              if search_type == "users"
  #              begin
                  tweets_array = client.timeline_for(:user, :id => row.chomp, :since_id => since_status_id, :count => 200) 
                  success = true
                # rescue
                #   log.info("error getting tweets. waiting to try again.")
                #   sleep 180
                # end
              elsif search_type == "terms"
  #              begin
                  tweets_array = client.search(:q => row.chomp, :since_id => since_status_id, :count => 200)
                  success = true
                # rescue
                #   log.info("error getting tweets. waiting to try again.")
                #   sleep 180
                # end
              elsif
                log.fatal("unsupported search type. exiting.")
                break
              end # search type
  #          end 

Code I’m trying based on answers:

        begin
          tweets_array = client.timeline_for(:user, :id => row.chomp, :since_id => since_status_id, :count => 200) 
          success = true
        rescue Twitter::RESTError => re
          log.info(row.chomp + ": " + re.code.to_s + " " + re.message.to_s + " " + re.uri.to_s)
          if re.code = 503
            sleep 180
          end
        end

Answer 1

You can check the HTTP status code of the result, it should be 503.

http://twitter.com/503