Question

I have a simple python script that updates that statuses of justin.tv streams in my database. It’s a Django based web application. This script worked perfectly before I moved it to my production server, but now it has issues with timing out or freezing. I’ve solved the time out problem by adding try/except blocks and making the script retry, but I still can’t figure out the freezing problem.

I know it freezes on the line streamOnline = manager.getStreamOnline(stream.name, LOG). That’s the same point where the socket.timeout exception occurs. Some times however, it just locks up for ever. I just can’t picture a scenario where python would freeze infinitely. Here is the code for the script that freezes. I’m linking website.networkmanagers below, as well as oauth and the justin.tv python library that I’m using.

import sys, os, socket

LOG = False

def updateStreamInfo():
    # Set necessary paths
    honstreams = os.path.realpath(os.path.dirname(__file__) + "../../../")
    sys.path.append(honstreams)
    os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

    # Import necessary moduels
    from website.models import Stream, StreamInfo
    from website.networkmanagers import get_manager, \
                                        NetworkManagerReturnedErrorException

    # Get all streams
    streams = Stream.objects.all()

    try:
        # Loop through them
        for stream in streams:

            skipstream = False

            print 'Checking %s...' % stream.name,
            # Get the appropriate network manager and
            manager = get_manager(stream.network.name)

            # Try to get stream status up to 3 times
            for i in xrange(3):
                try:
                    streamOnline = manager.getStreamOnline(stream.name, LOG)
                    break
                except socket.error as e:
                    code, message = e

                    # Retry up to 3 times
                    print 'Error: %s. Retrying...'

            # If this stream should be skipped
            if(skipstream):
                print 'Can\'t connect! Skipping %s' % stream.name
                continue

            # Skip if status has not changed
            if streamOnline == stream.online:
                print 'Skipping %s because the status has not changed' % \
                      stream.name
                continue

            # Save status
            stream.online = streamOnline
            stream.save()

            print 'Set %s to %s' % (stream.name, streamOnline)

    except NetworkManagerReturnedErrorException as e:
        print 'Stopped the status update loop:', e

if(__name__ == "__main__"):
    if(len(sys.argv) > 1 and sys.argv[1] == "log"):
        LOG = True

    if(LOG): print "Logging enabled"

    updateStreamInfo()

networkmanagers.py
oauth.py
JtvClient.py

Example of the script freezing

foo@bar:/…/honstreams/honstreams# python website/scripts/updateStreamStatus.py
Checking angrytestie… Skipping angrytestie because the status has not changed
Checking chustream… Skipping chustream because the status has not changed
Checking cilantrogamer… Skipping cilantrogamer because the status has not changed
| <- caret sits here blinking infinitely

---

Interesting update

Every time it freezes and I send a keyboard interrupt, it’s on the same line in socket.py:

root@husta:/home/honstreams/honstreams# python website/scripts/updateStreamStatus.py
Checking angrytestie... Skipping angrytestie because the status has not changed
Checking chustream... Skipping chustream because the status has not changed
^CChecking cilantrogamer...
Traceback (most recent call last):
  File "website/scripts/updateStreamStatus.py", line 64, in <module>
    updateStreamInfo()
  File "website/scripts/updateStreamStatus.py", line 31, in updateStreamInfo
    streamOnline = manager.getStreamOnline(stream.name, LOG)
  File "/home/honstreams/honstreams/website/networkmanagers.py", line 47, in getStreamOnline
    return self.getChannelLive(channelName, log)
  File "/home/honstreams/honstreams/website/networkmanagers.py", line 65, in getChannelLive
    response = client.get('/stream/list.json?channel=%s' % channelName)
  File "/home/honstreams/honstreams/website/JtvClient.py", line 51, in get
    return self._send_request(request, token)
  File "/home/honstreams/honstreams/website/JtvClient.py", line 90, in _send_request
    return conn.getresponse()
  File "/usr/lib/python2.6/httplib.py", line 986, in getresponse
    response.begin()
  File "/usr/lib/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.6/httplib.py", line 349, in _read_status
    line = self.fp.readline()
  File "/usr/lib/python2.6/socket.py", line 397, in readline
    data = recv(1)
KeyboardInterrupt

Any thoughts?

Answer 1

It turns out this HTTP connection isn’t passed a timeout in jtvClient.py

def _get_conn(self):
    return httplib.HTTPConnection("%s:%d" % (self.host, self.port))

Changed the last line to

return httplib.HTTPConnection("%s:%d" % (self.host, self.port), timeout=10)

Which solved it