Question

I have a Python script that for-loops through some objects and fetches data from a url in xml-format and stores it in a MySQL db. I have attached a (simplified) version of the script below. I schedule the script to run once per day with the Windows Task Scheduler. The script and the scheduling works fine most of the time but once or twice per month the script terminates unexpectedly somewhere halfway through without any logged exceptions. When I detect that the script has terminated and manually rerun the script it completes without any problems and without any changes. The Windows Scheduler never reports any issues when the script has terminated prematurely, i.e. the ‘History’-tab just reports Action completed/Task completed just like it does when everything works as planned.

Simplified version of script:

for Obj in objects:
   t=0
   dbdata = ''
   logger.info('Object: {s}\tID: {i}'.format(s=Obj.name, i=Obj.id))
    try:
        url = 'http://www.xyz.com/webproxy/DataProxy.aspx?Object=' + Obj.id
        logger.debug(url)
        data = urlopen(url)
        dom = minidom.parse(data)
        for node in reversed(dom.getElementsByTagName('t')):
            dbdata = dbdata + str(node.getAttribute('t')) + '\t' + str(float(node.getAttribute('p'))) + '\t' + str(int(node.getAttribute('v'))) + '\t' + str(node.getAttribute('id')) + '\t' + str(node.getAttribute('b')) + '\t' + str(node.getAttribute('s')) + '\n'
            t=t+1
        if len(dbdata)>0:
            cursor.execute(sql,(Obj.id,Obj.name,daydb,dbdata))
        logger.info('# rows: {n}'.format(n=t)
    except HTTPError, e:
        logger.error(e.msg)
        logger.error('HTTPError. Error code: ' + str(e.code))
    except ExpatError, ex:
        logger.error(ex.msg)
        logger.error('Expat Error. Error code: ' + str(ex.code))
    except Exception, e:
        logger.error(e.msg)
        logger.error('Exception. Error code: ' + str(e.code))

Does anyone have any ideas why the script terminates prematurely every once in a while or if there is anything that i can do in the script or in the scheduling to avoid this problem or at least add some clarity to what is happening? Thanks

Answer 1

you said: “without any logged exceptions”?

This may be because Exception is not the base class of all exceptions

>>> help(Exception)
Help on class Exception in module exceptions:

class Exception(BaseException)
 |  Common base class for all non-exit exceptions.
 |  
 |  Method resolution order:
 |      Exception
 |      BaseException
 |      __builtin__.object

BaseException is the base class.

However I recommend this if you want to get all exceptions:

try:
   ...
except:
    import sys
    error_type, error, traceback = sys.exc_info()

This way you also catch SystemExit

>>> SystemExit.mro()
[<type 'exceptions.SystemExit'>, <type 'exceptions.BaseException'>, <type 'object'>]