Home/ Questions/Q 8525051
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-11T07:51:45+00:00

I have a python cgi script that runs an application via subprocess over and

  • 0

I have a python cgi script that runs an application via subprocess over and over again (several thousand times). I keep getting the same error…

Traceback (most recent call last):
  File "/home/linuser/Webpages/cgi/SnpEdit.py", line 413, in <module>
    webpage()
  File "/home/linuser/Webpages/cgi/SnpEdit.py", line 406, in main
    displayOmpResult(form['odfFile'].value)
  File "/home/linuser/Webpages/cgi/SnpEdit.py", line 342, in displayContainerDiv
    makeSection(position,sAoiInput)
  File "/home/linuser/Webpages/cgi/SnpEdit.py", line 360, in displayData
    displayTable(i,j,lAmpAndVars,dOligoSet[key],position)
  File "/home/linuser/Webpages/cgi/SnpEdit.py", line 247, in displayTable
    p = subprocess.Popen(['/usr/bin/pDat',sInputFileLoc,sOutputFileLoc],stdout=fh, stderr=fh)
  File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
    errread, errwrite)
  File "/usr/lib/python2.6/subprocess.py", line 1039, in _execute_child
    errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files

The function causing it is below.

def displayTable(sData):

    # convert the data to the proper format
    sFormattedData = convertToFormat(sData)

    # write the formatted data to file
    sInputFile = tempfile.mkstemp(prefix='In_')[1]
    fOpen = open(sInputFile,'w')
    fOpen.write(sFormattedData)
    fOpen.close()

    sOutputFileLoc = sInputFile.replace('In_','Out_')

    # run app, requires two files; an input and an output
    # temp file to holds stdout stderr of subprocess
    fh = tempfile.TemporaryFile(mode='w',dir=tempfile.gettempdir())
    p = subprocess.Popen(['/usr/bin/pDat',sInputFileLoc,sOutputFileLoc],stdout=fh, stderr=fh)
    p.communicate()
    fh.close()

    # open output file and print parsed data into a list of dictionaries
    sOutput = open(sOutputFileLoc).read()
    lOutputData = parseOutput(sOutput)

    displayTableHeader(lOutputData)
    displaySimpleTable(lOutputData)

As far as I can tell, I’m closing the files properly. When I run…

import resource
print resource.getrlimit(resource.RLIMIT_NOFILE)

I get…

(1024, 1024)

Do I have to increase this value? I read that subprocess opens several file descriptors. I tried adding “close_fds = True” and I tried using the with statement when creating my file but the result was the same. I suspect the problem may be with the application that I’m subprocessing, pDat, but this program was made by someone else. It requires two inputs; an input file and the location of where you want the output file written to. I suspect it may not be closing the output file that it creates. Aside from this, I can’t see what I might be doing wrong. Any suggestions? Thanks.

EDIT:
I’m on ubuntu 10.04 running python 2.6.5 and apache 2.2.14

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Instead of this…

    sInputFile = tempfile.mkstemp(prefix='In_')[1]
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()

    I should have done this…

    iFileHandle,sInputFile = tempfile.mkstemp(prefix='In_')
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
os.close(iFileHandle)

    The mkstemp function makes OS level handles to a file and I wasn’t closing them. The solution is described in more detail here…
    http://www.logilab.org/blogentry/17873

    • 0