Question

I’m working on a Python script that needs to create about 50 distinct temporary files, which are all appended frequently during the course of the script and merged at the end. I’m sure that the tempfile module can do what I need, but I haven’t been able to figure out how from reading the documentation.

I want to use temporary files–as opposed to variables–to conserve system memory, as these data chunks grow large as the script processes tens of thousands of other files.

The following chunk of code is the hack I’m currently using to create these files (untemporarily) in an untemporary directory:

item = (string from another file)   # string must id file for future use
tmpfile = 'tmpfiles/' + item
if item not in totalitems:
   totalitems.add(item)
   with open(tmpfile, 'w') as itemfile:
      output = some stuff
      tmpfile.write(output)
else:
   with open(tmpfile, 'a') as itemfile:
      output = different stuff
      tmpfile.write(output)

I think what I need is tempfile.NamedTemporaryFile(). According to the documentation:

That name can be retrieved from the name member of the file object.

Unfortunately, I don’t understand what that means. I just need to be able to call each file again later when I run across its corresponding “item” again in the files I’m processing. I presume this is rather straight forward and I’m just being dense. In case it matters, I have versions of this script for both Python 2.7.1 and 3.2.3. I only really need for one or the other to work; I created both just as a learning exercise.

Answer 1

“That name can be retrieved from the name member of the file object.”

means that you can get the name of the temporary file created like so:

In [4]: import tempfile

In [5]: tf = tempfile.NamedTemporaryFile()  
In [6]: tf.name  # retrieve the name of the temp file just created
Out[6]: 'c:\\blabla\\locals~1\\temp\\tmptecp3i'

Note: By default the file will be deleted when it is closed. However, if the delete parameter is False, the file is not automatically deleted. See the Python docs on this for more information.

Since you can retrieve the name of each of the 50 temp files you want to create, you can save them, e.g., in a list, before you use them again later (as you say). Just be sure to set the delete value accordingly so that the files don’t disappear when you close them (in case you plan to close, and then later reopen them).

I explained how to create temporary filenames in more detail here Best way to generate random file names in Python