Question

Consider:

>>> import pickle
>>> thing = open('foobar.txt','w')
>>> pickle.dumps(thing)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.6/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.6/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/usr/lib/python2.6/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle file objects

Seems entirely reasonable – of course I can’t pickle an open file handle. But:

>>> pickle.dumps(thing, 2)
'\x80\x02c__builtin__\nfile\nq\x00)\x81q\x01.'
>>> pickle.loads(pickle.dumps(thing, 2))
<closed file '<uninitialized file>', mode '<uninitialized file>' at 0x7ff3c078>

Apparently I can pickle an open file, just not usefully.

Is this deliberate? It was obscuring a bug in my code, where I was wrongly pickling an object that owned a file. Under some conditions, that object also holds a pyodbc cursor, with the same result.

I don’t see anything in PEP 307 about it. Was it just an oversight, or is there something important going on that I’m missing, that could let me get the exception I want even when pickling using protocol 2?

I’m using Python 2.6.5. I know, I know, but it’s what comes with my distribution.

Answer 1

On the Python Wiki, it says

You cannot pickle open file objects, network connections, or database connections. When you think about it, it makes sense — pickle cannot will the connection for file object to exist when you unpickle your object, and the process of creating that connection goes beyond what pickle can automatically do for you. If you really want to pickle something that has an attribute that is causing problems, look at the pickle documentation for __getstate__, __setstate__, and __getinitargs__ — using these you can exclude problematic attributes.

However, I found this bug report which indicates that you actually can pickle file objects. This does seem to be unintentional. It’s been fixed in Python 3.2.

You could see if you could adapt that patch to Python 2.6 if you wanted to prevent it from happening. Otherwise, you just need to be careful what you pickle.