Question

I have a class which extends BufferedReader, and a list of file streams
b.close() is called for all except the last stream, I want to keep the streams open
How do I do this?

thanks

class TestReader(BufferedReader):
    pass

def test(streams):
    for stream in streams:
        b=TestReader(stream)
        do_something(b)
    #all the streams except streams[-1] are closed, how do I prevent this?

streams=[open('test1.txt','rb'),open('test2.txt','rb')]
test(streams)
streams.do_something_else()

Answer 1

Even though in the implementation BufferedIOBase classes wrap an IOBase object, their interface is a stream (everything inherits from IOBase), so the ordinary behavior of a IOBase object is to close themselves when they go out of scope. BufferedIOBase just delegates the close() call to the underlying stream.

You shouldn’t view a BufferedReader as a stream wrapper (although that is how it is implemented), but as a type-casting of an existing stream. The state of the two streams is completely bound together. However, you can unbind the wrapped stream with detach(), but this leaves the BufferedIOBase object useless.

Additionally, io.open returns a BufferedReader already when mode is rb, so you are double-buffering. You should use io.FileIO instead.

You have a few choices:

1. Create a new stream and a new underlying file descriptor, and pass around file names instead of streams. This is your easiest and safest option.

2. Create raw file descriptors and create streams from them as needed. This requires some
   care that multiple streams are not using the same file descriptor at the same time. For example:

   fd = os.open('test.txt', os.O_RDONLY)
   file1 = FileIO(fd, 'r', closefd=False)
   file2 = FileIO(fd, 'r', closefd=False)
   
   file1.read(100)
   assert file1.tell() == 100
   file2.read(100)
   assert file1.tell() == 200
   

3. detach() the underlying stream before your BufferedIOBase object closes its stream. (Remember to rewind the stream!)

   def test(streams):
       for stream in streams:
           b=TestReader(stream)
           do_something(b)
           wrappedstream = b.detach()
           assert wrappedstream is stream
   

   You can even implement this in your destructor:

   class TestReader(BufferedReader):
       def __del__(self):
           self.detach()
           # self.raw will not be closed,
           # rather left in the state it was in at detachment
   

   Or just disable close() delegation completely if you think the semantics are wrong:

   class TestReader(BufferedReader):
       def close(self):
           self.closed = True
   

I don’t have the big picture of what you are doing (possibly you need a different design), but this is how I would implement the code I see:

from io import FileIO, BufferedReader
import io
import os

class TestReader(BufferedReader):
    pass

def test(streams):
    for stream in streams:
        b = TestReader(stream)

def test_reset(streams):
    """Will try to leave stream state unchanged"""
    for stream in streams:
        pos = stream.tell()
        b = TestReader(stream)
        do_something(b)
        b.detach()
        stream.seek(pos)



filenames = ['test1.txt', 'test2.txt']

# option 1: just make new streams

streams = [FileIO(name, 'r') for name in filenames]
test(streams)
streams = [io.open(name, 'rb') for name in filenames]
#etc


# option 2: use file descriptors
fds = [os.open(name, os.O_RDONLY) for name in filenames]
#closefd = False means "do not close fd on __del__ or __exit__"
#this is only an option when you pass a fd instead of a file name
streams = [FileIO(fd, 'r', closefd=False) for fd in fds]
test(streams)
streams = []
for fd in fds:
    os.lseek(fd, 0, os.SEEK_SET)
    streams.append(io.open(fd, 'rb', closefd=False))
    # you can also .seek(0) on the BufferedReader objects
    # instead of os.lseek on the fds


# option 3: detach

streams = [FileIO(name, 'r') for name in filenames]
test_reset(streams)
# streams[*] should still be in the same state as when you passed it in