Question

I am reading data from a special device that is producing around 20 MB/s

/dev/dvb/adapter0/dvr0

I need to read the contents and copy to other fifos

    device_fd = os.open(video_device_file, os.O_RDONLY)
    while True:
        if jobm.has_jobs():
            chunk = os.read( device_fd, 1024 * 1000  )
            fifom.write2all(chunk)
        jobm.idle()

After running it for the whole night I received the following error:

[Errno 75] Value too large for defined data type

In the os.read line.

From google seems that is related to the c library and 32bit pointers but I don’t see how this can affect my python version that is a 64 bit one.

Any suggestions?

Answer 1

I don’t know what your jobm object is, I assume it’s created by you because this is the only Google result for a similar object, so correct me if I’m wrong.

You seem to be opening the video device for reading, the video device is constantly producing video data, and instead of reading it, you’re idling. You don’t seem to be sending the device_fd to your jobm thing, so it can’t know if there’s activity on the fd, and so it is sleeping when you should be reading, which creates a buffer overflow.

You need to always read data from device_fd if there is any available. Send the fd to the jobm, and make it utilize poll or select to look for activity on it.

Edit: Excuse me, I missed a detail from your question. What I said still applies, but not as directly – you need to watch both your inputs and outputs, not only your outputs, for the reason that I’ve stated.

If the fifos can’t handle the output, and as a result you miss a read on your device, you get the issue you’re getting. You need to either 1) create a buffer that holds up to several hundred MB of data in case a pipe is late for its writing, 2) drop packets. DVB apps should handle it if some part of the data is missing. You can also simply catch and ignore the exception, although it would lead to minor corruption and missing data in the output.