Question

I have a program which needs to hold approximately 3000 open file descriptors in Matlab.
The reason for this is that if I dont keep them open I need to open and close them
over 100 000 times which means 300 million open close operations. Considering that each file is appended to each time and fopen and fclose can take upwards of a second each (files are large i.e. 100mb+) it should be clear this situation is unacceptable.

I know that the Windows limit for file handles is set at 10000 but Matlab refuses to open more than 512 files with fopen. I cant figure out how to force it to increase that number.

Does someone know how to change the 512 limit? Where is it defined?
Is it even Matlab related?

Answer 1

Can’t you really review your program and structure it differently so as to work only from partial memory representation of the files content?

For instance, if this is to append 100 000 lines to 3000 files (i.e even no need to have any representation of what’s already in the files), you may do it in this way:

%% Main processing
function [] FullProcess()
%[        
    for block = 1:100,

        % Partial processing 
        lines = processBlock(block);

        % Save step   
        pushToFiles(block, lines);     

    end        
%]

With:

% Partial processing in memory
function [lines] = processBlock(block)
%[
    % Preallocate
    lines = cells(1000, 3000);

    % Do the processing for current block
    ...
    lines{500, 12} = 'kikou';
    ...
%]

And:

%% Save partial work
function [] = pushToFiles(block, lines)
%[
    fcount = size(lines, 2);
    lcount = size(lines, 1);
    for fi = 1:fcount,

       [fid, msg] = fopen(fprintf('f%i', fi), 'a'); % Open in append mode
       if (fid < 0), error(msg); end

       for li = 1:lcount,
           fprintf(fid, lines{li, fi});
       end

       fclose(fid);

    end
%]

This reduce things to do 100 fopen/fclose (by 3000 files though, but this is far less than expected before)