Question

If we consider the following generator function …

def loadrun(runs, varnames=None):
    for fle in runs:
        yield loadmat(fle, variable_names=varnames)

which we call as follows …

vnames = ['targetpos', 'cuepos', 'soa', 'response']
trials = loadrun(datafiles, vnames)  # datafiles is a list of paths to files

we get the following output when listing the length of the resulting dict:

[in] for t in trials:
         print len(t)

[out] 7
      128
      128
      128
      128
      128
      128
      128

All .mat files contain 124 variables, so only the first output is correct. What gives?

EDIT

I further reduced the scope of the problem. It seems as though loadmat‘s variable_names field fails after the first iteration in any loop. Any idea what’s going on here?

for df in datafiles:
    print len(loadmat(df, variable_names=vnames))

[out] 7
      128
      128
      128
      128
      128
      128
      128

Answer 1

scipy.io.loadmat is being naughty^[1] and mutating the list passed to variable_names. After the first call to loadmat, varnames is the empty list.

You can workaround this by defining a temporary variable, _varnames:

def loadrun(runs, varnames = None):
    for fle in runs:
        _varnames = None if varnames is None else varnames[:]
        yield sio.loadmat(fle, variable_names = _varnames)

---

For example:

import string
import scipy.io as sio
import numpy as np

letters = string.ascii_lowercase
datafiles = ['/tmp/data{i:d}.mat'.format(i = i) for i in range(3)]

def loadrun(runs, varnames = None):
    for fle in runs:
        _varnames = None if varnames is None else varnames[:]
        yield sio.loadmat(fle, variable_names = _varnames)

def create_datafiles():
    for filename in datafiles:
        arr = np.arange(9).reshape((3, 3))
        mdict = {letter : arr for letter in letters}
        sio.savemat(filename, mdict = mdict)

create_datafiles()
vnames = ['a', 'c', 'd']
trials = loadrun(datafiles, vnames)
for t in trials:
    print(t.keys())

---

Actually, it appears the problem originates in both mio4.py and mio5.py, where MatFile4Reader and MatFile5Reader calls the remove method on variable_names:

def get_variables(self, variable_names=None):
    while not self.end_of_stream():
        ...
        if variable_names:
            variable_names.remove(name)
            if len(variable_names) == 0:
                break