Question

So, I wrote small test code for calculating the averages from all the csv files in a folder.

parser = argparse.ArgumentParser(description='avg data from csv to csv')
parser.add_argument('--input', required = True)
parser.add_argument('--output', required = True)
args = parser.parse_args()

# Get command arguments
input_files_path = args.input
output = args.output

avg = []
for infile in glob.glob(input_files_path+"*.csv"):
        a = np.loadtxt(infile, delimiter=",")
        mean = np.mean(a, axis=0)
        avg.append((mean))
        print infile
np.savetxt(output,(avg), fmt = "%.1f", delimiter=", ")

but the issue now, when I am trying to save the name of the file as well in the csv file like this:

parser = argparse.ArgumentParser(description='avg data from csv to csv')
parser.add_argument('--input', required = True)
parser.add_argument('--output', required = True)
args = parser.parse_args()

# Get command arguments
input_files_path = args.input
output = args.output

avg = []
for infile in glob.glob(input_files_path+"*.csv"):
        a = np.loadtxt(infile, delimiter=",")
        mean = np.mean(a, axis=0)
        avg.append((infile,(mean)))
        print infile
np.savetxt(output,(avg), fmt = "%.1f", delimiter=", ")

it throws an error:

    np.savetxt(output,(avg), fmt = "%.1f", delimiter=", ")
  File "/usr/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 963, in savetxt
    X = np.asarray(X)
  File "/usr/lib/python2.7/dist-packages/numpy/core/numeric.py", line 235, in asarray
    return array(a, dtype, copy=False, order=order)
ValueError: cannot set an array element with a sequence

I don’t seem to understand what the issue is. Maybe it is because I am trying to append a string to a float? I check the fmts in numpy.. but I did not find anything interesting.

/usr/lib/python2.7/dist-packages/numpy/lib/npyio.py:773: UserWarning: loadtxt: Empty input file: "../robalo-31.1/op.csv"
  warnings.warn('loadtxt: Empty input file: "%s"' % fname)
/usr/lib/python2.7/dist-packages/numpy/core/fromnumeric.py:2374: RuntimeWarning: invalid value encountered in double_scalars
  return mean(axis, dtype, out)

Answer 1

I’m guessing there’s a problem with the format characters you’re providing with np.savetxt. (you have a string-float tuple and you’re delimiter implies you’re only using floats). I would suggest csv for this instead:

>>> import csv
>>> import StringIO
>>> g = StringIO.StringIO()
>>> writer = csv.writer(g,delimiter=',')  #this is the part that is applicable
>>> writer.writerows(lst)
>>> g.seek(0)
>>> print g.read()
foo,2
bar,4

So I’m guessing your code could be simply:

with open(output,'wb') as fout:
    writer = csv.writer(fout,delimiter=',') 
    writer.writerows(avg)

instead of

np.savetxt(output,(avg), fmt = "%.1f", delimiter=", ")