Question

I’m relatively new to Python and wondering how best to import data from multiple files into a single array. I have quite a few text files containing 50 rows of two columns of data (column delimited) such as:

Length=10.txt:     
1, 10    
2, 30    
3, 50   
#etc
END OF FILE

–

Length=20.txt
1, 50.7
2, 90.9
3, 10.3
#etc
END OF FILE

Let’s say I have 10 text files to import and import into a variable called data.

I’d like to create a single 3D array containing all data. That way, I can easily plot and manipulate the data by referring to the data by data[:,:,n] where n refers to the index of the text file.

I think the way I’d do this is to have an array of shape (50, 2, 10), but don’t know how best to use python to create it. I’ve thought about using a loop to import each text file as a 2D array, and then stack them to create a 2D array, although couldn’t find the appropriate commands to do this (I looked at vstack and column_stack in numpy but these don’t seem to add an extra dimension).

So far, I’ve written the import code:

    file_list = glob.glob(source_dir + '/*.TXT') #Get folder path containing text files

    for file_path in file_list:
      data = np.genfromtxt(file_path, delimiter=',', skip_header=3, skip_footer=18)

But the problem with this code, is that I can only process data when it’s in the for loop.

What I really want is an array of all data imported from the text files.

Any help would be greatly appreciated thanks!

Answer 1

“But the problem with this code, is that I can only process data when it’s in the for loop. “

Assuming your code works:

# Get folder path containing text files
file_list = glob.glob(source_dir + '/*.TXT')
data = []
for file_path in file_list:
    data.append(
        np.genfromtxt(file_path, delimiter=',', skip_header=3, skip_footer=18))
# now you can access it outside the "for loop..."
for d in data:
    print d