Question

I am using numpy.loadtxt to extract a large array of data from a text file and then using a loop to put different columns into different dictionary keys like this:

f = numpy.loadtxt(datafile, skiprows=5) # Open and read in the file, skipping to the data
d = {} # Create empty dictionary

for x in range(0, f.shape[1]):
    d[x] = f[:,x]     # Loop through the columns of the datafile, putting each one into
#a dictionary index

The row above the array in the text file contains all the titles for the variables in the array, is there a way to get each variable name and put it as the key name for the relevant dictionary? (ie, column one = data, d[date]={14/11/12,15/11/12 …. etc)

Answer 1

Pandas is a good idea, so “thumbs up” to the answer by reptilicus.

If you don’t want the dependency on Pandas, you can just as easily use the function numpy.genfromtxt to read the data directly into a numpy structured array. A structure array acts like both a numpy 1-d array and a dictionary.

For example, here’s a sample data file, “data.csv”:

alpha, beta, gamma
100, 0.5, 19.9
210, 0.25, 21.0
240, 0.45, 15.0
290, 0.75, 5.5

You can read this into a structured array as follows:

>>> data = genfromtxt('data.csv', delimiter=',', names=True, dtype=None)

The option names=True tells genfromtxt to use the columns headings as the field names in the structured array. Setting dtype=None tells genfromtxt to figure out the data type of the columns automatically (the default is to convert all values to double precision floating point values).

data looks like this.

>>> data
array([(100, 0.5, 19.9), (210, 0.25, 21.0), (240, 0.45, 15.0),
       (290, 0.75, 5.5)], 
      dtype=[('alpha', '<i4'), ('beta', '<f8'), ('gamma', '<f8')])

You can access individual elements (each is a structure containing three fields):

>>> data[0]
(100, 0.5, 19.9)

Or you can access columns using the dictionary-like interface:

>>> data['beta']
array([ 0.5 ,  0.25,  0.45,  0.75])

And you can combine those:

>>> data['beta'][1]
0.25
>>> data[1]['beta']
0.25