Question

I need to get some meteorological data into a MySQL database.
File inputFile.csv is a comma-delimited list of values. There are 241 lines and 481 values per line.
Each line maps to a certain latitude, and each value’s position within the line maps to a certain longitude.

There are two additional files with the same structure, lat.csv and lon.csv. These files contain the coordinates that the values in inputFile.csv map to.

So to find the latitude and longitude for a value in inputFile.csv, we need to refer to the values at the same line/position (or row/column) within lat.csv and lon.csv

I want to translate inputFile.csv using lat.csv and lon.csv such that my output file contains a list of values (from inputFile.csv),latitudes, and longitudes.

Here is a small visual example:

    inputFile.csv
    3,5,1,4,5
    1,4,1,2,5
    5,7,3,8,0

    lat.csv
    22,31,51,21,52
    55,21,24,66,12
    11,23,12,55,55

    lon.csv
    12,35,12,52,11
    35,11,25,33,42
    62,53,45,25,54


    output:
    val lat lon
    3   22  12
    5   31  35
    1   51  12
    4   21  52
    5   52  11 
    1   55  35
    4   21  11
    1   24  25  
    2   66  33
    etc

What is the best way to do this in python/numpy?

Answer 1

I suppose that since you know the total size the the array that you want, you can preallocate it:

a = np.empty((241*481,3))

Now you can add the data:

for i,fname in enumerate(('inputFile.csv','lat.csv','lon.csv')):
    with open(fname) as f:
        data = np.fromfile(f,sep=',')
        a[:,i] = data.ravel()

---

If you don’t know the number of elements up front, you can generate a 2-d list instead (a list of np.ndarrays):

alist = []
for fname in ('inputFile.csv','lat.csv','lon.csv'):
    with open(fname) as f:
        data = np.fromfile(f,sep=',')
        alist.append( data.ravel() )
a = np.array(alist).T