Question

I’m porting a Matlab script to Python. Below is an extract:

%// Create a list of unique trade dates
DateList = unique(AllData(:,1));

%// Loop through the dates
for DateIndex = 1:size(DateList,1)

    CalibrationDate = DateList(DateIndex);
    %// Extract the data for a single cablibration date (but all expiries)
    SubsetIndices = ismember(AllData(:,1) , DateList(DateIndex)) == 1;    
    SubsetAllExpiries = AllData(SubsetIndices, :);

AllData is an N-by-6 cell matrix, the first 2 columns are dates (strings) and the other 4 are numbers. In python I will be getting this data out of a csv so something like this:

import numpy as np
AllData = np.recfromcsv(open("MyCSV.csv", "rb"))

So now if I’m not mistaken AllData is a numpy array of ordinary tuples. Is this is best format to have this data in? The goal will be to extract a list of unique dates from column 1, and for each date extract the rows with that date in column 1 (column one is ordered). Then for each row in column one do some maths on the numbers and date in the remaining 5 columns.

So in matlab I can get the list of dates by unique(AllData(:,1)) and then I can get the records (rows) corresponding to that date (i.e. with that date in columns one) like this:

SubsetIndices = ismember(AllData(:,1) , MyDate) == 1;    
SubsetAllExpiries = AllData(SubsetIndices, :);

How can I best achieve the same results in Python?

Answer 1

To put things in context, np.recfromcsv is just a modified version of np.genfromtxt which outputs record arrays instead of structured arrays.

A structured array lets you access the individual fields (here, your columns) by their names, like in my_array["field_one"] while a record array gives you the same plus the possibility to access the fields as attributes, like in my_array.field_one. I’m not fond of “access-as-attributes”, so I usually stick to structured arrays.

For your information, structurede/record arrays are not arrays of tuples, but arrays of some numpy object call a np.void: it’s a block of memory composed of as many sub-blocks you have of fields, the size of each sub-block depending on its datatype.

That said, yes, what you seem to have in mind is exactly the kind of usage for a structured array. The approach would then be:

• to take your dates array and filter them to find the unique elements.
• to find the indices of these unique elements, as an array of integers we’ll call, say, matching;
• to use matching to access the corresponding records (eg, rows of your array) using fancy indexing, as
  my_array[matching].
• to perform your computations on the records, as you want.

Note that you can keep your dates as strings or transform them into datetime objects using a user-defined converter, as described in the documentation. For example, your could transform a YYYY-MM-DD into a datetime object with a lambda s:datetime.dateime.strptime(s,"%Y-%m-%d"). That way, instead of having, say, a N array where each row (a record) consists of two dates as strings and 4 floats, you would have a N array where each row consists of two datetime objects and 4 floats.

Note the shape of your array (via my_array.shape), it says (N,), meaning it’s a 1D array, even if it looks like a 2D table with multiple columns. You can access individual fields (each “column”) by using its name. For example, if we create an array consisting of one string field called first and one int field called second, like that:

x = np.array([('a',1),('b',2)], dtype=[('first',"|S10"),('second',int)])

you could access the first column with

>>> x['first']
array(['a', 'b'], 
      dtype='|S10')