Question

best wishes for 2013!

I’m using SciPy’s weave inline with some C++ code of mine to transpose huge matrices (about 200.000 x 15). It works like a charm, but I have a question about typecasting:

My input matrix is read from a file, comma separated etc., so all the entries are strings rather than floats (‘0.551’ rather than 0.551). This doesn’t affect the way the transpose function works, but later on I have to convert certain rows into numpy float arrays anyways, so I was wondering if this could be done in the C++ code instead. Let me explain with some code:

def transpose(lines, N, x):
    code =  """
            py::list matrix;
            for(int i = 0; i < x; i++) {
                py::list line;
                if(i == 1) { continue; }
                for(int j = 0; j < N; j++) {
                    line.append(lines[j][i]);
                }
                matrix.append(line);
            }
            return_val = matrix;
            """
    return weave.inline(code, ['lines', 'N', 'x'])

matrix = [['0.5','0.1'],['0.2','0.2']]
matrixT = transpose(matrix, len(matrix), len(matrix[0]))
final_result = np.array(matrixT[0], dtype=float)

In the example my small matrix will be transposed and my example result will be the first row of the transposed matrix converted to a numpy array of dtype float. Can this be done in C++ code instead? I’ve tried using double x = (double) lines[j][i] and things like that, but it somehow doesn’t work for appending to a py::list object.

Answer 1

The following can do the whole thing you are after:

def transpose(lines):
    code =  """
            for(int i = 0; i < x; i++) {
                for(int j = 0; j < N; j++) {
                    out[j + i * N] = atof(lines[j][i]);
                    // OUT2(i, j) = atof(lines[j][i]);
                }
            }
            """
    N = len(lines)
    x = len(lines[0])
    out = np.empty((x, N), dtype=np.float64)
    weave.inline(code, ['lines', 'N', 'x', 'out'])
    return out

>>> matrix = [['0.5', '0.1', '0.7'],['0.2','0.2', '0.4']]
>>> matrix
[['0.5', '0.1', '0.7'], ['0.2', '0.2', '0.4']]
>>> transpose(matrix)
array([[ 0.5,  0.2],
       [ 0.1,  0.2],
       [ 0.7,  0.4]])

Aside from constantly forgetting ; after something like 6 years without writing any C, I had a lot of trouble figuring out what out was turning into inside the C++ code, and in the end it is a pointer to the data itself, not to a PyArrayObject as the documentation states. There are two variables defined by weave that are available for use, out_array and py_out, which are of type PyArrayObject* and PyObject* respectively.

I have left an alternative version of the assignment commented out: weave automatically defines macros <VAR>1, <VAR>2, <VAR>3, and <VAR>4 to access items of arrays of the corresponding number of dimensions.