Question

I have a vectorization Q in R using matrices. I have 2 Cols that need to be regressed against each using certain indices. Data is

matrix_senttoR = [ ...
                  0.11 0.95
                  0.23 0.34
                  0.67 0.54
                  0.65 0.95
                  0.12 0.54
                  0.45 0.43 ] ;
indices_forR = [ ...
            1
            1
            1
            2
            2
            2 ] ;

Col1 in matrix is data for say MSFT and GOOG (3 rows each) and Col2 is the return from benchmark StkIndex, on corresponding dates. The data is in matrix format as it is sent from Matlab.

I currently use

slope <- by(    data.frame(matrix_senttoR),   indices_forR,   FUN=function(x)  
                         {zyp.sen (X1~X2,data=x) $coeff[2] }      ) 
betasFac <- sapply(slope , function(x) x+0)

I’m using data.frame above as I could not use cbind(). If I use cbind() then Matlab gives an error as it doesn’t understand that format of data. I’m running these commands from inside Matlab (http://www.mathworks.com/matlabcentral/fileexchange/5051). You can replace zyp (zyp.sen) with lm.

BY is slow here (may be because of dataframes?). Is there a better way to do it? It takes 14secs+ for 150k rows of data. Can I instead use matrix-vectorization in R? Thanks.

Answer 1

I still think that you are overcomplicating things by moving from MATLAB to R and back. And passing 150k rows of data must be slowing things down considerably.

zyp.sen is actually pretty trivial to port to MATLAB. Here you go:

function [intercept, slope, intercepts, slopes, rank, residuals] = ZypSen(x, y)
% Computes a Thiel-Sen estimate of slope for a vector of data.

n = length(x);

slopes = arrayfun(@(i) ZypSlopediff(i, x, y, n), 1:(n - 1), ...
    'UniformOutput', false);
slopes =  [slopes{:}];
sni = isfinite(slopes);
slope = median(slopes(sni));

intercepts = y - slope * x;
intercept = median(intercepts);

rank = 2;
residuals = x - slope * y + intercept;

end


function z = ZypSlopediff(i, x, y, n)

z = (y(1:(n - i)) - y((i + 1):n)) ./ ...
    (x(1:(n - i)) - x((i + 1):n));

end

I checked this using the R’s example(zyp.sen), and it gives the same answer.

x = [0 1 2 4 5]
y = [6 4 1 8 7]
[int, sl, ints, sls, ra, res] = ZypSen(x, y)

You should really do some further checking though, just to be sure.