Question

I understand how using vectorization in a language like MATLAB speeds up the code by removing the overhead of maintaining a loop variable, but how does the vectorization actually take place in the assembly / machine code? I mean there still has to be a loop somewhere, right?

Answer 1

Matlab ‘vectorization’ concept is completely different than the vector instructions concept, such as SSE. This is a common misunderstanding between two groups of people: matlab programmers and C/asm programmers. Matlab ‘vectorization’, as the word is commonly used, is only about expressing loops in the form of (vectors of) matrix indices, and sometimes about writing things in terms of basic matrix/vector operations (BLAS), instead of writing the loop itself. Matlab ‘vectorized’ code is not necessarily expressed as vectorized CPU instructions. Consider the following code:

A = rand(1000);
B = (A(1:2:end,:)+A(2:2:end,:))/2;

This code computes mean values for two adjacent matrix rows. It is a ‘vectorized’ matlab expression. However, since matlab stores matrices column-wise (columns are contiguous in memory), this operation is not trivially changed into operations on SSE vectors: since we perform the operations row-wise the data you need to load into the vectors is not stored contiguously in the memory.

This code on the other hand

A = rand(1000);
B = (A(:,1:2:end)+A(:,2:2:end))/2;

can take advantage of SSE instructions and streaming instructions, since we operate on two adjacent columns at a time.

So, matlab ‘vectorization’ is not equivalent to using CPU vector instructions. It is just a word used to signify the lack of a loop implemented in MATLAB. To add to the confusion, sometimes people even use the word to say that some loop has been implemented using a built-in function, such as arrayfun, or bsxfun. Which is even more misleading since those functions might be significantly slower than native matlab loops. As robince said, not all loops are slow in matlab nowadays, though you do need to know when they work, and when they don’t.

And in any way you always need a loop, it is just implemented in matlab built-in functions / BLAS instead of the users matlab code.