Question

Several SciPy functions are documented as taking a “condensed distance matrix as returned by scipy.spatial.distance.pdist“. Now, inspection shows that what pdist returns is the row-major 1D-array form of the upper off-diagonal part of the distance matrix. This is all well and good, and natural and obvious, but is it documented or defined anywhere? I’d rather not assume anything about a data structure that’ll suddenly change. (Granted, there isn’t a lot of things it could change to, but I guess one possibility would be to wrap the array in an object that allows matrix-like indexing.)

Answer 1

Honestly, this is a better question for the scipy users or dev list, as it’s about future plans for scipy.

However, the structure is fairly rigorously documented in the docstrings for both scipy.spatial.pdist and in scipy.spatial.squareform.

E.g. for pdist:

Returns a condensed distance matrix Y.  For
    each :math:`i` and :math:`j` (where :math:`i<j<n`), the
    metric ``dist(u=X[i], v=X[j])`` is computed and stored in the
    :math:`ij`th entry.

    See ``squareform`` for information on how to calculate the index of
    this entry or to convert the condensed distance matrix to a
    redundant square matrix.

Becuase of this, and the fact that so many other functions in scipy.spatial expect a distance matrix in this form, I’d seriously doubt it’s going to change without a number of depreciation warnings and announcements.

Modules in scipy itself (as opposed to scipy’s scikits) are fairly stable, and there’s a great deal of consideration put into backwards compatibility when changes are made (and because of this, there’s quite a bit of legacy “cruft” in scipy: e.g. the fact that the core scipy module is just numpy with different defaults on a couple of functions.).