Question

I have vectors of data that I feed through the filter() function — said filter was constructed to emit a reasonable approximation of the original signal that is then used to identify “bad” elements in the original data (said elements are typically caused by infrequent short-duration sensor malfunctions and are quite distinct from good data). After identifying these bad elements, I want to go back and replace them with something reasonable.

One approach would be to replace the bad values with the filtered output; however, the output was generated with the bad values, so it has some amount of undesired distortion.

Ideally, I’d like a way to tell filter() to assume that the bad element[s] are missing and that it should instead generate a reasonable interpolation of the missing value[s] (e.g., based on the surrounding values and the properties of the filter) for use when constructing the output.

I’ve been told that certain toolboxes allow insertion of special values (e.g., NaN) to indicate missing (but assumed to be well-behaved) data.

I looked at the source code for Octave’s filter() and nothing obvious leapt out at me — is there a special value (or other mechanism) to tell filter() to assume that well-behaved data is missing (and should be inserted as needed)?

Answer 1

Inserting NaN won’t work for this. The filter function is pretty simple–it simply implements an IIR filter.

If your signal is smooth and slowly-changing, you might get away with simply using interp1 to interpolate new values for the bad stretches based on the good data on either side.

If your signal has more complicated spectral content, I think “Wiener interpolation” is the phase to google for. For extrapolation you can use linear predictive coding.