Question

I have a 1 dimensional data set with some no data values which are set as 9999. Here is an extract as it is quite long:

this_array = [   4,    4,    1, 9999, 9999, 9999,   -5,   -4, ... ]

I would like to replace the no data values with the average of the closest values on either side, however as some no data values have closest values as no data values as well, replacing them is a little harder.
i.e. I would like the three no data values to be replaced with -2. I have created a loop to go through each of the scalars in the array and test for no data:

for k in this_array:
    if k == 9999:
        temp = np.where(k == 9999, (abs(this_array[k-1]-this_array[k+1])/2), this_array[k])
    else:
        pass
this_array[k] = temp

However I need to add in an if function or way to take the value before k-1 or after k+1 if that also is equal to 9999 e.g:

if np.logical_or(k+1 == 9999, k-1 == 9999):
    temp = np.where(k == 9999, (abs(this_array[k-2]-this_array[k+2])/2), this_array[k])

As one can tell, this code gets messy as one may end up taking the wrong value or ending up with loads of nested if functions.
Does anyone know of a cleaner way to implement this as it’s pretty variable throughout the dataset?

As requested: If the first and/or last points are no data, they would preferably be replaced with the closest data point.

Answer 1

There may be a more efficeint way to do this with numpy functions, but here is a solution using the itertools module:

from itertools import groupby

for k, g in groupby(range(len(this_array)), lambda i: this_array[i] == 9999):
    if k:
        indices = list(g)
        new_v = (this_array[indices[0]-1] + this_array[indices[-1]+1]) / 2
        this_array[indices[0]:indices[-1]+1].fill(new_v)

If the last element or first element can be 9999, you use the following:

from itertools import groupby

for k, g in groupby(range(len(this_array)), lambda i: this_array[i] == 9999):
    if k:
        indices = list(g)
        prev_i, next_i = indices[0]-1, indices[-1]+1
        before = this_array[prev_i] if prev_i != -1 else this_array[next_i]
        after = this_array[next_i] if next_i != len(this_array) else before
        this_array[indices[0]:next_i].fill((before + after) / 2)

Example using second version:

>>> from itertools import groupby
>>> this_array = np.array([9999, 4, 1, 9999, 9999, 9999, -5, -4, 9999])
>>> for k, g in groupby(range(len(this_array)), lambda i: this_array[i] == 9999):
...     if k:
...         indices = list(g)
...         prev_i, next_i = indices[0]-1, indices[-1]+1
...         before = this_array[prev_i] if prev_i != -1 else this_array[next_i]
...         after = this_array[next_i] if next_i != len(this_array) else before
...         this_array[indices[0]:next_i].fill((before + after) / 2)
...
>>> this_array
array([ 4,  4,  1, -2, -2, -2, -5, -4, -4])