Question

I am trying to match a small array with size of ~20 in an larger array with size of ~200000. Both arrays contains double values. Match in this case means the smallest error, because there won’t be an exact match.

Next thing is, that I have to change the values of the small array, because it should also match if it’s different but has same gaps between the values, which means:

array 1: [1.3, 1.4, 1.3, 1.5, 1.7]
array 2: [..., 2.3, 2.4, 2.4, 2.5, 2.7, ...]

I have to bring the last element of each comparison to the same number. There the above example would be an extremely good match because first i would +1.0 the whole array #1.

[edit]
To clarify the above statement: Before calculating the error the example array should look like this:

array 1: [2.3, 2.4, 2.3, 2.5, 2.7] 
// (+1 of each element so the last element of the small array, 
// and the last element of the part of the large array I am 
// comparing to, has the same values: in this case: 2.7)
array 2: [..., 2.3, 2.4, 2.4, 2.5, 2.7, ...]

[/edit]

I know it is possible to simply iterate through the big array, but it is too slow. And of course instead of calculating the error by iterating through the array i can use vector operations like norm(v1 – v2).

So i have heard, that python is quite good for math operations, but i couldn’t find anything how to compare 2 arrays (just one number in an array).

Finally, the question is: Any ideas, how i can solve the problem in a really fast way. Which language is good to solve these kinds of problem (octave isn’t because it’s just fast at vector calculation, but slow with iterations) – probably there are some good libraries at python?

Let me know if I have to explain it more detailed.

Answer 1

I admit that I’m a little fuzzy on how your defining best match, but this example can be adjusted pretty easily. The magic is in the closeness function which receives a slice of data which is the same length as target and returns a number. The lower the number, the better the match.

import random

target = [random.random() * 10 for i in range(20)]
data   = [random.random() * 10 for i in range(200000)]

def closeness(a_range):
    diffs = list(map(lambda e: e[0]-e[1], zip(a_range, target)))
    avg_diffs = float(sum(diffs)) / len(diffs)
    adjusted_target = [i + avg_diffs for i in target]
    return sum(adjusted_target)

ranges = [data[i:i+len(target)] for i in range(len(data)-len(target))]
best_match = min(ranges, key=closeness) 

print(best_match)