Question

I am asking this question to make sure some concept of parallel computing concept.

Lets give a simple example: We have a set of n numbers, what’s the best running time to search a item from it if we have at least n/3 parallel computers?

I think this will still be O(n), but not sure if I am right. Since the constant part of the big-Oh expression can be erased?

Thank you

Answer 1

It could be O(1) or O(ln n).

Given each of your n/3 computers n/(n/3) numbers; they all get essentially 3 values. It takes them individually constant time to search their constant sized-set and return a result (“0 –> not found”, k if found at the kth position in the array, if each is given K*(n/3) as the index in an array to start). So, the value is found in time O(1).

The issue comes in reporting the answer. Something has choose among the responses from the n/3 machines to pick a unique result. Typically this requires a “repeated” choice among the subsets of machines, which you can do in O(n) time but in parallel systems is often done with a “reduction” operator (such as SUM or MAX or …). Such reduction operators can be (and usually are) implemented using a reduction tree, which is logarithmic.

Some parallel hardware has very fast reduction hardware, but is it still logarithmic.
Weirdly enough, if you have n/1000 CPUs, you’ll still get O(1) search times (with a big constant), and O(ln n) reduction times with a very small constant. It’ll “look” like constant time if you ignore the O notation.