Question

When you have code:

for(int i = 0; i<N; i++)
{
   array[i] += N
}

Isn’t it comparing the variable i and N every time the loop iterates. For that matter isn’t it adding 1 to i every time the loop iterates as well?

So, isn’t this 3 operations per iteration of the loop?

Why do we usually ignore these operations and say that this code is O(n) ? Does it have something to do with how these operations are using the CPU?

Answer 1

Big-O notation does not deal with the actual cost of the operation, but how that cost grows with the size of the problem. To that extent, O(n) does not mean that the cost is n, but that the cost grows linearly with the size of the problem. Whatever the cost is for 100 elements, it will be doubled for 200 and tenfold for 1000. In the same way O(n^2) means that the cost grows quadratically, so if the size of the problem doubles the cost of the operation quadruples, if the size increases tenfold, the cost grows hundred-fold.

Constants don’t really matter here and they are usually factored out. More over, in many analysis the cost is not even expressed on actual time or memory cost, but the cost of other operations. For example, the std::map::find function is said to have O( log N ), regardless of the key type. The reason is exactly the same: O( log N ) means that whatever the cost of finding in a map with N elements is, it will grow logarithmically.

For a motivating example, consider a rather absurd problem: finding the author of a book from the full book contents, and two implementations. In the first implementation you use a std::map<std::string, std::string> where the first std::string is the contents of the book and the second the name of the author. The second implementation performs hashing of the contents of the book into an integer and stores that into an unordered std::vector< std::pair<int, std::string> >, the int being the hash and the std::string being the author’s name (assume no hash collisions). The cost of finding the author of the book in the map is O( log N ) and the cost of finding the author in the vector is O( N ), which is worse. But, those costs are hiding the complexity of the comparisons, the cost of comparing the whole book contents in the map might be huge compared to the cost of comparing the hash, up to the point that a single comparison of the contents of a book might be more expensive than all of the comparisons needed in the vector case.

Big-O notation deals only with how the cost grows with the size of the problem and hides the actual cost of each operation. When analyzing the complexity of an algorithm, the individual costs are ignored, but you should still be aware of them, as the Big-O notation does not tell the whole story and the practical cost of running your algorithm will incur in those constant costs that you ignored in the analysis.