This came up in an interview recently (the concept) and I was discussing it with a friend today. Here’s the idea:
Suppose you are calculating the dot product of a three-dimensional vector. Simple function would be “return (x1*x2)+(y1*y2)+(z1*z2)”.
However, if the first and second term are large positive numbers, they could overflow even if they answer is within the acceptable range. For example, let’s say the integer limit is 128. 100 + 100 – 80 = 120, but if you do the first two additions first you’ll overflow.
In a university class this came up on a C assignment where we were calculating something that we’d probably done a thousand times before but had never paid attention to overflow (this was the part when we were learning about writing sanitary code), like taking an average or something of the sort.
Anyone have any idea in what context this could have happened? I know it was some situation where you had to use comparisons either before or instead of addition/subtraction to avoid this overflow
Binary search is a classic example. Many many implementations do a computation that’s basically finding the midpoint of two indices:
(high + low) / 2
but if high and low are each near Integer.MAX_VALUE or the equivalent for your language, high+low overflows before the divide can happen, and your answer is wrong:
http://googleresearch.blogspot.com/2006/06/extra-extra-read-all-about-it-nearly.html
Easy fix is in that case to do:
high/2 + low/2
instead, which doesn’t overflow but it’s an almost universal bug in implementations of binary search, and it’s the very first thing to pop into mind when someone talks about an overflow in a computation that should result in a non-overflowing value.