Question

I was writing a function to figure out if a given system of linear inequalities has a solution, when all of a sudden it started giving the wrong answers after a seemingly innocuous change.

I undid some changes, re-did them, and then proceeded to fiddle for the next two hours, until I had reduced it to absurdity.

The following, inserted anywhere into the function body, but nowhere else in the program, fixes it:

if(0) {     __asm__('nop\n');     __asm__('nop\n');     __asm__('nop\n');     __asm__('nop\n'); } 

It’s for a school assignment, so I probably shouldn’t post the function on the web, but this is so ridiculous that I don’t think any context is going to help you. And all the function does is a bunch of math and looping. It doesn’t even touch memory that isn’t allocated on the stack.

Please help me make sense of the world! I’m loathe to chalk it up to the GCC, since the first rule of debugging is not to blame the compiler. But heck, I’m about to. I’m running Mac OS 10.5 on a G5 tower, and the compiler in question identifies itself as ‘powerpc-apple-darwin9-gcc-4.0.1’ but I’m thinking it could be an impostor…

UPDATE: Curiouser and curiouser… I diffed the .s files with nops and without. Not only are there too many differences to check, but with no nops the .s file is 196,620 bytes, and with it’s 156,719 bytes. (!)

UPDATE 2: Wow, should have posted the code! I came back to the code today, with fresh eyes, and immediately saw the error. See my sheepish self-answer below.

Answer 1

I came back to this after a few days busy with other things, and figured it out right away. Sorry I didn’t post the code sooner, but it was hard coming up with minimal example that displayed the problem.

The root problem was that I left out the return statements in the recursive function. I had:

bool function() {     /* lots of code */     function() } 

When it should have been:

bool function() {     /* lots of code */     return function() } 

This worked because, through the magic of optimization, the right value happened to be in the right register at the right time, and made it to the right place.

The bug was originally introduced when I broke the first call into its own special-cased function. And, at that point, the extra nops were the difference between this first case being inlined directly into the general recursive function.

Then, for reasons that I don’t fully understand, inlining this first case led to the right value not being in the right place at the right time, and the function returning junk.