Question

I wish to target ARMv6 with VFP Android device.

I have following line in my Android.mk file to enable VFP

LOCAL_CFLAGS    := -marm -mfloat-abi=softfp -mfpu=vfp -Wmultichar

I believe I target ARMv5 with VFP.

I edited android-ndk-r8b\toolchains\arm-linux-androideabi-4.6\setup.mk to remove -msoft-float. I also tried with original setup.mk

My code works fine 99.99% of time but some times goes crazy on ARMv6 devices.
I have special code to detect when it goes crazy.

Code

glm::vec3 D = P1 - P2;
float f1 = sqrtf(D.x*D.x + D.y*D.y + D.z*D.z);
if(!(f1 < 5)){
    // f1 is bigger then 5 or NaN
    mylog_fmt("Crazy %f %f %f %f", P1.x, P1.y, P1.z, f1);
    mylog_fmt("%f %f %f", P2.x, P2.y, P2.z);
}

LogCat:

12-14 00:59:08.214: I/APP(17091): Crazy -20.000031 0.000000 0.000000 20.000000
12-14 00:59:08.214: I/APP(17091): -20.000000 0.000000 0.000000

It calculates distance between 2 points. Usually it is 0.000031
But when crazy mode is on it is 20.0

The problem does not exists when I run it on ARMv7 CPU. It exists on ARMv6 CPU only.

I believe it should be some common known bug related to compiler settings or version. May be codes is missing memory barrier.

I would like to see some reference to similar bugs. Way to solve it. Or about nature of bug.

I also often get NaN values on ARMv6 when same code on ARMv7 does not give NaN.

I am debugging code for for 2 weeks already and searching the web. If someone could share link to similar problem it would be a great help!

PS. here is example of one of compile commands. I tried many different settings already.

Compiler Settings

c:/soft/Android/android-ndk-r8b/toolchains/arm-linux-androideabi-4.6/prebuilt/windows/bin/arm-linux-androideabi-g++
-MMD -MP -MF ./obj/local/armeabi/objs/main/sys/base.o.d -fpic -ffunction-sections -funwind-tables -fstack-protector 
-D__ARM_ARCH_5__ -D__ARM_ARCH_5T__ -D__ARM_ARCH_5E__
-D__ARM_ARCH_5TE__  
-march=armv5te -mtune=arm6 
-mfloat-abi=softfp -mfpu=vfp
-fno-exceptions -fno-rtti -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 
-Ijni/main/ -Ijni/main/sys -Ijni/main/bullet/src -Ijni/main/bullet/src/LinearMath -Ijni/main/bullet/src/BulletCollision/BroadphaseCollision 
-Ijni/main/bullet/src/BulletCollision/CollisionDispatch -Ijni/main/bullet/src/BulletCollision/CollisionShapes -Ijni/main/bullet/src/BulletCollision/NarrowPhaseCollision 
-Ijni/main/bullet/src/BulletDynamics/ConstraintSolver -Ijni/main/bullet/src/BulletDynamics/Dynamics -Ijni/main/../libzip/ -Ic:/soft/Android/android-ndk-r8b/sources/cxx-stl/stlport/stlport 
-Ic:/soft/Android/android-ndk-r8b/sources/cxx-stl//gabi++/include -Ijni/main 
-DANDROID

-marm -march=armv6 -mfloat-abi=softfp -mfpu=vfp -Wmultichar

-Wa,--noexecstack  -frtti  -O2 -DNDEBUG -g   -Ic:/soft/Android/android-ndk-r8b/platforms/android-5/arch-arm/usr/include -c  jni/main/sys/base.cpp
-o ./obj/local/armeabi/objs/main/sys/base.o

UPDATE 2

All these devices have Qualcomm MSM7227A
It has ARM1136JF-S

What I learnt so far is that the bug could relate to de-norms
I read somewhere ARMv7 differences WITH ARMv6 that is has denorms flush to zero by default and ARM1136SF-S has it optionally.
http://infocenter.arm.com/help/topic/com.arm.doc.ddi0211k/DDI0211K_arm1136_r1p5_trm.pdf

Not yet sure how to verify that Flush-To-ZERO flag on ARM.

UPDATE 3

This CPU’s VFP is called VFP11
I found --vfp11-denorm-fix option.
There is also --vfp-denorm-fix
They correct erratum in VFP11 cpus. Looks like my target problem.
Found few posts about VFP11 erratum. Hope it will fix the code.

Answer 1

Seems like I identified bug.

It is bug in VFP11 (ARMv6 coprocessor) denorm bug.
denormal numbers are very small number.

I get this numbers in physics code implementing spring with dumping

force1 = (Center - P1) * k1         // force1 directed to center 
force2 = - Velocity * k2            // force2 directed against velocity
Object->applyForce(force1)
Object->applyForce(force2)

Both forces get very small when object archieve Center and I get denormal values at the end.

I can re-write sring and dumping but I can’t re-write hole BulletPhysics or all math code and predict every (even internal) occurance of denormal number.

Linker has fix code options --vfp11-denorm-fix and --vfp-denorm-fix
http://sourceware.org/binutils/docs-2.19/ld/ARM.html

NDK linker has --vfp11-denorm-fix
This option helps. Code looks more repliable but it does not fix problem for 100%.

I see less bugs now.

BUt if I wait sping stabilize object then I finally I get denorm -> NaN

I have to wait longer but same problems arrive.

If you know solution that will fix code like --vfp11-denorm-fix should then I give you bounty.

I tried both --vfp11-denorm-fix=scalar and --vfp11-denorm-fix=vector

Flush to Zero bit

      int x;
      // compiles in ARM mode
      asm(
              "vmrs %[result],FPSCR \r\n"
              "orr %[result],%[result],#16777216 \r\n"
              "vmsr FPSCR,%[result]"
              :[result] "=r" (x) : :
      );

Not sure why but it requires LOCAL_ARM_MODE := arm in Android.mk
May be -mfpu=vfp-d16 instead of of just vfp is required.

Manually clear denormal numbers

I have spring code described above.
I improved it by clearing denormal number manually without using FPU with following function.

inline void fixDenorm(float & f){
    union FloatInt32 {
        unsigned int u32;
        float f32;
    };
        FloatInt32 fi;
        fi.f32 = f;

        unsigned int exponent = (fi.u32 >> 23) & ((1 << 8) - 1);
        if(exponent == 0)
            f = 0.f;
}

---

Original code was failing in 15-90 seconds from start in many places.

Current code showed issue possibly related to this bug in only one in place after 10 minutes of physics simulation.

---

Reference to bug and fix
http://sourceware.org/ml/binutils/2006-12/msg00196.html

They say that GCC uses only scalr code and --vfp11-denorm-fix=scalar is enough.
It adds 1 extra command to slow down. But even --vfp11-denorm-fix=vector that adds 2 extra commands is not enough.

Problem is not easier re-producible. On phones with higher frequency 800Mhz I see it more often then on slower one 600Mhz. It is possible that fix was done when there was no fast CPUs on market.

---

We have many files in project and every configuration compilations takes around 10 minutes.
Testing with current state of fix requires ~10 minutes to play on phone. + We heat phone under the lamp. Hot phone shows errors faster.

I wish to test different configurations and report what fix is most efficient. But right now we have to add hack to kill last bug possibly related to denorms.

I expected to find silver bullet that will fix it but only -msoft-float with 10x performance degradation or running app on ARMv7 does it.

---

After I replaced previous fixDenorm function with new fixDenormE in spring/dumping code and applying the new function for ViewMatrix I get rid of last bug.

inline void fixDenormE(float & f, float epsilon = 1e-8){
    union Data32 {
        unsigned int u32;
        float f32;
    };
        Data32 d;
        d.f32 = f;

        unsigned int exponent = (d.u32 >> 23) & ((1 << 8) - 1);
        if(exponent == 0)
            f = 0.f;
        if(fabsf(f) < epsilon){
          f = 0.f;
        }
}