I have about 400 unit tests that I have written for my Android application. If I run the tests package by package, everything works fine and all my tests pass. However, if I try to run them all at once, eventually (about 360 tests in), Android starts spitting out !!! Failed Binder Transaction Errors !!!. After about 10-20 of these errors, the process for the application I am testing gets killed and the unit tests never even complete.
I should note that during this time Eclipse reports Collecting test information in the Console. I think that is odd because it shows that it is running tests even though the JUnit UI doesn’t reflect that. When everything works properly, during the Collecting test information phase I see the tests run, the JUnit UI then comes up, and then all the tests are run again (I know this by reading logcat). That is (I think) a separate issue, but I felt I would mention it if anybody knew what that was about.
Edit as of June 6th 2011
As per Christopher’s answer below, I have verified that this only happens when attempting to run this through Eclipse. If i run my test suite using ANT, all the tests finally execute.
When I attempt to run my test suite now, Eclipse essentially hangs on “collecting test information”. I haven’t let it run for an extended period of time, but I will give that a try as soon as I can to see if it ever completes.
I investigated this again and, as far as I can tell, this is purely an issue with the Android Eclipse plugin. An issue that seems to be fixed, but still can occur sometimes (possibly it’s machine-dependent).
As you mention, the Eclipse plugin appears to run all the tests twice. What it’s actually doing the first time is gathering the test suite and test names, so that it can show all the test names in that nice hierarchical JUnit UI.
However, “running” the tests like this seems to cause a problem. As part of Android SDK Tools v8, a “workaround” was added which places a 15ms delay between checking each test. This is done to prevent “Binder transaction failures … for large test suites”.
Indeed, I went back to my project that was having the “FAILED BINDER TRANSACTION” errors, and I could not reproduce it on the command line. We also tried in Eclipse and it couldn’t be reproduced any more (even although I’m sure we had SDK Tools r8+ when we originally saw this).
However, I can still generally reproduce it in Eclipse. Try running your tests again from Eclipse with the latest Android tools, or try out this minimal GitHub repository I created and see if you can reproduce it:
https://github.com/orrc/android-large-test-failures#readme