Question

I’m experiencing different behavior when running some integration tests via IntelliJ IDEA 10 and using Ant from the command line. Specifically, tests that call singletons fail via Ant and succeed via IDEA. I’m aware of the downfalls of using singletons but this is code that I’ve inherited and would like to create tests before removing them 🙂

I’ve verified that both are using the same version of JUnit (4.8.1).

Here’s a snippet from the Ant build.xml file:

<junit printsummary="yes"
           fork="yes"
           forkmode="perTest"
           haltonfailure="no">
        <classpath refid="classpath.test"/>
        <formatter type="xml"/>
        <batchtest todir="${report.home}/tmp">
            <fileset dir="${test.home}">
                <include name="**/*Test.java"/>
                <exclude name="**/*TransactionalTests.java"/>
            </fileset>
        </batchtest>
    </junit>

Since I cannot find documentation on the default settings of JUnit in IDEA, I’m not able to determine if/how they fork, etc. when running tests.

Any suggestions on getting the same behavior via Ant and IntelliJ is greatly appreciated.

Answer 1

IDEA runs all the tests in the single JVM instance, while you are forking via Ant to run each test in its own JVM instance. Since you have singletons in your code, results could be different. IDEA has no option to fork tests at the moment, though this feature is planned.

The order of execution also might be the case since you are using singletons. IDEA runs tests in alphabetical order and there is no way to change it. So, to get identical behavior, you need to tell Ant to run your tests in the same order if the order of tests is important (which is not a good idea).

Also, you have some tests excluded via Ant, IDEA doesn’t have such option. If your other tests depend on the excluded tests, results will be different.