Question

Where does an application decide/cause to spawn a new console window (when linking dynamically)?
Is there some define or pragma I may have overlooked or some change in standard behavior in the default runtime dlls? I don’t feel very sure these are the right question to ask, please read the Background below and bear with me. I’m glad for every nudge in the right direction.

Background:
A console application (/SUBSYSTEM:CONSOLE) spawns a new daughter-console when started. Other applications using the same sources and configuration setup don’t. The only difference (seemingly) are the provided 3rd-party-dlls. Because of the new sub-console, I cannot redirect the output anymore (app.exe > dump.txt). Redirection is vital for this application.

I’m building that c/c++ console application on MS Visual Studio 2008. The application links dynamically to an awful messy lot of dlls. The whole Library-pack is provided by one 3rd party as release build with no debug info. The dll-pack includes, among others, msvcr90.dll (9.0.30729.1) msvcp90.dll (9.0.30729.1) and msvcr80.dll (8.0.50727.42).

Whatever causes this seem to be very global, since the first fprintf to stdout or stderr in main() goes to the new console window, not to the shell where the application was started.

In my first tries, I built the Application on Visual Studio 2005 (which we used until the latest version of the library package) and 2010 (which is recommended by the dll-provider). Those builds didn’t spawn consoles but did crash when memory was freed that probably was allocated in a different version of the runtime. Using Dependency Walker I could find the “main” runtime the dlls were linked against.

PS: Afaik linking against 2 different runtimes is dangerous enough. But it is an improvement over the previous version of the dll-pack, which included the r, c and m variants of 71, 80, 90 runtime dlls.

PPS: I mainly developed on linux before, so I could have made some very basic mistakes. Please accept my apologies for my ignorance in advance.

Update 1:

Following Anton Kovalenko’s advice, I chipped more and more libraries away. Then I removed more code. Finally I ended with:

#include <stdio.h>
#include <Windows.h>

int main(int _argc, char **_argv)
{
printf("application running ...\n");
fflush(stdout);
Sleep(2000);
exit(0);
}

Configuration Properties > c/c++ > Command Line:

/Od /Ob2 /D "_MBCS" /FD /EHsc /MD /Fo"a4input_interface_6.12_1.dir\Release\\" /Fd"a4input_interface_6.12_1.dir\Release\vc90.pdb" /W3 /nologo /c /TC /errorReport:prompt

Configuration Properties > Linker > Command Line:

/OUT:"D:\sascha\svn\a4_devel\cmakebuild\build\win32_release\inputinterfaces\.inter_612_1_32.exe" /VERSION:0.0 /INCREMENTAL:NO /NOLOGO /MANIFEST /MANIFESTFILE:"a4input_interface_6.12_1.dir\Release\.inter_612_1_32.exe.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /SUBSYSTEM:CONSOLE /DYNAMICBASE:NO /ERRORREPORT:PROMPT kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib

Configuration Properties > Manifest Tool > Command Line

/nologo /outputresource:"..\..\..\build\win32_release\inputinterfaces\.inter_612_1_32.exe;#1"

I still have the problem, that a shell is spawned and stdout cannot be redirected using (i.e. app.exe > test.txt). So the text “application running …” isn’t printed on the same shell where the exe is started.

I’m still clueless and still grateful for every hint.

Update 2:

I created a batch-file for the command lines. If I compile it with that, the exe works as intended.

cl.exe /Od /Ob2 /D "_MBCS" /FD /EHsc /MD /Fo"a4input_interface_6.12_1.dir\Release\\" /Fd"a4input_interface_6.12_1.dir\Release\vc90.pdb" /W3 /nologo /c /TC /errorReport:prompt main.c

link.exe /OUT:"D:\sascha\svn\a4_devel\cmakebuild\build\win32_release\inputinterfaces\.inter_612_1_32.exe" /VERSION:0.0 /INCREMENTAL:NO /NOLOGO /MANIFEST /MANIFESTFILE:"a4input_interface_6.12_1.dir\Release\.inter_612_1_32.exe.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /SUBSYSTEM:CONSOLE /DYNAMICBASE:NO /ERRORREPORT:PROMPT kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib a4input_interface_6.12_1.dir\Release\main.obj

mt.exe /manifest a4input_interface_6.12_1.dir\Release\.inter_612_1_32.exe.intermediate.manifest /nologo /outputresource:"D:\sascha\svn\a4_devel\cmakebuild\build\win32_release\inputinterfaces\.inter_612_1_32.exe";#1"

So there is something visual studio does that isn’t reflected in the Command Line it gives. What and why?
I’m still clueless and still grateful for every hint.

Solution:
There was a debugger configured for this name of executable. That’s why I had this behavior:

• inter_612_2_32.exe no sub shell
• intes_612_2_32.exe no sub shell
• inter_612_1_32.exe spawns sub shell
• intes_612_1_32.exe no sub shell

Using ProcessExplorer I found out, that inter_612_1_32.exe is a sub process of DbgHost.exe. Unfortunately I didn’t follow that lead and forgot about it.

The loaded Dlls actually didn’t matter. The problem was an entry in the Registry in:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\inter_612_1_32.exe

I guess the entry was created by DebugDiag, Application Verifier or some other tool but not removed when deleted in the tool’s GUI.

Thanks to everyone who thought about it.

Answer 1

Some of third-party dlls may be calling FreeConsole and AllocConsole, which have the effect you describe. If it’s done within DllMain for DLL_PROCESS_ATTACH, it will happen before entering main() for libraries which are your project dependencies.

There seems to be no alternative explanation related to your build environment.

You can learn more if you create a project which will use LoadLibrary for suspected dlls instead of linking against their import libraries: if some of them (or their dependencies) does FreeConsole and AllocConsole in DllMain, it will happen during LoadLibrary call.