Question

I have an application, let’s call it myapp.exe, which is dual-mode console/GUI, built as /SUBSYSTEM:WINDOWS (There’s a tiny 3KB shim myapp.com to cause cmd.exe to wait to display the new prompt.)

If I launch from a command prompt:

• myapp -> cmd.exe runs myapp.com which runs myapp.exe. stdout is initially a detached console, by using AttachConsole and freopen("CONOUT$", "w", stdout) my output appears in the command box. OK
• myapp.exe -> cmd.exe displays the prompt too early (known problem), otherwise same as previous. Not a normal usage scenario.
• myapp > log -> stdout is a file, normal use of std::cout ends up in the file. OK

If I launch from Windows explorer:

• myapp.com -> console is created, stdout is console, output goes into console. Same result as using /SUBSYSTEM:CONSOLE for the entire program, except that I’ve added a pause when myapp.com is the only process in the console. Not a normal usage scenario.
• myapp.exe -> stdout is a NULL handle, I detect this and hook std::cout to a GUI. OK

If I launch from Matlab shell:

• system('myapp') or system('myapp.com') or system('myapp.exe') -> For all three variations, stdout is piped to MatLab. OK

If I launch from a cygwin bash shell:

• ./myapp.com -> Just like launch from cmd.exe, the output appears in the command box. OK
• ./myapp -> (bash finds ./myapp.exe). This is the broken case. stdout is a non-NULL handle but output goes nowhere. This is the normal situation for running the program from bash and needs to be fixed!
• ./myapp > log -> Just like launch from cmd.exe with file redirection. OK
• ./myapp | cat -> Similar to file redirection, except output ends up on the console window. OK

Does anybody know what cygwin sets as stdout when launching a /SUBSYSTEM:WINDOWS process and how I can bind std::cout to it? Or at least tell me how to find out what kind of handle I’m getting back from GetStdHandle(STD_OUTPUT_HANDLE)?

My program is written with Visual C++ 2010, without /clr, in case that matters in any way. OS is Windows 7 64-bit.

EDIT: Additional information requested.

CYGWIN environment variable is empty (or non-existent).

GetFileType() returns FILE_TYPE_UNKNOWN. GetLastError() returns 6 (ERROR_INVALID_HANDLE). It doesn’t matter whether I check before or after calling AttachConsole().

However, if I simply ignore the invalid handle and freopen("CONOUT$", "w", stdout) then everything works great. I was just missing a way to distinguish between (busted) console output and file redirection, and GetFileType() provided that.

EDIT: Final code:

bool is_console(HANDLE h)
{
    if (!h) return false;

    ::AttachConsole(ATTACH_PARENT_PROCESS);

    if (FILE_TYPE_UNKNOWN == ::GetFileType(h) && ERROR_INVALID_HANDLE == GetLastError()) {
        /* workaround cygwin brokenness */
        h = ::CreateFile(_T("CONOUT$"), GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, 0, NULL);
        if (h) {
            ::CloseHandle(h);
            return true;
        }
    }

    CONSOLE_FONT_INFO cfi;
    return ::GetCurrentConsoleFont(h, FALSE, &cfi) != 0;
}


bool init( void )
{
    HANDLE out = ::GetStdHandle(STD_OUTPUT_HANDLE);

    if (out) {
        /* stdout exists, might be console, file, or pipe */
        if (is_console(out)) {
#pragma warning(push)
#pragma warning(disable: 4996)
            freopen("CONOUT$", "w", stdout);
#pragma warning(pop)
        }
        //std::stringstream msg;
        //DWORD result = ::GetFileType(out);
        //DWORD lasterror = ::GetLastError();
        //msg << result << std::ends;
        //::MessageBoxA(NULL, msg.str().c_str(), "GetFileType", MB_OK);
        //if (result == FILE_TYPE_UNKNOWN) {
        //  msg.str(std::string());
        //  msg << lasterror << std::ends;
        //  ::MessageBoxA(NULL, msg.str().c_str(), "GetLastError", MB_OK);
        //}
        return true;
    }
    else {
        /* no text-mode stdout, launch GUI (actual code removed) */
    }
}

Answer 1

The GetFileType() function allows to distinguish between some types of handles, in particular consoles, pipes, files, and broken handles.