Question

I have a MFC application that launches other (generic windowed, black box) applications as a pop-up and waits for their completion. No communication/interaction between parent and child is required and should be avoided. Only the “child app behaves as modal dialog of parent app” behavior is wanted.
What is the correct way to do this?

An example of “launch another application as child window” can be seen at: Activating an application as a child/popup of another application which leads to http://www.codeproject.com/Articles/18724/Hosting-exe-applications-into-a-dialog (this is not what I want, I want modal popup behavior)

For simplicity we can assume that both launching and launched applications have single “stack” of windows (one main window with modal dialogs that can have own modal dialogs).

My current pseudo-code (error handling and callback function omitted for simplicity)

//get the current MFC dialog of launcher program we are launching the other app from
parentWnd = AfxGetMainWnd()->GetActiveWindow(); 
parentHwnd = parentWnd->GetSafeHwnd(); //HWND

// launch child and retrieve basic info from PROCESSINFO structure
CreateProcess(childExecutable); // => childProcessHandle, childProcessId 

//get the "main" window of child application
EnumWindows(EnumProc_That_Retrieves_TopLevelWindow_With_childProcessId); // => childHwnd

//link the child window as popup
SetWindowLong(childHwnd, GW_OWNER, parentHwnd);

//disable input into parent window
parentWnd->EnableWindow(FALSE);

//remove taskbar entry for child
SetWindowLong(child, GWL_EXSTYLE, GetWindowLong(child, GWL_EXSTYLE)&~WS_APPWINDOW);

//now keep waiting for the child process termination and process parent messages (e.g. WM_PAINT)
while (MsgWaitForMultipleObjects(childProcessHandle and process QS_ALLINPUT) {
   while (PeekMessage(PM_NOREMOVE)) AfxGetApp()->PumpMessage();
}

//re-enable input into parent window
parentWnd->EnableWindow(TRUE);

Now what my lesser problems are with foreground visual style (e.g. foreground = blue titlebar vs. background = gray titlebar) and keyboard input focus behavior:

1) initial removal of childs WS_APPWINDOW style removes foreground visual and input focus from child app. No application has focus at that point.

2) when user clicks any parent application window, the child foreground visual style is toggled. Keyboard focus is retained in the child application.
example: child app has foreground + focus -> click parent 1st time -> child loses foreground, retains focus -> click parent 2nd time -> child gains foreground, keeps focus -> etc.

expected behavior (what “normal” MFC popups do):
child app has foreground + focus -> click parent -> child titlebar flashes briefly and retains foreground + focus
child app does not have foreground + focus -> click parent -> child titlebar gains foreground and keyboard focus

And now this is worst problem:
3) I have encountered a MFC application that when launched separately user can open a “stack of modal dialogs” A->B->C->D->E, the ownership of windows is exactly matching this (E is owned by D, D owned by C etc.). But if I open it from my MFC application (M), the ownership looks like M->A->B->C,D,E (C,D,E are all owned by B, B owned by A, A owned by my app window M). This leads to “stack without support” http://blogs.msdn.com/b/oldnewthing/archive/2005/02/24/379635.aspx problem.
This behavior disappears when I remove SetWindowLong(childHwnd, GW_OWNER, parentHwnd) so messing with ownership probably triggers the unwanted behavior of child application, but without that I can’t seem to guarantee the “one is positioned over the other” premise of modal dialog.

So the grand question again: what is the correct way to do this task and avoid the problems I have described.

Edit

Solution so far

We must avoid messing with owner-owned structures as @mfc suggests below, so the task is basically to reimplement this aspect of window manager for just our parent-child pair in another way. I have prototyped part of solution using Windows Hooks. However it seems quite complex and tedious to complete, so I decided to go with another primitive approach (deadlines, oh deadlines). For the sake of example, I will describe basic ideas of both.

Hook solution

Disclaimer: only parent focus hook has been confirmed to work, rest is theorycrafting. Maybe there is be cleaner/more lightweight implementation, one could get inspiration in actual Windows window manager implementation (remember the whole point is avoiding setting GW_OWNER which works fine for window manager, but can break the child black-box app behavior).

• add some “ignore input while child is running” into parent message loop for when child app is running (intermittently) without windows
• create shared memory and structures to hold [parentPid, parentHwnd,childPid] for each invocation
• create DLL instanced memory for [list of parent nonowned windows, their UI thread, child hook]
• hook systemwide to WH_CBT -> HCBT_CREATEWND, if childPid matches, register window in list, register another hook HCBT_ACTIVATE just for that child thread if not already present
• hook systemwide to WH_CBT -> HCBT_DESTROYWND, if childPid matches, unregister window in list, unregister HCBT_ACTIVATE hook if this was last window for given thread, if this was last window for child app, unhook parent HCBT_ACTIVATE hook and focus parent
• parent thread HCBT_ACTIVATE hook prevents gaining focus and focuses child app instead using EnumWindows.
• child thread HCBT_ACTIVATE hooks prevent focus loss if the target is parent, keep parent just below child in Z-order
• create child process suspended and resume only when hooks are in place
• remember to unhook everywhere

Primitive approach

basically applying focus switch in first point of previous solution, when parent is clicked it flickers over child as focus is transferred back and forth.

• “ignore input while child is running” (discard various keypress, click, etc. messages) in parent message loop for when child app is running, focus child app instead using EnumWindows.

Answer 1

Changing parent/child relationship of other windows that does not belong to you is tricky and error prone.
If the launcher program has no communication with the launched program and the main purposes is to avoid any UI with the launcher, you can simply hide the launcher after the secondary program is launched successfully using ShowWindow(SW_HIDE).
In hidden mode, it continues to monitor the launched program and un-hide itself when the secondary program terminates.