Question

I need to find the mode EXE/DLL was build looking at its headers. (Using C++ only without any external tools)

There is an old discussion on how to determine if DLL was built in Release or Debug mode.
http://forums.codeguru.com/archive/index.php/t-485996.html

But unfortunately, I did not find any clear answer.

Answer 1

I need to find the mode exe/dll was build looking at its headers.

If by “headers” you mean PE sections or resources (headers won’t tell you anything, and programs are not usually shipped with their development headers!), this is kind of possible, within limits, and unreliably. Otherwise, this is an entirely impossible endeavour unless you wrote the program yourself.

Generally, it is hard to do such a thing in a reliable way, even more so as “debug build” is a Microsoft Visual Studio simplification that does not exist as such under most compilers. For example, with GCC it is perfectly allowable to have an optimized build that nevertheless contains debug symbols. It is even possible to turn optimizations on and off with #pragma (and change the optimization level and even the target machine!) and thus have optimized functions (or groups of functions) in an unoptimized build, and vice versa.

The presence of debug symbols is your best guess for a program that you didn’t write. It is not possible (not realistically, in a simple, automated way, anyway) to tell from a generated binary whether it has been optimized or not.

The sections .debug$S and .debug$T contain debug symbols and debug types, respectively. There are some other sections starting with .debug as well, but they’re deprecated. A program that has been built in “debug mode” and that has not afterwards been stripped will contain some or all of these sections.
Using C++ with no external tools, you will want to skip over the DOS “MZ” stub and the PE header. After this come the section headers, which you can parse. Complete documenation of the file format can be downloaded here.
Most probably, reading the file in and doing a string match for .debug will be just as good.

Similarly, you can look at VERSIONINFO or the manifest file (they also allow to specify whether a program is a debug build), but these are not mandatory. You can write pretty much anything you want into these. Insofar, they’re even less reliable than looking for debug symbols.

Another hint, unreliable again, would be to check what versions of system libraries a program was linked with. If it’s the debug version, chances are it was a debug build. However, one could do a release build and still link with debug libraries, nothing can prevent you from doing that.

The next best guess would be the absence of calls to the CRT assert function (which you could do with a simple string match), since the assert macro (from which it’s normally called) is entirely stripped out in a build with NDEBUG defined. No use of that symbol, no string present in the binary.
Unluckily, a program that doesn’t have any asserts would be falsely identified as “release build” regardless of its actual build, and it is entirely possible to redefine the assert macro to do something completely different (such as printf a text and continue). And lastly, you don’t know wheter some static 3rd party library that you link with (which obviously has already passed the preprocessor) contains calls to assert that you don’t know about.

If you want to check a program that you wrote yourself, you can exploit the fact that the optimizer will completely remove things that are provably unreachable or not used. It may take 2-3 attempts to get it just right, but basically it should be as simple as defining a variable (or an exported function if your compiler/linker doesn’t export symbols that aren’t used) and writing two or three magic values to it from a program location that is not reachable. An optimizing compiler will at least collapse those several redundant moves into one, or more likely entirely eleminate them all.
You can then just do a binary string search for the magic values. If they’re not present, it’s an optimized build.